- Users - oVirt List Archives

[ANN] oVirt 3.5.5 Second Release Candidate is now available for testing
by Sandro Bonazzola 24 Sep '15

24 Sep '15

The oVirt Project is pleased to announce the availability of the Second Release Candidate of oVirt 3.5.5 for testing, as of September 24th, 2015. This release is available now for Red Hat Enterprise Linux 6.7, CentOS Linux 6.7 (or similar) and Red Hat Enterprise Linux 7.1, CentOS Linux 7.1 (or similar). This release supports Hypervisor Hosts running Red Hat Enterprise Linux 6.7, CentOS Linux 6.7 (or similar), Red Hat Enterprise Linux 7.1, CentOS Linux 7.1 (or similar) and #Fedora 21. This release of oVirt 3.5.5 includes new DWH and reports packages. See the release notes [1] for an initial list of bugs fixed. Please refer to release notes [1] for Installation / Upgrade instructions. New oVirt Node ISO will be available soon as well[2]. Please note that mirrors[3] may need usually one day before being synchronized. Please refer to the release notes for known issues in this release. Please test add yourself to the test page[4] if you're testing this release. [1] http://www.ovirt.org/OVirt_3.5.5_Release_Notes [2] http://plain.resources.ovirt.org/pub/ovirt-3.5-pre/iso/ [3] http://www.ovirt.org/Repository_mirrors#Current_mirrors [4] http://www.ovirt.org/Testing/oVirt_3.5.5_Testing -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

1 0

Strange disk behaviour within VMs
by Morten A. Middelthon 24 Sep '15

24 Sep '15

Hi, last Sunday I experienced a power outage with one of my two oVirt hypervisors. After power was restored I experienced some weirdness: - on one of the VMs running on this hypervisor the boot disk changed, so it was no longer able to boot. Looking at the console the VM would hang on "Booting from hard disk". After I noticed that the wrong virtual disk was marked as OS/bootable I got it booting again after correcting it to the proper boot disk. This was done from the oVirt management server. - on another VM I tried today to add another virtual disk to expand a LVM volume. In dmesg I can see the new device: [17167560.005768] vdc: unknown partition table However, when I tried to run pvcreate I got an error message saying that this was already marked as an LVM disk, and then running pvs give me the following error: # pvs Couldn't find device with uuid 7dDcyq-TZ6I-96Im-lfjL-cTUv-nff1-B11Mm7. PV VG Fmt Attr PSize PFree /dev/vda5 rit-kvm-ssweb02 lvm2 a-- 59.76g 0 /dev/vdb1 vg_syncshare lvm2 a-- 500.00g 0 /dev/vdc1 VG_SYNCSHARE01 lvm2 a-- 400.00g 0 unknown device VG_SYNCSHARE01 lvm2 a-m 1024.00g 0 As you can see there's already a PV called /dev/vdc1, as well another one named "unknown device". These two PVs belong to a VG that does NOT belong to this VM, VG_SYNCSHARE01. The uuids for these two PVs are: --- Physical volume --- PV Name unknown device VG Name VG_SYNCSHARE01 PV Size 1024.00 GiB / not usable 3.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 262143 Free PE 0 Allocated PE 262143 PV UUID 7dDcyq-TZ6I-96Im-lfjL-cTUv-nff1-B11Mm7 --- Physical volume --- PV Name /dev/vdc1 VG Name VG_SYNCSHARE01 PV Size 400.00 GiB / not usable 3.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 102399 Free PE 0 Allocated PE 102399 PV UUID oKSDoo-3pxU-0uue-zQ0H-kv1N-lyPa-P2M2FY The two PVs which doesn't belong on this VM actually belongs to a totally different VM. On VM number two: # pvdisplay --- Physical volume --- PV Name /dev/vdb1 VG Name VG_SYNCSHARE01 PV Size 1024.00 GiB / not usable 3.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 262143 Free PE 0 Allocated PE 262143 PV UUID 7dDcyq-TZ6I-96Im-lfjL-cTUv-nff1-B11Mm7 --- Physical volume --- PV Name /dev/vdd1 VG Name VG_SYNCSHARE01 PV Size 400.00 GiB / not usable 3.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 102399 Free PE 0 Allocated PE 102399 PV UUID oKSDoo-3pxU-0uue-zQ0H-kv1N-lyPa-P2M2FY As you can see, same uuid and VG name, but two different VMs. My setup: oVirt manager: oVirt 3.5 running on CentOS 6.7 oVirt hypervisors: two oVirt 3.5 servers running on CentOS 6.7 During the time of the power outage mentioned earlier I was running oVirt 3.4, but I upgraded today and rebooted the manager and both hypervisors, but NOT the VMs. Virtual machines: Debian wheezy 7.9 x86_64 Storage: HP LeftHand iSCSI I have tried to locate error messages in the logs which can be related to this behaviour, but so far no luck :( -- Morten A. Middelthon Email: morten(a)flipp.net

1 0

backup and restore the vm
by Budur Nagaraju 24 Sep '15

24 Sep '15

Hi can you pls provide me the tools to backup and restore the vm images in Ovirt? Thanks, Nagaraju

2 1

Not able to resume a VM which was paused because of gluster quorum issue
by Ramesh Nachimuthu 24 Sep '15

24 Sep '15

This is a multi-part message in MIME format. --------------050502080707090302020409 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hi, I am not able to resume a VM which was paused because of gluster client quorum issue. Here is what happened in my setup. 1. Created a gluster storage domain which is backed by gluster volume with replica 3. 2. Killed one brick process. So only two bricks are running in replica 3 setup. 3. Created two VMs 4. Started some IO using fio on both of the VMs 5. After some time got the following error in gluster mount and VMs moved to paused state. " server 10.70.45.17:49217 has not responded in the last 42 seconds, disconnecting." "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing WRITE as quorum is not met" more gluster mount logs at http://pastebin.com/UmiUQq0F 6. After some time gluster quorum is active and I am able to write the the gluster file system. 7. When I try to resume the VM it doesn't work and I got following error in vdsm log. http://pastebin.com/aXiamY15 Regards, Ramesh --------------050502080707090302020409 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, I am not able to resume a VM which was paused because of gluster client quorum issue. Here is what happened in my setup. 1. Created a gluster storage domain which is backed by gluster volume with replica 3. 2. Killed one brick process. So only two bricks are running in replica 3 setup. 3. Created two VMs 4. Started some IO using fio on both of the VMs 5. After some time got the following error in gluster mount and VMs moved to paused state. " <meta http-equiv="content-type" content="text/html; charset=utf-8"> <span style="color: rgb(51, 51, 51); font-family: monospace; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 13.2px; orphans: auto; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none; background-color: rgb(255, 255, 255);">server 10.70.45.17:49217 has not responded in the last 42 seconds, disconnecting." "<span style="color: rgb(51, 51, 51); font-family: monospace; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 13.2px; orphans: auto; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none; background-color: rgb(255, 255, 255);"><span style="color: rgb(51, 51, 51); font-family: monospace; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 13.2px; orphans: auto; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none; background-color: rgb(255, 255, 255);">vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing WRITE as quorum is not met" more gluster mount logs at <a class="moz-txt-link-freetext" href="http://pastebin.com/UmiUQq0F">http://pastebin.com/UmiUQq0F</a> 6. After some time gluster quorum is active and I am able to write the the gluster file system. 7. When I try to resume the VM it doesn't work and I got following error in vdsm log. <a class="moz-txt-link-freetext" href="http://pastebin.com/aXiamY15">http://pastebin.com/aXiamY15</a> Regards, Ramesh </body> </html> --------------050502080707090302020409--

4 10

LDAP Authentication
by Budur Nagaraju 23 Sep '15

23 Sep '15

HI All, Can someone help me in configuring LDAP authentication for Ovirt ? Thanks,, Nagaraju

4 22

Long export?
by Michael Kleinpaste 23 Sep '15

23 Sep '15

I'm exporting a VM as part of testing backing up VMs and this 75GB VM has been exporting for 3+ hours. The storage is running gluster on 10GbE so bandwidth isn't the issue. Importing these VMs from the same export took roughly 10 minutes. But I'm not seeing any errors. The only thing in the engine.log is: /var/log/ovirt-engine/engine.log:2015-09-21 13:16:14,459 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-6) [3ab846b4] Correlation ID: 1db67021, Job ID: b348e823-fb6a-42b8-8561-8ab6047a27c7, Call Stack: null, Custom Event ID: -1, Message: Starting export Vm asdf to export Is there a way to kill the export? -- *Michael Kleinpaste* Senior Systems Administrator SharperLending, LLC. www.SharperLending.com Michael.Kleinpaste(a)SharperLending.com (509) 324-1230 Fax: (509) 324-1234

3 4

FreeIPA
by suporte＠logicworks.pt 23 Sep '15

23 Sep '15

------=_Part_789470_851827211.1442587518543 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, Is there any documentation about FreeIPA integration with oVirt 3.5 and how to configure it? Thanks Jose -- Jose Ferradeira http://www.logicworks.pt ------=_Part_789470_851827211.1442587518543 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html><body><div style="font-family: Times New Roman; font-size: 10pt; color: #000000"><div>Hi, </div><div> </div><div>Is there any documentation about FreeIPA integration with oVirt 3.5 and how to configure it? </div><div> </div><div>Thanks </div><div> </div><div>Jose </div><div> </div><div>-- </div><div><hr style="width: 100%; height: 2px;" data-mce-style="width: 100%; height: 2px;">Jose Ferradeira http://www.logicworks.pt </div></div></body></html> ------=_Part_789470_851827211.1442587518543--

4 12

Re: [ovirt-users] Password getting failed while Conversion
by Richard W.M. Jones 23 Sep '15

23 Sep '15

On Wed, Sep 23, 2015 at 12:14:02PM -0400, Douglas Schilling Landgraf wrote: > > On 09/22/2015 12:27 AM, Budur Nagaraju wrote: > >Below is the format I have updated , > >but still am facing the same issues. To: Budur Nagaraju Please keep all your replies on the mailing list, as the mailing list archives are there to help others who may have the same problem in future. If you prefer to have personal help, you can pay for a Red Hat subscription. Please also update to the new virt-v2v version, as described in my previous email. The old version is unmaintained, and may not even work with oVirt (I don't know -- no one has tried it for about 3 years). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html

1 0

Problem migration VM
by Luca Bertoncello 23 Sep '15

23 Sep '15

Hi list! Sorry fort the previous E-Mail... problem on my Outlook... :( Here again... After a "war-week" I finally got a systemd-script to put the host in "maintenance" when a shutdown will started. Now the problem is, that the automatically migration of the VM does NOT work... I see in the Web console the host will "Preparing for maintenance" and the VM will start the migration, then the host is in "maintenance" and a couple of seconds later the VM will be killed on the other host... In the Log of the engine I see: 2015-09-23 11:14:17,165 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-30) [683624fe] Correlation ID: 52938d4d, Job ID: c0efe5b9-0bc3-4c81-9ee7-63ddf90a6afc, Call Stack: null, Custom Event ID: -1, Message: Mig ration failed while Host is in 'preparing for maintenance' state. Consider manual intervention: stopping/migrating Vms as Host's state will not turn to maintenance while VMs are still running on it.(VM: TestVM, Source: vmhost06, Destination: vmhost03). 2015-09-23 11:14:17,165 INFO [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (org.ovirt.thread.pool-8-thread-30) [683624fe] Lock freed to object EngineLock [exclusiveLocks= key: aabf6e76-8387-4441-a328-6a7dc32e2b4d value: VM , sharedLocks= ] (see http://pastebin.com/3Ca8W3vE) On the host I see these two errors: libvirtEventLoop::ERROR::2015-09-23 11:14:14,690::task::866::Storage.TaskManager.Task::(_setError) Task=`2670e82a-c9c7-4da6-b6f6-cff7bce25da1`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3209, in inappropriateDevices fails = supervdsm.getProxy().rmAppropriateRules(thiefId) File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in rmAppropriateRules File "/usr/lib64/python2.7/multiprocessing/managers.py", line 755, in _callmethod self._connect() File "/usr/lib64/python2.7/multiprocessing/managers.py", line 742, in _connect conn = self._Client(self._token.address, authkey=self._authkey) File "/usr/lib64/python2.7/multiprocessing/connection.py", line 173, in Client c = SocketClient(address) File "/usr/lib64/python2.7/multiprocessing/connection.py", line 308, in SocketClient s.connect(address) File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) error: [Errno 2] No such file or directory libvirtEventLoop::ERROR::2015-09-23 11:14:14,696::dispatcher::79::Storage.Dispatcher::(wrapper) [Errno 2] No such file or directory Traceback (most recent call last): File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/share/vdsm/storage/task.py", line 103, in wrapper return m(self, *a, **kw) File "/usr/share/vdsm/storage/task.py", line 1179, in prepare raise self.error error: [Errno 2] No such file or directory libvirtEventLoop::DEBUG::2015-09-23 11:14:14,697::vm::2799::vm.Vm::(setDownStatus) vmId=`aabf6e76-8387-4441-a328-6a7dc32e2b4d`::Changed state to Down: User shut down from within the guest (code=7) libvirtEventLoop::DEBUG::2015-09-23 11:14:14,698::sampling::425::vm.Vm::(stop) vmId=`aabf6e76-8387-4441-a328-6a7dc32e2b4d`::Stop statistics collection Thread-891::ERROR::2015-09-23 11:14:14,704::migration::161::vm.Vm::(_recover) vmId=`aabf6e76-8387-4441-a328-6a7dc32e2b4d`::'NoneType' object has no attribute 'XMLDesc' Thread-891::WARNING::2015-09-23 11:14:14,712::vm::1966::vm.Vm::(_set_lastStatus) vmId=`aabf6e76-8387-4441-a328-6a7dc32e2b4d`::trying to set state to Up when already Down Thread-891::ERROR::2015-09-23 11:14:14,712::migration::260::vm.Vm::(run) vmId=`aabf6e76-8387-4441-a328-6a7dc32e2b4d`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/virt/migration.py", line 231, in run self._setupRemoteMachineParams() File "/usr/share/vdsm/virt/migration.py", line 132, in _setupRemoteMachineParams self._machineParams['_srcDomXML'] = self._vm._dom.XMLDesc(0) AttributeError: 'NoneType' object has no attribute 'XMLDesc' Can someone help me finding the problem? Thanks Mit freundlichen Grüßen Luca Bertoncello -- Besuchen Sie unsere Webauftritte: www.queo.biz Agentur für Markenführung und Kommunikation www.queoflow.com IT-Consulting und Individualsoftwareentwicklung Luca Bertoncello Administrator Telefon: +49 351 21 30 38 0 Fax: +49 351 21 30 38 99 E-Mail: l.bertoncello(a)queo-group.com queo GmbH Tharandter Str. 13 01159 Dresden Sitz der Gesellschaft: Dresden Handelsregistereintrag: Amtsgericht Dresden HRB 22352 Geschäftsführer: Rüdiger Henke, André Pinkert USt-IdNr.: DE234220077

2 5

Re: [ovirt-users] adding gluster domains
by Sahina Bose 23 Sep '15

23 Sep '15

This is a multi-part message in MIME format. --------------090502070204020205010802 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit + ovirt-users Some clarity on your setup - sjcvhost03 - is this your arbiter node and ovirt management node? And are you running a compute + storage on the same nodes - i.e, sjcstorage01, sjcstorage02, sjcvhost03 (arbiter). CreateStorageDomainVDSCommand(HostName = sjcvhost03, CreateStorageDomainVDSCommandParameters:{runAsync='true', hostId='80245ac2-32a3-4d5d-b0fe-08019e2d1c9c', storageDomain='StorageDomainStatic:{name='sjcvmstore', id='597d5b5b-7c09-4de9-8840-6993bd9b61a6'}', args='sjcstorage01:/vmstore'}), log id: b9fe587 - fails with Error creating a storage domain's metadata: ("create meta file 'outbox' failed: [Errno 5] Input/output error", Are the vdsm logs you provided from sjcvhost03? There are no errors to be seen in the gluster log you provided. Could you provide mount log from sjcvhost03 (at /rhev/data-center/mnt/glusterSD/sjcstorage01:_vmstore.log most likely) If possible, /var/log/glusterfs/* from the 3 storage nodes. thanks sahina On 09/23/2015 05:02 AM, Brett Stevens wrote: > Hi Sahina, > > as requested here is some logs taken during a domain create. > > 2015-09-22 18:46:44,320 INFO > [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] > (DefaultQuartzScheduler_Worker-88) [] START, > GlusterVolumesListVDSCommand(HostName = sjcstorage01, > GlusterVolumesListVDSParameters:{runAsync='true', > hostId='c75682ba-1e4c-42a3-85c7-16e4bb2ce5da'}), log id: 2205ff1 > > 2015-09-22 18:46:44,413 WARN > [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] > (DefaultQuartzScheduler_Worker-88) [] Could not associate brick > 'sjcstorage01:/export/vmstore/brick01' of volume > '030f270a-0999-4df4-9b14-ae56eb0a2fb9' with correct network as no > gluster network found in cluster 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' > > 2015-09-22 18:46:44,417 WARN > [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] > (DefaultQuartzScheduler_Worker-88) [] Could not associate brick > 'sjcstorage02:/export/vmstore/brick01' of volume > '030f270a-0999-4df4-9b14-ae56eb0a2fb9' with correct network as no > gluster network found in cluster 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' > > 2015-09-22 18:46:44,417 WARN > [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] > (DefaultQuartzScheduler_Worker-88) [] Could not add brick > 'sjcvhost02:/export/vmstore/brick01' to volume > '030f270a-0999-4df4-9b14-ae56eb0a2fb9' - server uuid > '29b58278-9aa3-47c5-bfb4-1948ef7fdbba' not found in cluster > 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' > > 2015-09-22 18:46:44,418 INFO > [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] > (DefaultQuartzScheduler_Worker-88) [] FINISH, > GlusterVolumesListVDSCommand, return: > {030f270a-0999-4df4-9b14-ae56eb0a2fb9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@a0628f36}, > log id: 2205ff1 > > 2015-09-22 18:46:45,215 INFO > [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] > (default task-24) [5099cda3] Lock Acquired to object > 'EngineLock:{exclusiveLocks='[sjcstorage01:/vmstore=<STORAGE_CONNECTION, > ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' > > 2015-09-22 18:46:45,230 INFO > [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] > (default task-24) [5099cda3] Running command: > AddStorageServerConnectionCommand internal: false. Entities affected > : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group > CREATE_STORAGE_DOMAIN with role type ADMIN > > 2015-09-22 18:46:45,233 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] > (default task-24) [5099cda3] START, > ConnectStorageServerVDSCommand(HostName = sjcvhost03, > StorageServerConnectionManagementVDSParameters:{runAsync='true', > hostId='80245ac2-32a3-4d5d-b0fe-08019e2d1c9c', > storagePoolId='00000000-0000-0000-0000-000000000000', > storageType='GLUSTERFS', > connectionList='[StorageServerConnections:{id='null', > connection='sjcstorage01:/vmstore', iqn='null', vfsType='glusterfs', > mountOptions='null', nfsVersion='null', nfsRetrans='null', > nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6a112292 > > 2015-09-22 18:46:48,065 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] > (default task-24) [5099cda3] FINISH, ConnectStorageServerVDSCommand, > return: {00000000-0000-0000-0000-000000000000=0}, log id: 6a112292 > > 2015-09-22 18:46:48,073 INFO > [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] > (default task-24) [5099cda3] Lock freed to object > 'EngineLock:{exclusiveLocks='[sjcstorage01:/vmstore=<STORAGE_CONNECTION, > ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' > > 2015-09-22 18:46:48,188 INFO > [org.ovirt.engine.core.bll.storage.AddGlusterFsStorageDomainCommand] > (default task-23) [6410419] Running command: > AddGlusterFsStorageDomainCommand internal: false. Entities affected : > ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group > CREATE_STORAGE_DOMAIN with role type ADMIN > > 2015-09-22 18:46:48,206 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] > (default task-23) [6410419] START, > ConnectStorageServerVDSCommand(HostName = sjcvhost03, > StorageServerConnectionManagementVDSParameters:{runAsync='true', > hostId='80245ac2-32a3-4d5d-b0fe-08019e2d1c9c', > storagePoolId='00000000-0000-0000-0000-000000000000', > storageType='GLUSTERFS', > connectionList='[StorageServerConnections:{id='ec5ab31e-b5b9-4a8e-a2b2-0876df71a21e', > connection='sjcstorage01:/vmstore', iqn='null', vfsType='glusterfs', > mountOptions='null', nfsVersion='null', nfsRetrans='null', > nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 38a2b0d > > 2015-09-22 18:46:48,219 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] > (default task-23) [6410419] FINISH, ConnectStorageServerVDSCommand, > return: {ec5ab31e-b5b9-4a8e-a2b2-0876df71a21e=0}, log id: 38a2b0d > > 2015-09-22 18:46:48,221 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] > (default task-23) [6410419] START, > CreateStorageDomainVDSCommand(HostName = sjcvhost03, > CreateStorageDomainVDSCommandParameters:{runAsync='true', > hostId='80245ac2-32a3-4d5d-b0fe-08019e2d1c9c', > storageDomain='StorageDomainStatic:{name='sjcvmstore', > id='597d5b5b-7c09-4de9-8840-6993bd9b61a6'}', > args='sjcstorage01:/vmstore'}), log id: b9fe587 > > 2015-09-22 18:46:48,744 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (default task-23) [6410419] Correlation ID: null, Call Stack: null, > Custom Event ID: -1, Message: VDSM sjcvhost03 command failed: Error > creating a storage domain's metadata: ("create meta file 'outbox' > failed: [Errno 5] Input/output error",) > > 2015-09-22 18:46:48,744 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] > (default task-23) [6410419] Command > 'org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand' > return value 'StatusOnlyReturnForXmlRpc [status=StatusForXmlRpc > [code=362, message=Error creating a storage domain's metadata: > ("create meta file 'outbox' failed: [Errno 5] Input/output error",)]]' > > 2015-09-22 18:46:48,744 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] > (default task-23) [6410419] HostName = sjcvhost03 > > 2015-09-22 18:46:48,745 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] > (default task-23) [6410419] Command > 'CreateStorageDomainVDSCommand(HostName = sjcvhost03, > CreateStorageDomainVDSCommandParameters:{runAsync='true', > hostId='80245ac2-32a3-4d5d-b0fe-08019e2d1c9c', > storageDomain='StorageDomainStatic:{name='sjcvmstore', > id='597d5b5b-7c09-4de9-8840-6993bd9b61a6'}', > args='sjcstorage01:/vmstore'})' execution failed: VDSGenericException: > VDSErrorException: Failed in vdscommand to CreateStorageDomainVDS, > error = Error creating a storage domain's metadata: ("create meta file > 'outbox' failed: [Errno 5] Input/output error",) > > 2015-09-22 18:46:48,745 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] > (default task-23) [6410419] FINISH, CreateStorageDomainVDSCommand, log > id: b9fe587 > > 2015-09-22 18:46:48,745 ERROR > [org.ovirt.engine.core.bll.storage.AddGlusterFsStorageDomainCommand] > (default task-23) [6410419] Command > 'org.ovirt.engine.core.bll.storage.AddGlusterFsStorageDomainCommand' > failed: EngineException: > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: > VDSGenericException: VDSErrorException: Failed in vdscommand to > CreateStorageDomainVDS, error = Error creating a storage domain's > metadata: ("create meta file 'outbox' failed: [Errno 5] Input/output > error",) (Failed with error StorageDomainMetadataCreationError and > code 362) > > 2015-09-22 18:46:48,755 INFO > [org.ovirt.engine.core.bll.storage.AddGlusterFsStorageDomainCommand] > (default task-23) [6410419] Command > [id=5ae15f53-69a1-47c5-b3a5-82f32c20e48f]: Compensating NEW_ENTITY_ID > of org.ovirt.engine.core.common.businessentities.StorageDomainDynamic; > snapshot: 597d5b5b-7c09-4de9-8840-6993bd9b61a6. > > 2015-09-22 18:46:48,758 INFO > [org.ovirt.engine.core.bll.storage.AddGlusterFsStorageDomainCommand] > (default task-23) [6410419] Command > [id=5ae15f53-69a1-47c5-b3a5-82f32c20e48f]: Compensating NEW_ENTITY_ID > of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; > snapshot: 597d5b5b-7c09-4de9-8840-6993bd9b61a6. > > 2015-09-22 18:46:48,769 ERROR > [org.ovirt.engine.core.bll.storage.AddGlusterFsStorageDomainCommand] > (default task-23) [6410419] Transaction rolled-back for command > 'org.ovirt.engine.core.bll.storage.AddGlusterFsStorageDomainCommand'. > > 2015-09-22 18:46:48,784 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (default task-23) [6410419] Correlation ID: 6410419, Job ID: > 78692780-a06f-49a5-b6b1-e6c24a820d62, Call Stack: null, Custom Event > ID: -1, Message: Failed to add Storage Domain sjcvmstore. (User: > admin@internal) > > 2015-09-22 18:46:48,996 INFO > [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] > (default task-32) [1635a244] Lock Acquired to object > 'EngineLock:{exclusiveLocks='[ec5ab31e-b5b9-4a8e-a2b2-0876df71a21e=<STORAGE_CONNECTION, > ACTION_TYPE_FAILED_OBJECT_LOCKED>, > sjcstorage01:/vmstore=<STORAGE_CONNECTION, > ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' > > 2015-09-22 18:46:49,018 INFO > [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] > (default task-32) [1635a244] Running command: > RemoveStorageServerConnectionCommand internal: false. Entities > affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: > SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN > > 2015-09-22 18:46:49,024 INFO > [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] > (default task-32) [1635a244] Removing connection > 'ec5ab31e-b5b9-4a8e-a2b2-0876df71a21e' from database > > 2015-09-22 18:46:49,026 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] > (default task-32) [1635a244] START, > DisconnectStorageServerVDSCommand(HostName = sjcvhost03, > StorageServerConnectionManagementVDSParameters:{runAsync='true', > hostId='80245ac2-32a3-4d5d-b0fe-08019e2d1c9c', > storagePoolId='00000000-0000-0000-0000-000000000000', > storageType='GLUSTERFS', > connectionList='[StorageServerConnections:{id='ec5ab31e-b5b9-4a8e-a2b2-0876df71a21e', > connection='sjcstorage01:/vmstore', iqn='null', vfsType='glusterfs', > mountOptions='null', nfsVersion='null', nfsRetrans='null', > nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 39d3b568 > > 2015-09-22 18:46:49,248 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] > (default task-32) [1635a244] FINISH, > DisconnectStorageServerVDSCommand, return: > {ec5ab31e-b5b9-4a8e-a2b2-0876df71a21e=0}, log id: 39d3b568 > > 2015-09-22 18:46:49,252 INFO > [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] > (default task-32) [1635a244] Lock freed to object > 'EngineLock:{exclusiveLocks='[ec5ab31e-b5b9-4a8e-a2b2-0876df71a21e=<STORAGE_CONNECTION, > ACTION_TYPE_FAILED_OBJECT_LOCKED>, > sjcstorage01:/vmstore=<STORAGE_CONNECTION, > ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' > > 2015-09-22 18:46:49,431 INFO > [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] > (DefaultQuartzScheduler_Worker-3) [] START, > GlusterVolumesListVDSCommand(HostName = sjcstorage01, > GlusterVolumesListVDSParameters:{runAsync='true', > hostId='c75682ba-1e4c-42a3-85c7-16e4bb2ce5da'}), log id: 17014ae8 > > 2015-09-22 18:46:49,511 WARN > [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] > (DefaultQuartzScheduler_Worker-3) [] Could not associate brick > 'sjcstorage01:/export/vmstore/brick01' of volume > '030f270a-0999-4df4-9b14-ae56eb0a2fb9' with correct network as no > gluster network found in cluster 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' > > 2015-09-22 18:46:49,515 WARN > [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] > (DefaultQuartzScheduler_Worker-3) [] Could not associate brick > 'sjcstorage02:/export/vmstore/brick01' of volume > '030f270a-0999-4df4-9b14-ae56eb0a2fb9' with correct network as no > gluster network found in cluster 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' > > 2015-09-22 18:46:49,516 WARN > [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] > (DefaultQuartzScheduler_Worker-3) [] Could not add brick > 'sjcvhost02:/export/vmstore/brick01' to volume > '030f270a-0999-4df4-9b14-ae56eb0a2fb9' - server uuid > '29b58278-9aa3-47c5-bfb4-1948ef7fdbba' not found in cluster > 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' > > 2015-09-22 18:46:49,516 INFO > [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] > (DefaultQuartzScheduler_Worker-3) [] FINISH, > GlusterVolumesListVDSCommand, return: > {030f270a-0999-4df4-9b14-ae56eb0a2fb9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@92ed0f75}, > log id: 17014ae8 > > > > ovirt engine thinks that sjcstorage01 is sjcstorage01, its all testbed > at the moment and is all short names, defined in /etc/hosts (all > copied to each server for consistancy) > > > volume info for vmstore is > > > Status of volume: vmstore > > Gluster process TCP Port RDMA Port Online Pid > > ------------------------------------------------------------------------------ > > Brick sjcstorage01:/export/vmstore/brick01 49157 0 Y 7444 > > Brick sjcstorage02:/export/vmstore/brick01 49157 0 Y 4063 > > Brick sjcvhost02:/export/vmstore/brick01 49156 0 Y 3243 > > NFS Server on localhost 2049 0 Y 3268 > > Self-heal Daemon on localhost N/A N/A Y 3284 > > NFS Server on sjcstorage01 2049 0 Y 7463 > > Self-heal Daemon on sjcstorage01 N/A N/A Y > 7472 > > NFS Server on sjcstorage02 2049 0 Y 4082 > > Self-heal Daemon on sjcstorage02 N/A N/A Y > 4090 > > Task Status of Volume vmstore > > ------------------------------------------------------------------------------ > > There are no active volume tasks > > > > vdsm logs from time the domain is added > > > hread-789::DEBUG::2015-09-22 > 19:12:05,865::stompreactor::304::yajsonrpc.StompServer::(send) Sending > response > > Thread-790::DEBUG::2015-09-22 > 19:12:07,797::task::595::Storage.TaskManager.Task::(_updateState) > Task=`93731f26-a48f-45c9-9959-42c96b09cf85`::moving from state init -> > state preparing > > Thread-790::INFO::2015-09-22 > 19:12:07,797::logUtils::48::dispatcher::(wrapper) Run and protect: > repoStats(options=None) > > Thread-790::INFO::2015-09-22 > 19:12:07,797::logUtils::51::dispatcher::(wrapper) Run and protect: > repoStats, Return response: {} > > Thread-790::DEBUG::2015-09-22 > 19:12:07,797::task::1191::Storage.TaskManager.Task::(prepare) > Task=`93731f26-a48f-45c9-9959-42c96b09cf85`::finished: {} > > Thread-790::DEBUG::2015-09-22 > 19:12:07,797::task::595::Storage.TaskManager.Task::(_updateState) > Task=`93731f26-a48f-45c9-9959-42c96b09cf85`::moving from state > preparing -> state finished > > Thread-790::DEBUG::2015-09-22 > 19:12:07,797::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) > Owner.releaseAll requests {} resources {} > > Thread-790::DEBUG::2015-09-22 > 19:12:07,797::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) > Owner.cancelAll requests {} > > Thread-790::DEBUG::2015-09-22 > 19:12:07,797::task::993::Storage.TaskManager.Task::(_decref) > Task=`93731f26-a48f-45c9-9959-42c96b09cf85`::ref 0 aborting False > > Thread-790::DEBUG::2015-09-22 > 19:12:07,802::stompreactor::304::yajsonrpc.StompServer::(send) Sending > response > > Reactor thread::INFO::2015-09-22 > 19:12:14,816::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) > Accepting connection from 127.0.0.1:52510 <http://127.0.0.1:52510> > > Reactor thread::DEBUG::2015-09-22 > 19:12:14,822::protocoldetector::82::ProtocolDetector.Detector::(__init__) > Using required_size=11 > > Reactor thread::INFO::2015-09-22 > 19:12:14,823::protocoldetector::118::ProtocolDetector.Detector::(handle_read) > Detected protocol xml from 127.0.0.1:52510 <http://127.0.0.1:52510> > > Reactor thread::DEBUG::2015-09-22 > 19:12:14,823::bindingxmlrpc::1297::XmlDetector::(handle_socket) xml > over http detected from ('127.0.0.1', 52510) > > BindingXMLRPC::INFO::2015-09-22 > 19:12:14,823::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting > request handler for 127.0.0.1:52510 <http://127.0.0.1:52510> > > Thread-791::INFO::2015-09-22 > 19:12:14,823::xmlrpc::84::vds.XMLRPCServer::(_process_requests) > Request handler for 127.0.0.1:52510 <http://127.0.0.1:52510> started > > Thread-791::INFO::2015-09-22 > 19:12:14,825::xmlrpc::92::vds.XMLRPCServer::(_process_requests) > Request handler for 127.0.0.1:52510 <http://127.0.0.1:52510> stopped > > Thread-792::DEBUG::2015-09-22 > 19:12:20,872::stompreactor::304::yajsonrpc.StompServer::(send) Sending > response > > Thread-793::DEBUG::2015-09-22 > 19:12:22,832::task::595::Storage.TaskManager.Task::(_updateState) > Task=`a1f48f6f-a9ba-4dac-b024-ae6289f4a7dd`::moving from state init -> > state preparing > > Thread-793::INFO::2015-09-22 > 19:12:22,832::logUtils::48::dispatcher::(wrapper) Run and protect: > repoStats(options=None) > > Thread-793::INFO::2015-09-22 > 19:12:22,832::logUtils::51::dispatcher::(wrapper) Run and protect: > repoStats, Return response: {} > > Thread-793::DEBUG::2015-09-22 > 19:12:22,832::task::1191::Storage.TaskManager.Task::(prepare) > Task=`a1f48f6f-a9ba-4dac-b024-ae6289f4a7dd`::finished: {} > > Thread-793::DEBUG::2015-09-22 > 19:12:22,832::task::595::Storage.TaskManager.Task::(_updateState) > Task=`a1f48f6f-a9ba-4dac-b024-ae6289f4a7dd`::moving from state > preparing -> state finished > > Thread-793::DEBUG::2015-09-22 > 19:12:22,833::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) > Owner.releaseAll requests {} resources {} > > Thread-793::DEBUG::2015-09-22 > 19:12:22,833::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) > Owner.cancelAll requests {} > > Thread-793::DEBUG::2015-09-22 > 19:12:22,833::task::993::Storage.TaskManager.Task::(_decref) > Task=`a1f48f6f-a9ba-4dac-b024-ae6289f4a7dd`::ref 0 aborting False > > Thread-793::DEBUG::2015-09-22 > 19:12:22,837::stompreactor::304::yajsonrpc.StompServer::(send) Sending > response > > Reactor thread::INFO::2015-09-22 > 19:12:29,841::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) > Accepting connection from 127.0.0.1:52511 <http://127.0.0.1:52511> > > Reactor thread::DEBUG::2015-09-22 > 19:12:29,848::protocoldetector::82::ProtocolDetector.Detector::(__init__) > Using required_size=11 > > Reactor thread::INFO::2015-09-22 > 19:12:29,849::protocoldetector::118::ProtocolDetector.Detector::(handle_read) > Detected protocol xml from 127.0.0.1:52511 <http://127.0.0.1:52511> > > Reactor thread::DEBUG::2015-09-22 > 19:12:29,849::bindingxmlrpc::1297::XmlDetector::(handle_socket) xml > over http detected from ('127.0.0.1', 52511) > > BindingXMLRPC::INFO::2015-09-22 > 19:12:29,849::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting > request handler for 127.0.0.1:52511 <http://127.0.0.1:52511> > > Thread-794::INFO::2015-09-22 > 19:12:29,849::xmlrpc::84::vds.XMLRPCServer::(_process_requests) > Request handler for 127.0.0.1:52511 <http://127.0.0.1:52511> started > > Thread-794::INFO::2015-09-22 > 19:12:29,851::xmlrpc::92::vds.XMLRPCServer::(_process_requests) > Request handler for 127.0.0.1:52511 <http://127.0.0.1:52511> stopped > > Thread-795::DEBUG::2015-09-22 > 19:12:35,520::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) > Calling 'StoragePool.connectStorageServer' in bridge with > {u'connectionParams': [{u'id': > u'00000000-0000-0000-0000-000000000000', u'connection': > u'sjcstorage01:/vmstore', u'iqn': u'', u'user': u'', u'tpgt': u'1', > u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], > u'storagepoolID': u'00000000-0000-0000-0000-000000000000', > u'domainType': 7} > > Thread-795::DEBUG::2015-09-22 > 19:12:35,520::task::595::Storage.TaskManager.Task::(_updateState) > Task=`6e8aec06-556f-4659-9ee8-efc60b637ff6`::moving from state init -> > state preparing > > Thread-795::INFO::2015-09-22 > 19:12:35,521::logUtils::48::dispatcher::(wrapper) Run and protect: > connectStorageServer(domType=7, > spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'id': > u'00000000-0000-0000-0000-000000000000', u'connection': > u'sjcstorage01:/vmstore', u'iqn': u'', u'user': u'', u'tpgt': u'1', > u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], > options=None) > > Thread-795::DEBUG::2015-09-22 > 19:12:35,539::fileUtils::143::Storage.fileUtils::(createdir) Creating > directory: /rhev/data-center/mnt/glusterSD/sjcstorage01:_vmstore mode: > None > > Thread-795::DEBUG::2015-09-22 > 19:12:35,540::mount::229::Storage.Misc.excCmd::(_runcmd) /usr/bin/sudo > -n /usr/bin/systemd-run --scope --slice=vdsm-glusterfs /usr/bin/mount > -t glusterfs -o backup-volfile-servers=sjcstorage02:sjcvhost02 > sjcstorage01:/vmstore > /rhev/data-center/mnt/glusterSD/sjcstorage01:_vmstore (cwd None) > > Thread-795::DEBUG::2015-09-22 > 19:12:35,706::hsm::2417::Storage.HSM::(__prefetchDomains) > glusterDomPath: glusterSD/* > > Thread-796::DEBUG::2015-09-22 > 19:12:35,707::__init__::298::IOProcessClient::(_run) Starting IOProcess... > > Thread-797::DEBUG::2015-09-22 > 19:12:35,712::__init__::298::IOProcessClient::(_run) Starting IOProcess... > > Thread-795::DEBUG::2015-09-22 > 19:12:35,721::hsm::2429::Storage.HSM::(__prefetchDomains) Found SD > uuids: () > > Thread-795::DEBUG::2015-09-22 > 19:12:35,721::hsm::2489::Storage.HSM::(connectStorageServer) knownSDs: > {41b75ca9-9783-42a7-9a23-10a2ae3cbb96: storage.glusterSD.findDomain, > 597d5b5b-7c09-4de9-8840-6993bd9b61a6: storage.glusterSD.findDomain, > ef17fec4-fecf-4d7e-b815-d1db4ef65225: storage.glusterSD.findDomain} > > Thread-795::INFO::2015-09-22 > 19:12:35,721::logUtils::51::dispatcher::(wrapper) Run and protect: > connectStorageServer, Return response: {'statuslist': [{'status': 0, > 'id': u'00000000-0000-0000-0000-000000000000'}]} > > Thread-795::DEBUG::2015-09-22 > 19:12:35,722::task::1191::Storage.TaskManager.Task::(prepare) > Task=`6e8aec06-556f-4659-9ee8-efc60b637ff6`::finished: {'statuslist': > [{'status': 0, 'id': u'00000000-0000-0000-0000-000000000000'}]} > > Thread-795::DEBUG::2015-09-22 > 19:12:35,722::task::595::Storage.TaskManager.Task::(_updateState) > Task=`6e8aec06-556f-4659-9ee8-efc60b637ff6`::moving from state > preparing -> state finished > > Thread-795::DEBUG::2015-09-22 > 19:12:35,722::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) > Owner.releaseAll requests {} resources {} > > Thread-795::DEBUG::2015-09-22 > 19:12:35,722::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) > Owner.cancelAll requests {} > > Thread-795::DEBUG::2015-09-22 > 19:12:35,722::task::993::Storage.TaskManager.Task::(_decref) > Task=`6e8aec06-556f-4659-9ee8-efc60b637ff6`::ref 0 aborting False > > Thread-795::DEBUG::2015-09-22 > 19:12:35,722::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) > Return 'StoragePool.connectStorageServer' in bridge with [{'status': > 0, 'id': u'00000000-0000-0000-0000-000000000000'}] > > Thread-795::DEBUG::2015-09-22 > 19:12:35,722::stompreactor::304::yajsonrpc.StompServer::(send) Sending > response > > Thread-798::DEBUG::2015-09-22 > 19:12:35,775::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) > Calling 'StoragePool.connectStorageServer' in bridge with > {u'connectionParams': [{u'id': > u'cd55e6a1-022a-4b32-8a94-cab506a9b054', u'connection': > u'sjcstorage01:/vmstore', u'iqn': u'', u'user': u'', u'tpgt': u'1', > u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], > u'storagepoolID': u'00000000-0000-0000-0000-000000000000', > u'domainType': 7} > > Thread-798::DEBUG::2015-09-22 > 19:12:35,775::task::595::Storage.TaskManager.Task::(_updateState) > Task=`b2c91515-bdda-45e5-a031-61a1e2c53c4d`::moving from state init -> > state preparing > > Thread-798::INFO::2015-09-22 > 19:12:35,776::logUtils::48::dispatcher::(wrapper) Run and protect: > connectStorageServer(domType=7, > spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'id': > u'cd55e6a1-022a-4b32-8a94-cab506a9b054', u'connection': > u'sjcstorage01:/vmstore', u'iqn': u'', u'user': u'', u'tpgt': u'1', > u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], > options=None) > > Thread-798::DEBUG::2015-09-22 > 19:12:35,777::hsm::2417::Storage.HSM::(__prefetchDomains) > glusterDomPath: glusterSD/* > > Thread-798::DEBUG::2015-09-22 > 19:12:35,782::hsm::2429::Storage.HSM::(__prefetchDomains) Found SD > uuids: () > > Thread-798::DEBUG::2015-09-22 > 19:12:35,782::hsm::2489::Storage.HSM::(connectStorageServer) knownSDs: > {41b75ca9-9783-42a7-9a23-10a2ae3cbb96: storage.glusterSD.findDomain, > 597d5b5b-7c09-4de9-8840-6993bd9b61a6: storage.glusterSD.findDomain, > ef17fec4-fecf-4d7e-b815-d1db4ef65225: storage.glusterSD.findDomain} > > Thread-798::INFO::2015-09-22 > 19:12:35,782::logUtils::51::dispatcher::(wrapper) Run and protect: > connectStorageServer, Return response: {'statuslist': [{'status': 0, > 'id': u'cd55e6a1-022a-4b32-8a94-cab506a9b054'}]} > > Thread-798::DEBUG::2015-09-22 > 19:12:35,783::task::1191::Storage.TaskManager.Task::(prepare) > Task=`b2c91515-bdda-45e5-a031-61a1e2c53c4d`::finished: {'statuslist': > [{'status': 0, 'id': u'cd55e6a1-022a-4b32-8a94-cab506a9b054'}]} > > Thread-798::DEBUG::2015-09-22 > 19:12:35,783::task::595::Storage.TaskManager.Task::(_updateState) > Task=`b2c91515-bdda-45e5-a031-61a1e2c53c4d`::moving from state > preparing -> state finished > > Thread-798::DEBUG::2015-09-22 > 19:12:35,783::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) > Owner.releaseAll requests {} resources {} > > Thread-798::DEBUG::2015-09-22 > 19:12:35,783::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) > Owner.cancelAll requests {} > > Thread-798::DEBUG::2015-09-22 > 19:12:35,783::task::993::Storage.TaskManager.Task::(_decref) > Task=`b2c91515-bdda-45e5-a031-61a1e2c53c4d`::ref 0 aborting False > > Thread-798::DEBUG::2015-09-22 > 19:12:35,783::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) > Return 'StoragePool.connectStorageServer' in bridge with [{'status': > 0, 'id': u'cd55e6a1-022a-4b32-8a94-cab506a9b054'}] > > Thread-798::DEBUG::2015-09-22 > 19:12:35,783::stompreactor::304::yajsonrpc.StompServer::(send) Sending > response > > Thread-801::DEBUG::2015-09-22 > 19:12:35,787::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) > Calling 'StorageDomain.create' in bridge with {u'name': > u'sjcvmstore01', u'domainType': 7, u'domainClass': 1, u'typeArgs': > u'sjcstorage01:/vmstore', u'version': u'3', u'storagedomainID': > u'c02fda97-62e3-40d3-9a6e-ac5d100f8ad3'} > > Thread-801::DEBUG::2015-09-22 > 19:12:35,788::task::595::Storage.TaskManager.Task::(_updateState) > Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::moving from state init -> > state preparing > > Thread-801::INFO::2015-09-22 > 19:12:35,788::logUtils::48::dispatcher::(wrapper) Run and protect: > createStorageDomain(storageType=7, > sdUUID=u'c02fda97-62e3-40d3-9a6e-ac5d100f8ad3', > domainName=u'sjcvmstore01', typeSpecificArg=u'sjcstorage01:/vmstore', > domClass=1, domVersion=u'3', options=None) > > Thread-801::DEBUG::2015-09-22 > 19:12:35,788::misc::733::Storage.SamplingMethod::(__call__) Trying to > enter sampling method (storage.sdc.refreshStorage) > > Thread-801::DEBUG::2015-09-22 > 19:12:35,788::misc::736::Storage.SamplingMethod::(__call__) Got in to > sampling method > > Thread-801::DEBUG::2015-09-22 > 19:12:35,788::misc::733::Storage.SamplingMethod::(__call__) Trying to > enter sampling method (storage.iscsi.rescan) > > Thread-801::DEBUG::2015-09-22 > 19:12:35,788::misc::736::Storage.SamplingMethod::(__call__) Got in to > sampling method > > Thread-801::DEBUG::2015-09-22 > 19:12:35,788::iscsi::431::Storage.ISCSI::(rescan) Performing SCSI > scan, this will take up to 30 seconds > > Thread-801::DEBUG::2015-09-22 > 19:12:35,788::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) > /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None) > > Thread-801::DEBUG::2015-09-22 > 19:12:35,821::misc::743::Storage.SamplingMethod::(__call__) Returning > last result > > Thread-801::DEBUG::2015-09-22 > 19:12:35,821::misc::733::Storage.SamplingMethod::(__call__) Trying to > enter sampling method (storage.hba.rescan) > > Thread-801::DEBUG::2015-09-22 > 19:12:35,821::misc::736::Storage.SamplingMethod::(__call__) Got in to > sampling method > > Thread-801::DEBUG::2015-09-22 > 19:12:35,821::hba::56::Storage.HBA::(rescan) Starting scan > > Thread-802::DEBUG::2015-09-22 > 19:12:35,882::stompreactor::304::yajsonrpc.StompServer::(send) Sending > response > > Thread-801::DEBUG::2015-09-22 > 19:12:35,912::hba::62::Storage.HBA::(rescan) Scan finished > > Thread-801::DEBUG::2015-09-22 > 19:12:35,912::misc::743::Storage.SamplingMethod::(__call__) Returning > last result > > Thread-801::DEBUG::2015-09-22 > 19:12:35,912::multipath::77::Storage.Misc.excCmd::(rescan) > /usr/bin/sudo -n /usr/sbin/multipath (cwd None) > > Thread-801::DEBUG::2015-09-22 > 19:12:35,936::multipath::77::Storage.Misc.excCmd::(rescan) SUCCESS: > <err> = ''; <rc> = 0 > > Thread-801::DEBUG::2015-09-22 > 19:12:35,936::utils::661::root::(execCmd) /sbin/udevadm settle > --timeout=5 (cwd None) > > Thread-801::DEBUG::2015-09-22 > 19:12:35,946::utils::679::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 > > Thread-801::DEBUG::2015-09-22 > 19:12:35,947::lvm::498::Storage.OperationMutex::(_invalidateAllPvs) > Operation 'lvm invalidate operation' got the operation mutex > > Thread-801::DEBUG::2015-09-22 > 19:12:35,947::lvm::500::Storage.OperationMutex::(_invalidateAllPvs) > Operation 'lvm invalidate operation' released the operation mutex > > Thread-801::DEBUG::2015-09-22 > 19:12:35,947::lvm::509::Storage.OperationMutex::(_invalidateAllVgs) > Operation 'lvm invalidate operation' got the operation mutex > > Thread-801::DEBUG::2015-09-22 > 19:12:35,948::lvm::511::Storage.OperationMutex::(_invalidateAllVgs) > Operation 'lvm invalidate operation' released the operation mutex > > Thread-801::DEBUG::2015-09-22 > 19:12:35,948::lvm::529::Storage.OperationMutex::(_invalidateAllLvs) > Operation 'lvm invalidate operation' got the operation mutex > > Thread-801::DEBUG::2015-09-22 > 19:12:35,948::lvm::531::Storage.OperationMutex::(_invalidateAllLvs) > Operation 'lvm invalidate operation' released the operation mutex > > Thread-801::DEBUG::2015-09-22 > 19:12:35,948::misc::743::Storage.SamplingMethod::(__call__) Returning > last result > > Thread-801::ERROR::2015-09-22 > 19:12:35,949::sdc::138::Storage.StorageDomainCache::(_findDomain) > looking for unfetched domain c02fda97-62e3-40d3-9a6e-ac5d100f8ad3 > > Thread-801::ERROR::2015-09-22 > 19:12:35,949::sdc::155::Storage.StorageDomainCache::(_findUnfetchedDomain) > looking for domain c02fda97-62e3-40d3-9a6e-ac5d100f8ad3 > > Thread-801::DEBUG::2015-09-22 > 19:12:35,949::lvm::371::Storage.OperationMutex::(_reloadvgs) Operation > 'lvm reload operation' got the operation mutex > > Thread-801::DEBUG::2015-09-22 > 19:12:35,950::lvm::291::Storage.Misc.excCmd::(cmd) /usr/bin/sudo -n > /usr/sbin/lvm vgs --config ' devices { preferred_names = > ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 > disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ > '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 > wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 > retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' > --ignoreskippedcluster -o > uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name > c02fda97-62e3-40d3-9a6e-ac5d100f8ad3 (cwd None) > > Thread-801::DEBUG::2015-09-22 > 19:12:35,985::lvm::291::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' > WARNING: lvmetad is running but disabled. Restart lvmetad before > enabling it!\n Volume group "c02fda97-62e3-40d3-9a6e-ac5d100f8ad3" > not found\n Cannot process volume group > c02fda97-62e3-40d3-9a6e-ac5d100f8ad3\n'; <rc> = 5 > > Thread-801::WARNING::2015-09-22 > 19:12:35,986::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] > [' WARNING: lvmetad is running but disabled. Restart lvmetad before > enabling it!', ' Volume group "c02fda97-62e3-40d3-9a6e-ac5d100f8ad3" > not found', ' Cannot process volume group > c02fda97-62e3-40d3-9a6e-ac5d100f8ad3'] > > Thread-801::DEBUG::2015-09-22 > 19:12:35,987::lvm::416::Storage.OperationMutex::(_reloadvgs) Operation > 'lvm reload operation' released the operation mutex > > Thread-801::ERROR::2015-09-22 > 19:12:35,997::sdc::144::Storage.StorageDomainCache::(_findDomain) > domain c02fda97-62e3-40d3-9a6e-ac5d100f8ad3 not found > > Traceback (most recent call last): > > File "/usr/share/vdsm/storage/sdc.py", line 142, in _findDomain > > dom = findMethod(sdUUID) > > File "/usr/share/vdsm/storage/sdc.py", line 172, in _findUnfetchedDomain > > raise se.StorageDomainDoesNotExist(sdUUID) > > StorageDomainDoesNotExist: Storage domain does not exist: > (u'c02fda97-62e3-40d3-9a6e-ac5d100f8ad3',) > > Thread-801::INFO::2015-09-22 > 19:12:35,998::nfsSD::69::Storage.StorageDomain::(create) > sdUUID=c02fda97-62e3-40d3-9a6e-ac5d100f8ad3 domainName=sjcvmstore01 > remotePath=sjcstorage01:/vmstore domClass=1 > > Thread-801::DEBUG::2015-09-22 > 19:12:36,015::__init__::298::IOProcessClient::(_run) Starting IOProcess... > > Thread-801::ERROR::2015-09-22 > 19:12:36,067::task::866::Storage.TaskManager.Task::(_setError) > Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::Unexpected error > > Traceback (most recent call last): > > File "/usr/share/vdsm/storage/task.py", line 873, in _run > > return fn(*args, **kargs) > > File "/usr/share/vdsm/logUtils.py", line 49, in wrapper > > res = f(*args, **kwargs) > > File "/usr/share/vdsm/storage/hsm.py", line 2697, in createStorageDomain > > domVersion) > > File "/usr/share/vdsm/storage/nfsSD.py", line 84, in create > > remotePath, storageType, version) > > File "/usr/share/vdsm/storage/fileSD.py", line 264, in _prepareMetadata > > "create meta file '%s' failed: %s" % (metaFile, str(e))) > > StorageDomainMetadataCreationError: Error creating a storage domain's > metadata: ("create meta file 'outbox' failed: [Errno 5] Input/output > error",) > > Thread-801::DEBUG::2015-09-22 > 19:12:36,067::task::885::Storage.TaskManager.Task::(_run) > Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::Task._run: > d2d29352-8677-45cb-a4ab-06aa32cf1acb (7, > u'c02fda97-62e3-40d3-9a6e-ac5d100f8ad3', u'sjcvmstore01', > u'sjcstorage01:/vmstore', 1, u'3') {} failed - stopping task > > Thread-801::DEBUG::2015-09-22 > 19:12:36,067::task::1246::Storage.TaskManager.Task::(stop) > Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::stopping in state > preparing (force False) > > Thread-801::DEBUG::2015-09-22 > 19:12:36,067::task::993::Storage.TaskManager.Task::(_decref) > Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::ref 1 aborting True > > Thread-801::INFO::2015-09-22 > 19:12:36,067::task::1171::Storage.TaskManager.Task::(prepare) > Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::aborting: Task is > aborted: "Error creating a storage domain's metadata" - code 362 > > Thread-801::DEBUG::2015-09-22 > 19:12:36,068::task::1176::Storage.TaskManager.Task::(prepare) > Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::Prepare: aborted: Error > creating a storage domain's metadata > > Thread-801::DEBUG::2015-09-22 > 19:12:36,068::task::993::Storage.TaskManager.Task::(_decref) > Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::ref 0 aborting True > > Thread-801::DEBUG::2015-09-22 > 19:12:36,068::task::928::Storage.TaskManager.Task::(_doAbort) > Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::Task._doAbort: force False > > Thread-801::DEBUG::2015-09-22 > 19:12:36,068::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) > Owner.cancelAll requests {} > > Thread-801::DEBUG::2015-09-22 > 19:12:36,068::task::595::Storage.TaskManager.Task::(_updateState) > Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::moving from state > preparing -> state aborting > > Thread-801::DEBUG::2015-09-22 > 19:12:36,068::task::550::Storage.TaskManager.Task::(__state_aborting) > Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::_aborting: recover policy > none > > Thread-801::DEBUG::2015-09-22 > 19:12:36,068::task::595::Storage.TaskManager.Task::(_updateState) > Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::moving from state > aborting -> state failed > > Thread-801::DEBUG::2015-09-22 > 19:12:36,068::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) > Owner.releaseAll requests {} resources {} > > Thread-801::DEBUG::2015-09-22 > 19:12:36,068::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) > Owner.cancelAll requests {} > > Thread-801::ERROR::2015-09-22 > 19:12:36,068::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': > {'message': 'Error creating a storage domain\'s metadata: ("create > meta file \'outbox\' failed: [Errno 5] Input/output error",)', 'code': > 362}} > > Thread-801::DEBUG::2015-09-22 > 19:12:36,069::stompreactor::304::yajsonrpc.StompServer::(send) Sending > response > > Thread-807::DEBUG::2015-09-22 > 19:12:36,180::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) > Calling 'StoragePool.disconnectStorageServer' in bridge with > {u'connectionParams': [{u'id': > u'cd55e6a1-022a-4b32-8a94-cab506a9b054', u'connection': > u'sjcstorage01:/vmstore', u'iqn': u'', u'user': u'', u'tpgt': u'1', > u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], > u'storagepoolID': u'00000000-0000-0000-0000-000000000000', > u'domainType': 7} > > Thread-807::DEBUG::2015-09-22 > 19:12:36,181::task::595::Storage.TaskManager.Task::(_updateState) > Task=`01af6594-9c7b-4ec7-b08f-02627db8f421`::moving from state init -> > state preparing > > Thread-807::INFO::2015-09-22 > 19:12:36,182::logUtils::48::dispatcher::(wrapper) Run and protect: > disconnectStorageServer(domType=7, > spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'id': > u'cd55e6a1-022a-4b32-8a94-cab506a9b054', u'connection': > u'sjcstorage01:/vmstore', u'iqn': u'', u'user': u'', u'tpgt': u'1', > u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], > options=None) > > Thread-807::DEBUG::2015-09-22 > 19:12:36,182::mount::229::Storage.Misc.excCmd::(_runcmd) /usr/bin/sudo > -n /usr/bin/umount -f -l > /rhev/data-center/mnt/glusterSD/sjcstorage01:_vmstore (cwd None) > > Thread-807::DEBUG::2015-09-22 > 19:12:36,222::misc::733::Storage.SamplingMethod::(__call__) Trying to > enter sampling method (storage.sdc.refreshStorage) > > Thread-807::DEBUG::2015-09-22 > 19:12:36,222::misc::736::Storage.SamplingMethod::(__call__) Got in to > sampling method > > Thread-807::DEBUG::2015-09-22 > 19:12:36,222::misc::733::Storage.SamplingMethod::(__call__) Trying to > enter sampling method (storage.iscsi.rescan) > > Thread-807::DEBUG::2015-09-22 > 19:12:36,222::misc::736::Storage.SamplingMethod::(__call__) Got in to > sampling method > > Thread-807::DEBUG::2015-09-22 > 19:12:36,223::iscsi::431::Storage.ISCSI::(rescan) Performing SCSI > scan, this will take up to 30 seconds > > Thread-807::DEBUG::2015-09-22 > 19:12:36,223::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) > /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None) > > Thread-807::DEBUG::2015-09-22 > 19:12:36,258::misc::743::Storage.SamplingMethod::(__call__) Returning > last result > > Thread-807::DEBUG::2015-09-22 > 19:12:36,258::misc::733::Storage.SamplingMethod::(__call__) Trying to > enter sampling method (storage.hba.rescan) > > Thread-807::DEBUG::2015-09-22 > 19:12:36,258::misc::736::Storage.SamplingMethod::(__call__) Got in to > sampling method > > Thread-807::DEBUG::2015-09-22 > 19:12:36,258::hba::56::Storage.HBA::(rescan) Starting scan > > Thread-807::DEBUG::2015-09-22 > 19:12:36,350::hba::62::Storage.HBA::(rescan) Scan finished > > Thread-807::DEBUG::2015-09-22 > 19:12:36,350::misc::743::Storage.SamplingMethod::(__call__) Returning > last result > > Thread-807::DEBUG::2015-09-22 > 19:12:36,350::multipath::77::Storage.Misc.excCmd::(rescan) > /usr/bin/sudo -n /usr/sbin/multipath (cwd None) > > Thread-807::DEBUG::2015-09-22 > 19:12:36,374::multipath::77::Storage.Misc.excCmd::(rescan) SUCCESS: > <err> = ''; <rc> = 0 > > Thread-807::DEBUG::2015-09-22 > 19:12:36,374::utils::661::root::(execCmd) /sbin/udevadm settle > --timeout=5 (cwd None) > > Thread-807::DEBUG::2015-09-22 > 19:12:36,383::utils::679::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 > > Thread-807::DEBUG::2015-09-22 > 19:12:36,384::lvm::498::Storage.OperationMutex::(_invalidateAllPvs) > Operation 'lvm invalidate operation' got the operation mutex > > Thread-807::DEBUG::2015-09-22 > 19:12:36,385::lvm::500::Storage.OperationMutex::(_invalidateAllPvs) > Operation 'lvm invalidate operation' released the operation mutex > > Thread-807::DEBUG::2015-09-22 > 19:12:36,385::lvm::509::Storage.OperationMutex::(_invalidateAllVgs) > Operation 'lvm invalidate operation' got the operation mutex > > Thread-807::DEBUG::2015-09-22 > 19:12:36,385::lvm::511::Storage.OperationMutex::(_invalidateAllVgs) > Operation 'lvm invalidate operation' released the operation mutex > > Thread-807::DEBUG::2015-09-22 > 19:12:36,386::lvm::529::Storage.OperationMutex::(_invalidateAllLvs) > Operation 'lvm invalidate operation' got the operation mutex > > Thread-807::DEBUG::2015-09-22 > 19:12:36,386::lvm::531::Storage.OperationMutex::(_invalidateAllLvs) > Operation 'lvm invalidate operation' released the operation mutex > > Thread-807::DEBUG::2015-09-22 > 19:12:36,386::misc::743::Storage.SamplingMethod::(__call__) Returning > last result > > Thread-807::INFO::2015-09-22 > 19:12:36,386::logUtils::51::dispatcher::(wrapper) Run and protect: > disconnectStorageServer, Return response: {'statuslist': [{'status': > 0, 'id': u'cd55e6a1-022a-4b32-8a94-cab506a9b054'}]} > > Thread-807::DEBUG::2015-09-22 > 19:12:36,387::task::1191::Storage.TaskManager.Task::(prepare) > Task=`01af6594-9c7b-4ec7-b08f-02627db8f421`::finished: {'statuslist': > [{'status': 0, 'id': u'cd55e6a1-022a-4b32-8a94-cab506a9b054'}]} > > Thread-807::DEBUG::2015-09-22 > 19:12:36,387::task::595::Storage.TaskManager.Task::(_updateState) > Task=`01af6594-9c7b-4ec7-b08f-02627db8f421`::moving from state > preparing -> state finished > > Thread-807::DEBUG::2015-09-22 > 19:12:36,387::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) > Owner.releaseAll requests {} resources {} > > Thread-807::DEBUG::2015-09-22 > 19:12:36,387::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) > Owner.cancelAll requests {} > > Thread-807::DEBUG::2015-09-22 > 19:12:36,387::task::993::Storage.TaskManager.Task::(_decref) > Task=`01af6594-9c7b-4ec7-b08f-02627db8f421`::ref 0 aborting False > > Thread-807::DEBUG::2015-09-22 > 19:12:36,388::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) > Return 'StoragePool.disconnectStorageServer' in bridge with > [{'status': 0, 'id': u'cd55e6a1-022a-4b32-8a94-cab506a9b054'}] > > Thread-807::DEBUG::2015-09-22 > 19:12:36,388::stompreactor::304::yajsonrpc.StompServer::(send) Sending > response > > Thread-808::DEBUG::2015-09-22 > 19:12:37,868::task::595::Storage.TaskManager.Task::(_updateState) > Task=`adddaa68-dd1d-4d2e-9853-b7894ee4809c`::moving from state init -> > state preparing > > Thread-808::INFO::2015-09-22 > 19:12:37,868::logUtils::48::dispatcher::(wrapper) Run and protect: > repoStats(options=None) > > Thread-808::INFO::2015-09-22 > 19:12:37,868::logUtils::51::dispatcher::(wrapper) Run and protect: > repoStats, Return response: {} > > Thread-808::DEBUG::2015-09-22 > 19:12:37,868::task::1191::Storage.TaskManager.Task::(prepare) > Task=`adddaa68-dd1d-4d2e-9853-b7894ee4809c`::finished: {} > > Thread-808::DEBUG::2015-09-22 > 19:12:37,868::task::595::Storage.TaskManager.Task::(_updateState) > Task=`adddaa68-dd1d-4d2e-9853-b7894ee4809c`::moving from state > preparing -> state finished > > Thread-808::DEBUG::2015-09-22 > 19:12:37,868::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) > Owner.releaseAll requests {} resources {} > > Thread-808::DEBUG::2015-09-22 > 19:12:37,868::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) > Owner.cancelAll requests {} > > Thread-808::DEBUG::2015-09-22 > 19:12:37,868::task::993::Storage.TaskManager.Task::(_decref) > Task=`adddaa68-dd1d-4d2e-9853-b7894ee4809c`::ref 0 aborting False > > Thread-808::DEBUG::2015-09-22 > 19:12:37,873::stompreactor::304::yajsonrpc.StompServer::(send) Sending > response > > Reactor thread::INFO::2015-09-22 > 19:12:44,867::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) > Accepting connection from 127.0.0.1:52512 <http://127.0.0.1:52512> > > Reactor thread::DEBUG::2015-09-22 > 19:12:44,874::protocoldetector::82::ProtocolDetector.Detector::(__init__) > Using required_size=11 > > Reactor thread::INFO::2015-09-22 > 19:12:44,875::protocoldetector::118::ProtocolDetector.Detector::(handle_read) > Detected protocol xml from 127.0.0.1:52512 <http://127.0.0.1:52512> > > Reactor thread::DEBUG::2015-09-22 > 19:12:44,875::bindingxmlrpc::1297::XmlDetector::(handle_socket) xml > over http detected from ('127.0.0.1', 52512) > > BindingXMLRPC::INFO::2015-09-22 > 19:12:44,875::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting > request handler for 127.0.0.1:52512 <http://127.0.0.1:52512> > > Thread-809::INFO::2015-09-22 > 19:12:44,876::xmlrpc::84::vds.XMLRPCServer::(_process_requests) > Request handler for 127.0.0.1:52512 <http://127.0.0.1:52512> started > > Thread-809::INFO::2015-09-22 > 19:12:44,877::xmlrpc::92::vds.XMLRPCServer::(_process_requests) > Request handler for 127.0.0.1:52512 <http://127.0.0.1:52512> stopped > > Thread-810::DEBUG::2015-09-22 > 19:12:50,889::stompreactor::304::yajsonrpc.StompServer::(send) Sending > response > > Thread-811::DEBUG::2015-09-22 > 19:12:52,902::task::595::Storage.TaskManager.Task::(_updateState) > Task=`d9fb30bc-dff3-4df3-a25e-2ad09a940fde`::moving from state init -> > state preparing > > Thread-811::INFO::2015-09-22 > 19:12:52,902::logUtils::48::dispatcher::(wrapper) Run and protect: > repoStats(options=None) > > Thread-811::INFO::2015-09-22 > 19:12:52,902::logUtils::51::dispatcher::(wrapper) Run and protect: > repoStats, Return response: {} > > Thread-811::DEBUG::2015-09-22 > 19:12:52,902::task::1191::Storage.TaskManager.Task::(prepare) > Task=`d9fb30bc-dff3-4df3-a25e-2ad09a940fde`::finished: {} > > Thread-811::DEBUG::2015-09-22 > 19:12:52,903::task::595::Storage.TaskManager.Task::(_updateState) > Task=`d9fb30bc-dff3-4df3-a25e-2ad09a940fde`::moving from state > preparing -> state finished > > Thread-811::DEBUG::2015-09-22 > 19:12:52,903::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) > Owner.releaseAll requests {} resources {} > > Thread-811::DEBUG::2015-09-22 > 19:12:52,903::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) > Owner.cancelAll requests {} > > Thread-811::DEBUG::2015-09-22 > 19:12:52,903::task::993::Storage.TaskManager.Task::(_decref) > Task=`d9fb30bc-dff3-4df3-a25e-2ad09a940fde`::ref 0 aborting False > > Thread-811::DEBUG::2015-09-22 > 19:12:52,908::stompreactor::304::yajsonrpc.StompServer::(send) Sending > response > > Reactor thread::INFO::2015-09-22 > 19:12:59,895::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) > Accepting connection from 127.0.0.1:52513 <http://127.0.0.1:52513> > > Reactor thread::DEBUG::2015-09-22 > 19:12:59,902::protocoldetector::82::ProtocolDetector.Detector::(__init__) > Using required_size=11 > > Reactor thread::INFO::2015-09-22 > 19:12:59,902::protocoldetector::118::ProtocolDetector.Detector::(handle_read) > Detected protocol xml from 127.0.0.1:52513 <http://127.0.0.1:52513> > > Reactor thread::DEBUG::2015-09-22 > 19:12:59,902::bindingxmlrpc::1297::XmlDetector::(handle_socket) xml > over http detected from ('127.0.0.1', 52513) > > BindingXMLRPC::INFO::2015-09-22 > 19:12:59,903::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting > request handler for 127.0.0.1:52513 <http://127.0.0.1:52513> > > Thread-812::INFO::2015-09-22 > 19:12:59,903::xmlrpc::84::vds.XMLRPCServer::(_process_requests) > Request handler for 127.0.0.1:52513 <http://127.0.0.1:52513> started > > Thread-812::INFO::2015-09-22 > 19:12:59,904::xmlrpc::92::vds.XMLRPCServer::(_process_requests) > Request handler for 127.0.0.1:52513 <http://127.0.0.1:52513> stopped > > Thread-813::DEBUG::2015-09-22 > 19:13:05,898::stompreactor::304::yajsonrpc.StompServer::(send) Sending > response > > Thread-814::DEBUG::2015-09-22 > 19:13:07,934::task::595::Storage.TaskManager.Task::(_updateState) > Task=`c526c56c-6254-4497-9c3e-ffe09ed54af2`::moving from state init -> > state preparing > > Thread-814::INFO::2015-09-22 > 19:13:07,935::logUtils::48::dispatcher::(wrapper) Run and protect: > repoStats(options=None) > > Thread-814::INFO::2015-09-22 > 19:13:07,935::logUtils::51::dispatcher::(wrapper) Run and protect: > repoStats, Return response: {} > > Thread-814::DEBUG::2015-09-22 > 19:13:07,935::task::1191::Storage.TaskManager.Task::(prepare) > Task=`c526c56c-6254-4497-9c3e-ffe09ed54af2`::finished: {} > > Thread-814::DEBUG::2015-09-22 > 19:13:07,935::task::595::Storage.TaskManager.Task::(_updateState) > Task=`c526c56c-6254-4497-9c3e-ffe09ed54af2`::moving from state > preparing -> state finished > > Thread-814::DEBUG::2015-09-22 > 19:13:07,935::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) > Owner.releaseAll requests {} resources {} > > Thread-814::DEBUG::2015-09-22 > 19:13:07,935::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) > Owner.cancelAll requests {} > > Thread-814::DEBUG::2015-09-22 > 19:13:07,935::task::993::Storage.TaskManager.Task::(_decref) > Task=`c526c56c-6254-4497-9c3e-ffe09ed54af2`::ref 0 aborting False > > Thread-814::DEBUG::2015-09-22 > 19:13:07,939::stompreactor::304::yajsonrpc.StompServer::(send) Sending > response > > Reactor thread::INFO::2015-09-22 > 19:13:14,921::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) > Accepting connection from 127.0.0.1:52515 <http://127.0.0.1:52515> > > Reactor thread::DEBUG::2015-09-22 > 19:13:14,927::protocoldetector::82::ProtocolDetector.Detector::(__init__) > Using required_size=11 > > Reactor thread::INFO::2015-09-22 > 19:13:14,928::protocoldetector::118::ProtocolDetector.Detector::(handle_read) > Detected protocol xml from 127.0.0.1:52515 <http://127.0.0.1:52515> > > Reactor thread::DEBUG::2015-09-22 > 19:13:14,928::bindingxmlrpc::1297::XmlDetector::(handle_socket) xml > over http detected from ('127.0.0.1', 52515) > > BindingXMLRPC::INFO::2015-09-22 > 19:13:14,928::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting > request handler for 127.0.0.1:52515 <http://127.0.0.1:52515> > > Thread-815::INFO::2015-09-22 > 19:13:14,928::xmlrpc::84::vds.XMLRPCServer::(_process_requests) > Request handler for 127.0.0.1:52515 <http://127.0.0.1:52515> started > > Thread-815::INFO::2015-09-22 > 19:13:14,930::xmlrpc::92::vds.XMLRPCServer::(_process_requests) > Request handler for 127.0.0.1:52515 <http://127.0.0.1:52515> stopped > > Thread-816::DEBUG::2015-09-22 > 19:13:20,906::stompreactor::304::yajsonrpc.StompServer::(send) Sending > response > > > > gluster logs > > +------------------------------------------------------------------------------+ > > 1: volume vmstore-client-0 > > 2: type protocol/client > > 3: option ping-timeout 42 > > 4: option remote-host sjcstorage01 > > 5: option remote-subvolume /export/vmstore/brick01 > > 6: option transport-type socket > > 7: option send-gids true > > 8: end-volume > > 9: > > 10: volume vmstore-client-1 > > 11: type protocol/client > > 12: option ping-timeout 42 > > 13: option remote-host sjcstorage02 > > 14: option remote-subvolume /export/vmstore/brick01 > > 15: option transport-type socket > > 16: option send-gids true > > 17: end-volume > > 18: > > 19: volume vmstore-client-2 > > 20: type protocol/client > > 21: option ping-timeout 42 > > 22: option remote-host sjcvhost02 > > 23: option remote-subvolume /export/vmstore/brick01 > > 24: option transport-type socket > > 25: option send-gids true > > 26: end-volume > > 27: > > 28: volume vmstore-replicate-0 > > 29: type cluster/replicate > > 30: option arbiter-count 1 > > 31: subvolumes vmstore-client-0 vmstore-client-1 vmstore-client-2 > > 32: end-volume > > 33: > > 34: volume vmstore-dht > > 35: type cluster/distribute > > 36: subvolumes vmstore-replicate-0 > > 37: end-volume > > 38: > > 39: volume vmstore-write-behind > > 40: type performance/write-behind > > 41: subvolumes vmstore-dht > > 42: end-volume > > 43: > > 44: volume vmstore-read-ahead > > 45: type performance/read-ahead > > 46: subvolumes vmstore-write-behind > > 47: end-volume > > 48: > > 49: volume vmstore-readdir-ahead > > 50: type performance/readdir-ahead > > 52: end-volume > > 53: > > 54: volume vmstore-io-cache > > 55: type performance/io-cache > > 56: subvolumes vmstore-readdir-ahead > > 57: end-volume > > 58: > > 59: volume vmstore-quick-read > > 60: type performance/quick-read > > 61: subvolumes vmstore-io-cache > > 62: end-volume > > 63: > > 64: volume vmstore-open-behind > > 65: type performance/open-behind > > 66: subvolumes vmstore-quick-read > > 67: end-volume > > 68: > > 69: volume vmstore-md-cache > > 70: type performance/md-cache > > 71: subvolumes vmstore-open-behind > > 72: end-volume > > 73: > > 74: volume vmstore > > 75: type debug/io-stats > > 76: option latency-measurement off > > 77: option count-fop-hits off > > 78: subvolumes vmstore-md-cache > > 79: end-volume > > 80: > > 81: volume meta-autoload > > 82: type meta > > 83: subvolumes vmstore > > 84: end-volume > > 85: > > +------------------------------------------------------------------------------+ > > [2015-09-22 05:29:07.586205] I [rpc-clnt.c:1851:rpc_clnt_reconfig] > 0-vmstore-client-0: changing port to 49153 (from 0) > > [2015-09-22 05:29:07.586325] I [rpc-clnt.c:1851:rpc_clnt_reconfig] > 0-vmstore-client-1: changing port to 49153 (from 0) > > [2015-09-22 05:29:07.586480] I [rpc-clnt.c:1851:rpc_clnt_reconfig] > 0-vmstore-client-2: changing port to 49153 (from 0) > > [2015-09-22 05:29:07.595052] I [MSGID: 114057] > [client-handshake.c:1437:select_server_supported_programs] > 0-vmstore-client-0: Using Program GlusterFS 3.3, Num (1298437), > Version (330) > > [2015-09-22 05:29:07.595397] I [MSGID: 114057] > [client-handshake.c:1437:select_server_supported_programs] > 0-vmstore-client-1: Using Program GlusterFS 3.3, Num (1298437), > Version (330) > > [2015-09-22 05:29:07.595576] I [MSGID: 114057] > [client-handshake.c:1437:select_server_supported_programs] > 0-vmstore-client-2: Using Program GlusterFS 3.3, Num (1298437), > Version (330) > > [2015-09-22 05:29:07.595721] I [MSGID: 114046] > [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-0: > Connected to vmstore-client-0, attached to remote volume > '/export/vmstore/brick01'. > > [2015-09-22 05:29:07.595738] I [MSGID: 114047] > [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-0: > Server and Client lk-version numbers are not same, reopening the fds > > [2015-09-22 05:29:07.596044] I [MSGID: 108005] > [afr-common.c:3998:afr_notify] 0-vmstore-replicate-0: Subvolume > 'vmstore-client-0' came back up; going online. > > [2015-09-22 05:29:07.596170] I [MSGID: 114046] > [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-1: > Connected to vmstore-client-1, attached to remote volume > '/export/vmstore/brick01'. > > [2015-09-22 05:29:07.596189] I [MSGID: 114047] > [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-1: > Server and Client lk-version numbers are not same, reopening the fds > > [2015-09-22 05:29:07.596495] I [MSGID: 114046] > [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-2: > Connected to vmstore-client-2, attached to remote volume : > > [2015-09-22 05:29:07.596189] I [MSGID: 114047] > [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-1: > Server and Client lk-version numbers are not same, reopening the fds > > [2015-09-22 05:29:07.596495] I [MSGID: 114046] > [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-2: > Connected to vmstore-client-2, attached to remote volume > '/export/vmstore/brick01'. > > [2015-09-22 05:29:07.596506] I [MSGID: 114047] > [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-2: > Server and Client lk-version numbers are not same, reopening the fds > > [2015-09-22 05:29:07.608758] I [fuse-bridge.c:5053:fuse_graph_setup] > 0-fuse: switched to graph 0 > > [2015-09-22 05:29:07.608910] I [MSGID: 114035] > [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-0: > Server lk version = 1 > > [2015-09-22 05:29:07.608936] I [MSGID: 114035] > [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-1: > Server lk version = 1 > > [2015-09-22 05:29:07.608950] I [MSGID: 114035] > [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-2: > Server lk version = 1 > > [2015-09-22 05:29:07.609695] I [MSGID: 101190] > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started > thread with index 2 > > [2015-09-22 05:29:07.609868] I [fuse-bridge.c:3979:fuse_init] > 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 > kernel 7.22 > > [2015-09-22 05:29:07.616577] I [MSGID: 109063] > [dht-layout.c:702:dht_layout_normalize] 0-vmstore-dht: Found anomalies > in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0 > > [2015-09-22 05:29:07.620230] I [MSGID: 109036] > [dht-common.c:7754:dht_log_new_layout_for_dir_selfheal] 0-vmstore-dht: > Setting layout of / with [Subvol_name: vmstore-replicate-0, Err: -1 , > Start: 0 , Stop: 4294967295 , Hash: 1 ], > > [2015-09-22 05:29:08.122415] W [fuse-bridge.c:1230:fuse_err_cbk] > 0-glusterfs-fuse: 26: REMOVEXATTR() /__DIRECT_IO_TEST__ => -1 (No data > available) > > [2015-09-22 05:29:08.137359] I [MSGID: 109036] > [dht-common.c:7754:dht_log_new_layout_for_dir_selfheal] 0-vmstore-dht: > Setting layout of /061b73d5-ae59-462e-b674-ea9c60d436c2 with > [Subvol_name: vmstore-replicate-0, Err: -1 , Start: 0 , Stop: > 4294967295 , Hash: 1 ], > > [2015-09-22 05:29:08.145835] I [MSGID: 109036] > [dht-common.c:7754:dht_log_new_layout_for_dir_selfheal] 0-vmstore-dht: > Setting layout of /061b73d5-ae59-462e-b674-ea9c60d436c2/dom_md with > [Subvol_name: vmstore-replicate-0, Err: -1 , Start: 0 , Stop: > 4294967295 , Hash: 1 ], > > [2015-09-22 05:30:57.897819] I [MSGID: 100030] > [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running > /usr/sbin/glusterfs version 3.7.4 (args: /usr/sbin/glusterfs > --volfile-server=sjcvhost02 --volfile-server=sjcstorage01 > --volfile-server=sjcstorage02 --volfile-id=/vmstore > /rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore) > > [2015-09-22 05:30:57.909889] I [MSGID: 101190] > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started > thread with index 1 > > [2015-09-22 05:30:57.923087] I [MSGID: 114020] [client.c:2118:notify] > 0-vmstore-client-0: parent translators are ready, attempting connect > on transport > > [2015-09-22 05:30:57.925701] I [MSGID: 114020] [client.c:2118:notify] > 0-vmstore-client-1: parent translators are ready, attempting connect > on transport > > [2015-09-22 05:30:57.927984] I [MSGID: 114020] [client.c:2118:notify] > 0-vmstore-client-2: parent translators are ready, attempting connect > on transport > > Final graph: > > +------------------------------------------------------------------------------+ > > 1: volume vmstore-client-0 > > 2: type protocol/client > > 3: option ping-timeout 42 > > 4: option remote-host sjcstorage01 > > 5: option remote-subvolume /export/vmstore/brick01 > > 6: option transport-type socket > > 7: option send-gids true > > 8: end-volume > > 9: > > 10: volume vmstore-client-1 > > 11: type protocol/client > > 12: option ping-timeout 42 > > 13: option remote-host sjcstorage02 > > 14: option remote-subvolume /export/vmstore/brick01 > > 15: option transport-type socket > > 16: option send-gids true > > 17: end-volume > > 18: > > 19: volume vmstore-client-2 > > 20: type protocol/client > > 21: option ping-timeout 42 > > 22: option remote-host sjcvhost02 > > 23: option remote-subvolume /export/vmstore/brick01 > > 24: option transport-type socket > > 25: option send-gids true > > 26: end-volume > > 27: > > 28: volume vmstore-replicate-0 > > 29: type cluster/replicate > > 30: option arbiter-count 1 > > 31: subvolumes vmstore-client-0 vmstore-client-1 vmstore-client-2 > > 32: end-volume > > 33: > > 34: volume vmstore-dht > > 35: type cluster/distribute > > 36: subvolumes vmstore-replicate-0 > > 37: end-volume > > 38: > > 39: volume vmstore-write-behind > > 40: type performance/write-behind > > 41: subvolumes vmstore-dht > > 42: end-volume > > 43: > > 44: volume vmstore-read-ahead > > 45: type performance/read-ahead > > 46: subvolumes vmstore-write-behind > > 47: end-volume > > 48: > > 49: volume vmstore-readdir-ahead > > 50: type performance/readdir-ahead > > 51: subvolumes vmstore-read-ahead > > 52: end-volume > > 53: > > 54: volume vmstore-io-cache > > 55: type performance/io-cache > > 56: subvolumes vmstore-readdir-ahead > > 57: end-volume > > 58: > > 59: volume vmstore-quick-read > > 60: type performance/quick-read > > 61: subvolumes vmstore-io-cache > > 62: end-volume > > 63: > > 64: volume vmstore-open-behind > > 65: type performance/open-behind > > 66: subvolumes vmstore-quick-read > > 67: end-volume > > 68: > > 69: volume vmstore-md-cache > > 70: type performance/md-cache > > 71: subvolumes vmstore-open-behind > > 72: end-volume > > 73: > > 74: volume vmstore > > 75: type debug/io-stats > > 76: option latency-measurement off > > 77: option count-fop-hits off > > 78: subvolumes vmstore-md-cache > > 79: end-volume > > 80: > > 81: volume meta-autoload > > 82: type meta > > 83: subvolumes vmstore > > 84: end-volume > > 85: > > +------------------------------------------------------------------------------+ > > [2015-09-22 05:30:57.934021] I [rpc-clnt.c:1851:rpc_clnt_reconfig] > 0-vmstore-client-0: changing port to 49153 (from 0) > > [2015-09-22 05:30:57.934145] I [rpc-clnt.c:1851:rpc_clnt_reconfig] > 0-vmstore-client-1: changing port to 49153 (from 0) > > [2015-09-22 05:30:57.934491] I [rpc-clnt.c:1851:rpc_clnt_reconfig] > 0-vmstore-client-2: changing port to 49153 (from 0) > > [2015-09-22 05:30:57.942198] I [MSGID: 114057] > [client-handshake.c:1437:select_server_supported_programs] > 0-vmstore-client-0: Using Program GlusterFS 3.3, Num (1298437), > Version (330) > > [2015-09-22 05:30:57.942545] I [MSGID: 114057] > [client-handshake.c:1437:select_server_supported_programs] > 0-vmstore-client-1: Using Program GlusterFS 3.3, Num (1298437), > Version (330) > > [2015-09-22 05:30:57.942659] I [MSGID: 114057] > [client-handshake.c:1437:select_server_supported_programs] > 0-vmstore-client-2: Using Program GlusterFS 3.3, Num (1298437), > Version (330) > > [2015-09-22 05:30:57.942797] I [MSGID: 114046] > [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-0: > Connected to vmstore-client-0, attached to remote volume > '/export/vmstore/brick01'. > > [2015-09-22 05:30:57.942808] I [MSGID: 114047] > [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-0: > Server and Client lk-version numbers are not same, reopening the fds > > [2015-09-22 05:30:57.943036] I [MSGID: 108005] > [afr-common.c:3998:afr_notify] 0-vmstore-replicate-0: Subvolume > 'vmstore-client-0' came back up; going online. > > [2015-09-22 05:30:57.943078] I [MSGID: 114046] > [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-1: > Connected to vmstore-client-1, attached to remote volume > '/export/vmstore/brick01'. > > [2015-09-22 05:30:57.943086] I [MSGID: 114047] > [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-1: > Server and Client lk-version numbers are not same, reopening the fds > > [2015-09-22 05:30:57.943292] I [MSGID: 114046] > [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-2: > Connected to vmstore-client-2, attached to remote volume > '/export/vmstore/brick01'. > > [2015-09-22 05:30:57.943302] I [MSGID: 114047] > [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-2: > Server and Client lk-version numbers are not same, reopening the fds > > [2015-09-22 05:30:57.953887] I [fuse-bridge.c:5053:fuse_graph_setup] > 0-fuse: switched to graph 0 > > [2015-09-22 05:30:57.954071] I [MSGID: 114035] > [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-0: > Server lk version = 1 > > [2015-09-22 05:30:57.954105] I [MSGID: 114035] > [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-1: > Server lk version = 1 > > [2015-09-22 05:30:57.954124] I [MSGID: 114035] > [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-2: > Server lk version = 1 > > [2015-09-22 05:30:57.955282] I [fuse-bridge.c:3979:fuse_init] > 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 > kernel 7.22 > > [2015-09-22 05:30:57.955738] I [MSGID: 101190] > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started > thread with index 2 > > [2015-09-22 05:30:57.970232] I [fuse-bridge.c:4900:fuse_thread_proc] > 0-fuse: unmounting /rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore > > [2015-09-22 05:30:57.970834] W [glusterfsd.c:1219:cleanup_and_exit] > (-->/lib64/libpthread.so.0(+0x7df5) [0x7f187139fdf5] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f1872a09785] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x69) [0x7f1872a09609] ) 0-: > received signum (15), shutting down > > [2015-09-22 05:30:57.970848] I [fuse-bridge.c:5595:fini] 0-fuse: > Unmounting '/rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore'. > > [2015-09-22 05:30:58.420973] I [fuse-bridge.c:4900:fuse_thread_proc] > 0-fuse: unmounting /rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore > > [2015-09-22 05:30:58.421355] W [glusterfsd.c:1219:cleanup_and_exit] > (-->/lib64/libpthread.so.0(+0x7df5) [0x7f8267cd4df5] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f826933e785] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x69) [0x7f826933e609] ) 0-: > received signum (15), shutting down > > [2015-09-22 05:30:58.421369] I [fuse-bridge.c:5595:fini] 0-fuse: > Unmounting '/rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore'. > > [2015-09-22 05:31:09.534410] I [MSGID: 100030] > [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running > /usr/sbin/glusterfs version 3.7.4 (args: /usr/sbin/glusterfs > --volfile-server=sjcvhost02 --volfile-server=sjcstorage01 > --volfile-server=sjcstorage02 --volfile-id=/vmstore > /rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore) > > [2015-09-22 05:31:09.545686] I [MSGID: 101190] > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started > thread with index 1 > > [2015-09-22 05:31:09.553019] I [MSGID: 114020] [client.c:2118:notify] > 0-vmstore-client-0: parent translators are ready, attempting connect > on transport > > [2015-09-22 05:31:09.555552] I [MSGID: 114020] [client.c:2118:notify] > 0-vmstore-client-1: parent translators are ready, attempting connect > on transport > > [2015-09-22 05:31:09.557989] I [MSGID: 114020] [client.c:2118:notify] > 0-vmstore-client-2: parent translators are ready, attempting connect > on transport > > Final graph: > > +------------------------------------------------------------------------------+ > > 1: volume vmstore-client-0 > > 2: type protocol/client > > 3: option ping-timeout 42 > > 4: option remote-host sjcstorage01 > > 5: option remote-subvolume /export/vmstore/brick01 > > 6: option transport-type socket > > 7: option send-gids true > > 8: end-volume > > 9: > > 10: volume vmstore-client-1 > > 11: type protocol/client > > 12: option ping-timeout 42 > > 13: option remote-host sjcstorage02 > > 14: option remote-subvolume /export/vmstore/brick01 > > 15: option transport-type socket > > 16: option send-gids true > > 17: end-volume > > 18: > > 19: volume vmstore-client-2 > > 20: type protocol/client > > 21: option ping-timeout 42 > > 22: option remote-host sjcvhost02 > > 23: option remote-subvolume /export/vmstore/brick01 > > 24: option transport-type socket > > 25: option send-gids true > > 26: end-volume > > 27: > > 28: volume vmstore-replicate-0 > > 29: type cluster/replicate > > 30: option arbiter-count 1 > > 31: subvolumes vmstore-client-0 vmstore-client-1 vmstore-client-2 > > 32: end-volume > > 33: > > 34: volume vmstore-dht > > 35: type cluster/distribute > > 36: subvolumes vmstore-replicate-0 > > 37: end-volume > > 38: > > 39: volume vmstore-write-behind > > 40: type performance/write-behind > > 41: subvolumes vmstore-dht > > 42: end-volume > > 43: > > 44: volume vmstore-read-ahead > > 45: type performance/read-ahead > > 46: subvolumes vmstore-write-behind > > 47: end-volume > > 48: > > 49: volume vmstore-readdir-ahead > > 50: type performance/readdir-ahead > > 51: subvolumes vmstore-read-ahead > > 52: end-volume > > 53: > > 54: volume vmstore-io-cache > > 55: type performance/io-cache > > 56: subvolumes vmstore-readdir-ahead > > 57: end-volume > > 58: > > 59: volume vmstore-quick-read > > 60: type performance/quick-read > > 61: subvolumes vmstore-io-cache > > 62: end-volume > > 63: > > 64: volume vmstore-open-behind > > 65: type performance/open-behind > > 66: subvolumes vmstore-quick-read > > 67: end-volume > > 68: > > 69: volume vmstore-md-cache > > 70: type performance/md-cache > > 71: subvolumes vmstore-open-behind > > 72: end-volume > > 73: > > 74: volume vmstore > > 75: type debug/io-stats > > 76: option latency-measurement off > > 77: option count-fop-hits off > > 78: subvolumes vmstore-md-cache > > 79: end-volume > > 80: > > 81: volume meta-autoload > > 82: type meta > > 83: subvolumes vmstore > > 84: end-volume > > 85: > > +------------------------------------------------------------------------------+ > > [2015-09-22 05:31:09.563262] I [rpc-clnt.c:1851:rpc_clnt_reconfig] > 0-vmstore-client-0: changing port to 49153 (from 0) > > [2015-09-22 05:31:09.563431] I [rpc-clnt.c:1851:rpc_clnt_reconfig] > 0-vmstore-client-1: changing port to 49153 (from 0) > > [2015-09-22 05:31:09.563877] I [rpc-clnt.c:1851:rpc_clnt_reconfig] > 0-vmstore-client-2: changing port to 49153 (from 0) > > [2015-09-22 05:31:09.572443] I [MSGID: 114057] > [client-handshake.c:1437:select_server_supported_programs] > 0-vmstore-client-1: Using Program GlusterFS 3.3, Num (1298437), > Version (330) > > [2015-09-22 05:31:09.572599] I [MSGID: 114057] > [client-handshake.c:1437:select_server_supported_programs] > 0-vmstore-client-0: Using Program GlusterFS 3.3, Num (1298437), > Version (330) > > [2015-09-22 05:31:09.572742] I [MSGID: 114057] > [client-handshake.c:1437:select_server_supported_programs] > 0-vmstore-client-2: Using Program GlusterFS 3.3, Num (1298437), > Version (330) > > [2015-09-22 05:31:09.573165] I [MSGID: 114046] > [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-1: > Connected to vmstore-client-1, attached to remote volume > '/export/vmstore/brick01'. > > [2015-09-22 05:31:09.573186] I [MSGID: 114047] > [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-1: > Server and Client lk-version numbers are not same, reopening the fds > > [2015-09-22 05:31:09.573395] I [MSGID: 108005] > [afr-common.c:3998:afr_notify] 0-vmstore-replicate-0: Subvolume > 'vmstore-client-1' came back up; going online. > > [2015-09-22 05:31:09.573427] I [MSGID: 114046] > [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-0: > Connected to vmstore-client-0, attached to remote volume > '/export/vmstore/brick01'. > > [2015-09-22 05:31:09.573435] I [MSGID: 114047] > [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-0: > Server and Client lk-version numbers are not same, reopening the fds > > [2015-09-22 05:31:09.573754] I [MSGID: 114046] > [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-2: > Connected to vmstore-client-2, attached to remote volume > '/export/vmstore/brick01'. > > [2015-09-22 05:31:09.573783] I [MSGID: 114047] > [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-2: > Server and Client lk-version numbers are not same, reopen: > > [2015-09-22 05:31:09.577192] I [fuse-bridge.c:5053:fuse_graph_setup] > 0-fuse: switched to graph 0 > > [2015-09-22 05:31:09.577302] I [MSGID: 114035] > [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-1: > Server lk version = 1 > > [2015-09-22 05:31:09.577325] I [MSGID: 114035] > [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-0: > Server lk version = 1 > > [2015-09-22 05:31:09.577339] I [MSGID: 114035] > [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-2: > Server lk version = 1 > > [2015-09-22 05:31:09.578125] I [fuse-bridge.c:3979:fuse_init] > 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 > kernel 7.22 > > [2015-09-22 05:31:09.578636] I [MSGID: 101190] > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started > thread with index 2 > > [2015-09-22 05:31:10.073698] I [fuse-bridge.c:4900:fuse_thread_proc] > 0-fuse: unmounting /rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore > > [2015-09-22 05:31:10.073977] W [glusterfsd.c:1219:cleanup_and_exit] > (-->/lib64/libpthread.so.0(+0x7df5) [0x7f6b9ba88df5] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f6b9d0f2785] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x69) [0x7f6b9d0f2609] ) 0-: > received signum (15), shutting down > > [2015-09-22 05:31:10.073993] I [fuse-bridge.c:5595:fini] 0-fuse: > Unmounting '/rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore'. > > [2015-09-22 05:31:20.184700] I [MSGID: 100030] > [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running > /usr/sbin/glusterfs version 3.7.4 (args: /usr/sbin/glusterfs > --volfile-server=sjcvhost02 --volfile-server=sjcstorage01 > --volfile-server=sjcstorage02 --volfile-id=/vmstore > /rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore) > > [2015-09-22 05:31:20.194928] I [MSGID: 101190] > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started > thread with index 1 > > [2015-09-22 05:31:20.200701] I [MSGID: 114020] [client.c:2118:notify] > 0-vmstore-client-0: parent translators are ready, attempting connect > on transport > > [2015-09-22 05:31:20.203110] I [MSGID: 114020] [client.c:2118:notify] > 0-vmstore-client-1: parent translators are ready, attempting connect > on transport > > [2015-09-22 05:31:20.205708] I [MSGID: 114020] [client.c:2118:notify] > 0-vmstore-client-2: parent translators are ready, attempting connect > on transport > > Final graph: > > > > Hope this helps. > > > thanks again > > > Brett Stevens > > > > On Tue, Sep 22, 2015 at 10:14 PM, Sahina Bose <sabose(a)redhat.com > <mailto:sabose@redhat.com>> wrote: > > > > On 09/22/2015 02:17 PM, Brett Stevens wrote: >> Hi. First time on the lists. I've searched for this but no luck >> so sorry if this has been covered before. >> >> Im working with the latest 3.6 beta with the following >> infrastructure. >> >> 1 management host (to be used for a number of tasks so chose not >> to use self hosted, we are a school and will need to keep an eye >> on hardware costs) >> 2 compute nodes >> 2 gluster nodes >> >> so far built one gluster volume using the gluster cli to give me >> 2 nodes and one arbiter node (management host) >> >> so far, every time I create a volume, it shows up strait away on >> the ovirt gui. however no matter what I try, I cannot create or >> import it as a data domain. >> >> the current error in the ovirt gui is "Error while executing >> action AddGlusterFsStorageDomain: Error creating a storage >> domain's metadata" > > Please provide vdsm and gluster logs > >> >> logs, continuously rolling the following errors around >> >> Scheduler_Worker-53) [] START, >> GlusterVolumesListVDSCommand(HostName = sjcstorage02, >> GlusterVolumesListVDSParameters:{runAsync='true', >> hostId='c75682ba-1e4c-42a3-85c7-16e4bb2ce5da'}), log id: 24198fbf >> >> 2015-09-22 03:57:29,903 WARN >> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] >> (DefaultQuartzScheduler_Worker-53) [] Could not associate brick >> 'sjcstorage01:/export/vmstore/brick01' of volume >> '878a316d-2394-4aae-bdf8-e10eea38225e' with correct network as no >> gluster network found in cluster >> 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' >> > > What is the hostname provided in ovirt engine for sjcstorage01 ? > Does this host have multiple nics? > > Could you provide output of gluster volume info? > Please note, that these errors are not related to error in > creating storage domain. However, these errors could prevent you > from monitoring the state of gluster volume from oVirt > >> 2015-09-22 03:57:29,905 WARN >> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] >> (DefaultQuartzScheduler_Worker-53) [] Could not associate brick >> 'sjcstorage02:/export/vmstore/brick01' of volume >> '878a316d-2394-4aae-bdf8-e10eea38225e' with correct network as no >> gluster network found in cluster >> 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' >> >> 2015-09-22 03:57:29,905 WARN >> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] >> (DefaultQuartzScheduler_Worker-53) [] Could not add brick >> 'sjcvhost02:/export/vmstore/brick01' to volume >> '878a316d-2394-4aae-bdf8-e10eea38225e' - server uuid >> '29b58278-9aa3-47c5-bfb4-1948ef7fdbba' not found in cluster >> 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' >> >> 2015-09-22 03:57:29,905 INFO >> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] >> (DefaultQuartzScheduler_Worker-53) [] FINISH, >> GlusterVolumesListVDSCommand, return: >> {878a316d-2394-4aae-bdf8-e10eea38225e=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@41e93fb1}, >> log id: 24198fbf >> >> >> I'm new to ovirt and gluster, so any help would be great >> >> >> thanks >> >> >> Brett Stevens >> >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org <mailto:Users@ovirt.org> >> http://lists.ovirt.org/mailman/listinfo/users > > --------------090502070204020205010802 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> + ovirt-users Some clarity on your setup - sjcvhost03 - is this your arbiter node and ovirt management node? And are you running a compute + storage on the same nodes - i.e, sjcstorage01, sjcstorage02, sjcvhost03 (arbiter). CreateStorageDomainVDSCommand(HostName = sjcvhost03, CreateStorageDomainVDSCommandParameters:{runAsync='true', hostId='80245ac2-32a3-4d5d-b0fe-08019e2d1c9c', storageDomain='StorageDomainStatic:{name='sjcvmstore', id='597d5b5b-7c09-4de9-8840-6993bd9b61a6'}', args='sjcstorage01:/vmstore'}), log id: b9fe587 - fails with Error creating a storage domain's metadata: ("create meta file 'outbox' failed: [Errno 5] Input/output error", Are the vdsm logs you provided from sjcvhost03? There are no errors to be seen in the gluster log you provided. Could you provide mount log from sjcvhost03 (at /rhev/data-center/mnt/glusterSD/sjcstorage01:_vmstore.log most likely) If possible, /var/log/glusterfs/* from the 3 storage nodes. thanks sahina <div class="moz-cite-prefix">On 09/23/2015 05:02 AM, Brett Stevens wrote: </div> <blockquote cite="mid:CAK02sjsh7JXf56xuMSEW_knZcNem9FNsdjEhd3NAQOQiLjeTrA@mail.gmail.com" type="cite"> <div dir="ltr">Hi Sahina, <div> </div> <div>as requested here is some logs taken during a domain create.</div> <div> </div> <div> 2015-09-22 18:46:44,320 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-88) [] START, GlusterVolumesListVDSCommand(HostName = sjcstorage01, GlusterVolumesListVDSParameters:{runAsync='true', hostId='c75682ba-1e4c-42a3-85c7-16e4bb2ce5da'}), log id: 2205ff1 2015-09-22 18:46:44,413 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-88) [] Could not associate brick 'sjcstorage01:/export/vmstore/brick01' of volume '030f270a-0999-4df4-9b14-ae56eb0a2fb9' with correct network as no gluster network found in cluster 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' 2015-09-22 18:46:44,417 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-88) [] Could not associate brick 'sjcstorage02:/export/vmstore/brick01' of volume '030f270a-0999-4df4-9b14-ae56eb0a2fb9' with correct network as no gluster network found in cluster 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' 2015-09-22 18:46:44,417 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-88) [] Could not add brick 'sjcvhost02:/export/vmstore/brick01' to volume '030f270a-0999-4df4-9b14-ae56eb0a2fb9' - server uuid '29b58278-9aa3-47c5-bfb4-1948ef7fdbba' not found in cluster 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' 2015-09-22 18:46:44,418 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-88) [] FINISH, GlusterVolumesListVDSCommand, return: {030f270a-0999-4df4-9b14-ae56eb0a2fb9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@a0628f36}, log id: 2205ff1 2015-09-22 18:46:45,215 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (default task-24) [5099cda3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[sjcstorage01:/vmstore=<STORAGE_CONNECTION, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' 2015-09-22 18:46:45,230 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (default task-24) [5099cda3] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2015-09-22 18:46:45,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-24) [5099cda3] START, ConnectStorageServerVDSCommand(HostName = sjcvhost03, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='80245ac2-32a3-4d5d-b0fe-08019e2d1c9c', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='null', connection='sjcstorage01:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6a112292 2015-09-22 18:46:48,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-24) [5099cda3] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 6a112292 2015-09-22 18:46:48,073 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (default task-24) [5099cda3] Lock freed to object 'EngineLock:{exclusiveLocks='[sjcstorage01:/vmstore=<STORAGE_CONNECTION, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' 2015-09-22 18:46:48,188 INFO [org.ovirt.engine.core.bll.storage.AddGlusterFsStorageDomainCommand] (default task-23) [6410419] Running command: AddGlusterFsStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2015-09-22 18:46:48,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-23) [6410419] START, ConnectStorageServerVDSCommand(HostName = sjcvhost03, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='80245ac2-32a3-4d5d-b0fe-08019e2d1c9c', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='ec5ab31e-b5b9-4a8e-a2b2-0876df71a21e', connection='sjcstorage01:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 38a2b0d 2015-09-22 18:46:48,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-23) [6410419] FINISH, ConnectStorageServerVDSCommand, return: {ec5ab31e-b5b9-4a8e-a2b2-0876df71a21e=0}, log id: 38a2b0d 2015-09-22 18:46:48,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-23) [6410419] START, CreateStorageDomainVDSCommand(HostName = sjcvhost03, CreateStorageDomainVDSCommandParameters:{runAsync='true', hostId='80245ac2-32a3-4d5d-b0fe-08019e2d1c9c', storageDomain='StorageDomainStatic:{name='sjcvmstore', id='597d5b5b-7c09-4de9-8840-6993bd9b61a6'}', args='sjcstorage01:/vmstore'}), log id: b9fe587 2015-09-22 18:46:48,744 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-23) [6410419] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM sjcvhost03 command failed: Error creating a storage domain's metadata: ("create meta file 'outbox' failed: [Errno 5] Input/output error",) 2015-09-22 18:46:48,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-23) [6410419] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand' return value 'StatusOnlyReturnForXmlRpc [status=StatusForXmlRpc [code=362, message=Error creating a storage domain's metadata: ("create meta file 'outbox' failed: [Errno 5] Input/output error",)]]' 2015-09-22 18:46:48,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-23) [6410419] HostName = sjcvhost03 2015-09-22 18:46:48,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-23) [6410419] Command 'CreateStorageDomainVDSCommand(HostName = sjcvhost03, CreateStorageDomainVDSCommandParameters:{runAsync='true', hostId='80245ac2-32a3-4d5d-b0fe-08019e2d1c9c', storageDomain='StorageDomainStatic:{name='sjcvmstore', id='597d5b5b-7c09-4de9-8840-6993bd9b61a6'}', args='sjcstorage01:/vmstore'})' execution failed: VDSGenericException: VDSErrorException: Failed in vdscommand to CreateStorageDomainVDS, error = Error creating a storage domain's metadata: ("create meta file 'outbox' failed: [Errno 5] Input/output error",) 2015-09-22 18:46:48,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-23) [6410419] FINISH, CreateStorageDomainVDSCommand, log id: b9fe587 2015-09-22 18:46:48,745 ERROR [org.ovirt.engine.core.bll.storage.AddGlusterFsStorageDomainCommand] (default task-23) [6410419] Command 'org.ovirt.engine.core.bll.storage.AddGlusterFsStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed in vdscommand to CreateStorageDomainVDS, error = Error creating a storage domain's metadata: ("create meta file 'outbox' failed: [Errno 5] Input/output error",) (Failed with error StorageDomainMetadataCreationError and code 362) 2015-09-22 18:46:48,755 INFO [org.ovirt.engine.core.bll.storage.AddGlusterFsStorageDomainCommand] (default task-23) [6410419] Command [id=5ae15f53-69a1-47c5-b3a5-82f32c20e48f]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainDynamic; snapshot: 597d5b5b-7c09-4de9-8840-6993bd9b61a6. 2015-09-22 18:46:48,758 INFO [org.ovirt.engine.core.bll.storage.AddGlusterFsStorageDomainCommand] (default task-23) [6410419] Command [id=5ae15f53-69a1-47c5-b3a5-82f32c20e48f]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: 597d5b5b-7c09-4de9-8840-6993bd9b61a6. 2015-09-22 18:46:48,769 ERROR [org.ovirt.engine.core.bll.storage.AddGlusterFsStorageDomainCommand] (default task-23) [6410419] Transaction rolled-back for command 'org.ovirt.engine.core.bll.storage.AddGlusterFsStorageDomainCommand'. 2015-09-22 18:46:48,784 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-23) [6410419] Correlation ID: 6410419, Job ID: 78692780-a06f-49a5-b6b1-e6c24a820d62, Call Stack: null, Custom Event ID: -1, Message: Failed to add Storage Domain sjcvmstore. (User: admin@internal) 2015-09-22 18:46:48,996 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (default task-32) [1635a244] Lock Acquired to object 'EngineLock:{exclusiveLocks='[ec5ab31e-b5b9-4a8e-a2b2-0876df71a21e=<STORAGE_CONNECTION, ACTION_TYPE_FAILED_OBJECT_LOCKED>, sjcstorage01:/vmstore=<STORAGE_CONNECTION, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' 2015-09-22 18:46:49,018 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (default task-32) [1635a244] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2015-09-22 18:46:49,024 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (default task-32) [1635a244] Removing connection 'ec5ab31e-b5b9-4a8e-a2b2-0876df71a21e' from database 2015-09-22 18:46:49,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (default task-32) [1635a244] START, DisconnectStorageServerVDSCommand(HostName = sjcvhost03, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='80245ac2-32a3-4d5d-b0fe-08019e2d1c9c', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='ec5ab31e-b5b9-4a8e-a2b2-0876df71a21e', connection='sjcstorage01:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 39d3b568 2015-09-22 18:46:49,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (default task-32) [1635a244] FINISH, DisconnectStorageServerVDSCommand, return: {ec5ab31e-b5b9-4a8e-a2b2-0876df71a21e=0}, log id: 39d3b568 2015-09-22 18:46:49,252 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (default task-32) [1635a244] Lock freed to object 'EngineLock:{exclusiveLocks='[ec5ab31e-b5b9-4a8e-a2b2-0876df71a21e=<STORAGE_CONNECTION, ACTION_TYPE_FAILED_OBJECT_LOCKED>, sjcstorage01:/vmstore=<STORAGE_CONNECTION, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' 2015-09-22 18:46:49,431 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-3) [] START, GlusterVolumesListVDSCommand(HostName = sjcstorage01, GlusterVolumesListVDSParameters:{runAsync='true', hostId='c75682ba-1e4c-42a3-85c7-16e4bb2ce5da'}), log id: 17014ae8 2015-09-22 18:46:49,511 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-3) [] Could not associate brick 'sjcstorage01:/export/vmstore/brick01' of volume '030f270a-0999-4df4-9b14-ae56eb0a2fb9' with correct network as no gluster network found in cluster 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' 2015-09-22 18:46:49,515 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-3) [] Could not associate brick 'sjcstorage02:/export/vmstore/brick01' of volume '030f270a-0999-4df4-9b14-ae56eb0a2fb9' with correct network as no gluster network found in cluster 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' 2015-09-22 18:46:49,516 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-3) [] Could not add brick 'sjcvhost02:/export/vmstore/brick01' to volume '030f270a-0999-4df4-9b14-ae56eb0a2fb9' - server uuid '29b58278-9aa3-47c5-bfb4-1948ef7fdbba' not found in cluster 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' 2015-09-22 18:46:49,516 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-3) [] FINISH, GlusterVolumesListVDSCommand, return: {030f270a-0999-4df4-9b14-ae56eb0a2fb9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@92ed0f75}, log id: 17014ae8 ovirt engine thinks that sjcstorage01 is sjcstorage01, its all testbed at the moment and is all short names, defined in /etc/hosts (all copied to each server for consistancy) volume info for vmstore is Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick sjcstorage01:/export/vmstore/brick01 49157 0 Y 7444 Brick sjcstorage02:/export/vmstore/brick01 49157 0 Y 4063 Brick sjcvhost02:/export/vmstore/brick01 49156 0 Y 3243 NFS Server on localhost 2049 0 Y 3268 Self-heal Daemon on localhost N/A N/A Y 3284 NFS Server on sjcstorage01 2049 0 Y 7463 Self-heal Daemon on sjcstorage01 N/A N/A Y 7472 NFS Server on sjcstorage02 2049 0 Y 4082 Self-heal Daemon on sjcstorage02 N/A N/A Y 4090 Task Status of Volume vmstore ------------------------------------------------------------------------------ There are no active volume tasks vdsm logs from time the domain is added hread-789::DEBUG::2015-09-22 19:12:05,865::stompreactor::304::yajsonrpc.StompServer::(send) Sending response Thread-790::DEBUG::2015-09-22 19:12:07,797::task::595::Storage.TaskManager.Task::(_updateState) Task=`93731f26-a48f-45c9-9959-42c96b09cf85`::moving from state init -> state preparing Thread-790::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:07,797::logUtils::48::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-790::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:07,797::logUtils::51::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-790::DEBUG::2015-09-22 19:12:07,797::task::1191::Storage.TaskManager.Task::(prepare) Task=`93731f26-a48f-45c9-9959-42c96b09cf85`::finished: {} Thread-790::DEBUG::2015-09-22 19:12:07,797::task::595::Storage.TaskManager.Task::(_updateState) Task=`93731f26-a48f-45c9-9959-42c96b09cf85`::moving from state preparing -> state finished Thread-790::DEBUG::2015-09-22 19:12:07,797::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-790::DEBUG::2015-09-22 19:12:07,797::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-790::DEBUG::2015-09-22 19:12:07,797::task::993::Storage.TaskManager.Task::(_decref) Task=`93731f26-a48f-45c9-9959-42c96b09cf85`::ref 0 aborting False Thread-790::DEBUG::2015-09-22 19:12:07,802::stompreactor::304::yajsonrpc.StompServer::(send) Sending response Reactor thread::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:14,816::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) Accepting connection from <a moz-do-not-send="true" href="http://127.0.0.1:52510">127.0.0.1:52510</a> Reactor thread::DEBUG::2015-09-22 19:12:14,822::protocoldetector::82::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:14,823::protocoldetector::118::ProtocolDetector.Detector::(handle_read) Detected protocol xml from <a moz-do-not-send="true" href="http://127.0.0.1:52510">127.0.0.1:52510</a> Reactor thread::DEBUG::2015-09-22 19:12:14,823::bindingxmlrpc::1297::XmlDetector::(handle_socket) xml over http detected from ('127.0.0.1', 52510) BindingXMLRPC::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:14,823::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for <a moz-do-not-send="true" href="http://127.0.0.1:52510">127.0.0.1:52510</a> Thread-791::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:14,823::xmlrpc::84::vds.XMLRPCServer::(_process_requests) Request handler for <a moz-do-not-send="true" href="http://127.0.0.1:52510">127.0.0.1:52510</a> started Thread-791::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:14,825::xmlrpc::92::vds.XMLRPCServer::(_process_requests) Request handler for <a moz-do-not-send="true" href="http://127.0.0.1:52510">127.0.0.1:52510</a> stopped Thread-792::DEBUG::2015-09-22 19:12:20,872::stompreactor::304::yajsonrpc.StompServer::(send) Sending response Thread-793::DEBUG::2015-09-22 19:12:22,832::task::595::Storage.TaskManager.Task::(_updateState) Task=`a1f48f6f-a9ba-4dac-b024-ae6289f4a7dd`::moving from state init -> state preparing Thread-793::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:22,832::logUtils::48::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-793::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:22,832::logUtils::51::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-793::DEBUG::2015-09-22 19:12:22,832::task::1191::Storage.TaskManager.Task::(prepare) Task=`a1f48f6f-a9ba-4dac-b024-ae6289f4a7dd`::finished: {} Thread-793::DEBUG::2015-09-22 19:12:22,832::task::595::Storage.TaskManager.Task::(_updateState) Task=`a1f48f6f-a9ba-4dac-b024-ae6289f4a7dd`::moving from state preparing -> state finished Thread-793::DEBUG::2015-09-22 19:12:22,833::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-793::DEBUG::2015-09-22 19:12:22,833::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-793::DEBUG::2015-09-22 19:12:22,833::task::993::Storage.TaskManager.Task::(_decref) Task=`a1f48f6f-a9ba-4dac-b024-ae6289f4a7dd`::ref 0 aborting False Thread-793::DEBUG::2015-09-22 19:12:22,837::stompreactor::304::yajsonrpc.StompServer::(send) Sending response Reactor thread::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:29,841::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) Accepting connection from <a moz-do-not-send="true" href="http://127.0.0.1:52511">127.0.0.1:52511</a> Reactor thread::DEBUG::2015-09-22 19:12:29,848::protocoldetector::82::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:29,849::protocoldetector::118::ProtocolDetector.Detector::(handle_read) Detected protocol xml from <a moz-do-not-send="true" href="http://127.0.0.1:52511">127.0.0.1:52511</a> Reactor thread::DEBUG::2015-09-22 19:12:29,849::bindingxmlrpc::1297::XmlDetector::(handle_socket) xml over http detected from ('127.0.0.1', 52511) BindingXMLRPC::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:29,849::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for <a moz-do-not-send="true" href="http://127.0.0.1:52511">127.0.0.1:52511</a> Thread-794::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:29,849::xmlrpc::84::vds.XMLRPCServer::(_process_requests) Request handler for <a moz-do-not-send="true" href="http://127.0.0.1:52511">127.0.0.1:52511</a> started Thread-794::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:29,851::xmlrpc::92::vds.XMLRPCServer::(_process_requests) Request handler for <a moz-do-not-send="true" href="http://127.0.0.1:52511">127.0.0.1:52511</a> stopped Thread-795::DEBUG::2015-09-22 19:12:35,520::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.connectStorageServer' in bridge with {u'connectionParams': [{u'id': u'00000000-0000-0000-0000-000000000000', u'connection': u'sjcstorage01:/vmstore', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 7} Thread-795::DEBUG::2015-09-22 19:12:35,520::task::595::Storage.TaskManager.Task::(_updateState) Task=`6e8aec06-556f-4659-9ee8-efc60b637ff6`::moving from state init -> state preparing Thread-795::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:35,521::logUtils::48::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'id': u'00000000-0000-0000-0000-000000000000', u'connection': u'sjcstorage01:/vmstore', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], options=None) Thread-795::DEBUG::2015-09-22 19:12:35,539::fileUtils::143::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt/glusterSD/sjcstorage01:_vmstore mode: None Thread-795::DEBUG::2015-09-22 19:12:35,540::mount::229::Storage.Misc.excCmd::(_runcmd) /usr/bin/sudo -n /usr/bin/systemd-run --scope --slice=vdsm-glusterfs /usr/bin/mount -t glusterfs -o backup-volfile-servers=sjcstorage02:sjcvhost02 sjcstorage01:/vmstore /rhev/data-center/mnt/glusterSD/sjcstorage01:_vmstore (cwd None) Thread-795::DEBUG::2015-09-22 19:12:35,706::hsm::2417::Storage.HSM::(__prefetchDomains) glusterDomPath: glusterSD/* Thread-796::DEBUG::2015-09-22 19:12:35,707::__init__::298::IOProcessClient::(_run) Starting IOProcess... Thread-797::DEBUG::2015-09-22 19:12:35,712::__init__::298::IOProcessClient::(_run) Starting IOProcess... Thread-795::DEBUG::2015-09-22 19:12:35,721::hsm::2429::Storage.HSM::(__prefetchDomains) Found SD uuids: () Thread-795::DEBUG::2015-09-22 19:12:35,721::hsm::2489::Storage.HSM::(connectStorageServer) knownSDs: {41b75ca9-9783-42a7-9a23-10a2ae3cbb96: storage.glusterSD.findDomain, 597d5b5b-7c09-4de9-8840-6993bd9b61a6: storage.glusterSD.findDomain, ef17fec4-fecf-4d7e-b815-d1db4ef65225: storage.glusterSD.findDomain} Thread-795::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:35,721::logUtils::51::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': u'00000000-0000-0000-0000-000000000000'}]} Thread-795::DEBUG::2015-09-22 19:12:35,722::task::1191::Storage.TaskManager.Task::(prepare) Task=`6e8aec06-556f-4659-9ee8-efc60b637ff6`::finished: {'statuslist': [{'status': 0, 'id': u'00000000-0000-0000-0000-000000000000'}]} Thread-795::DEBUG::2015-09-22 19:12:35,722::task::595::Storage.TaskManager.Task::(_updateState) Task=`6e8aec06-556f-4659-9ee8-efc60b637ff6`::moving from state preparing -> state finished Thread-795::DEBUG::2015-09-22 19:12:35,722::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-795::DEBUG::2015-09-22 19:12:35,722::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-795::DEBUG::2015-09-22 19:12:35,722::task::993::Storage.TaskManager.Task::(_decref) Task=`6e8aec06-556f-4659-9ee8-efc60b637ff6`::ref 0 aborting False Thread-795::DEBUG::2015-09-22 19:12:35,722::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.connectStorageServer' in bridge with [{'status': 0, 'id': u'00000000-0000-0000-0000-000000000000'}] Thread-795::DEBUG::2015-09-22 19:12:35,722::stompreactor::304::yajsonrpc.StompServer::(send) Sending response Thread-798::DEBUG::2015-09-22 19:12:35,775::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.connectStorageServer' in bridge with {u'connectionParams': [{u'id': u'cd55e6a1-022a-4b32-8a94-cab506a9b054', u'connection': u'sjcstorage01:/vmstore', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 7} Thread-798::DEBUG::2015-09-22 19:12:35,775::task::595::Storage.TaskManager.Task::(_updateState) Task=`b2c91515-bdda-45e5-a031-61a1e2c53c4d`::moving from state init -> state preparing Thread-798::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:35,776::logUtils::48::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'id': u'cd55e6a1-022a-4b32-8a94-cab506a9b054', u'connection': u'sjcstorage01:/vmstore', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], options=None) Thread-798::DEBUG::2015-09-22 19:12:35,777::hsm::2417::Storage.HSM::(__prefetchDomains) glusterDomPath: glusterSD/* Thread-798::DEBUG::2015-09-22 19:12:35,782::hsm::2429::Storage.HSM::(__prefetchDomains) Found SD uuids: () Thread-798::DEBUG::2015-09-22 19:12:35,782::hsm::2489::Storage.HSM::(connectStorageServer) knownSDs: {41b75ca9-9783-42a7-9a23-10a2ae3cbb96: storage.glusterSD.findDomain, 597d5b5b-7c09-4de9-8840-6993bd9b61a6: storage.glusterSD.findDomain, ef17fec4-fecf-4d7e-b815-d1db4ef65225: storage.glusterSD.findDomain} Thread-798::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:35,782::logUtils::51::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': u'cd55e6a1-022a-4b32-8a94-cab506a9b054'}]} Thread-798::DEBUG::2015-09-22 19:12:35,783::task::1191::Storage.TaskManager.Task::(prepare) Task=`b2c91515-bdda-45e5-a031-61a1e2c53c4d`::finished: {'statuslist': [{'status': 0, 'id': u'cd55e6a1-022a-4b32-8a94-cab506a9b054'}]} Thread-798::DEBUG::2015-09-22 19:12:35,783::task::595::Storage.TaskManager.Task::(_updateState) Task=`b2c91515-bdda-45e5-a031-61a1e2c53c4d`::moving from state preparing -> state finished Thread-798::DEBUG::2015-09-22 19:12:35,783::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-798::DEBUG::2015-09-22 19:12:35,783::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-798::DEBUG::2015-09-22 19:12:35,783::task::993::Storage.TaskManager.Task::(_decref) Task=`b2c91515-bdda-45e5-a031-61a1e2c53c4d`::ref 0 aborting False Thread-798::DEBUG::2015-09-22 19:12:35,783::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.connectStorageServer' in bridge with [{'status': 0, 'id': u'cd55e6a1-022a-4b32-8a94-cab506a9b054'}] Thread-798::DEBUG::2015-09-22 19:12:35,783::stompreactor::304::yajsonrpc.StompServer::(send) Sending response Thread-801::DEBUG::2015-09-22 19:12:35,787::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StorageDomain.create' in bridge with {u'name': u'sjcvmstore01', u'domainType': 7, u'domainClass': 1, u'typeArgs': u'sjcstorage01:/vmstore', u'version': u'3', u'storagedomainID': u'c02fda97-62e3-40d3-9a6e-ac5d100f8ad3'} Thread-801::DEBUG::2015-09-22 19:12:35,788::task::595::Storage.TaskManager.Task::(_updateState) Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::moving from state init -> state preparing Thread-801::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:35,788::logUtils::48::dispatcher::(wrapper) Run and protect: createStorageDomain(storageType=7, sdUUID=u'c02fda97-62e3-40d3-9a6e-ac5d100f8ad3', domainName=u'sjcvmstore01', typeSpecificArg=u'sjcstorage01:/vmstore', domClass=1, domVersion=u'3', options=None) Thread-801::DEBUG::2015-09-22 19:12:35,788::misc::733::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) Thread-801::DEBUG::2015-09-22 19:12:35,788::misc::736::Storage.SamplingMethod::(__call__) Got in to sampling method Thread-801::DEBUG::2015-09-22 19:12:35,788::misc::733::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) Thread-801::DEBUG::2015-09-22 19:12:35,788::misc::736::Storage.SamplingMethod::(__call__) Got in to sampling method Thread-801::DEBUG::2015-09-22 19:12:35,788::iscsi::431::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds Thread-801::DEBUG::2015-09-22 19:12:35,788::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None) Thread-801::DEBUG::2015-09-22 19:12:35,821::misc::743::Storage.SamplingMethod::(__call__) Returning last result Thread-801::DEBUG::2015-09-22 19:12:35,821::misc::733::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.hba.rescan) Thread-801::DEBUG::2015-09-22 19:12:35,821::misc::736::Storage.SamplingMethod::(__call__) Got in to sampling method Thread-801::DEBUG::2015-09-22 19:12:35,821::hba::56::Storage.HBA::(rescan) Starting scan Thread-802::DEBUG::2015-09-22 19:12:35,882::stompreactor::304::yajsonrpc.StompServer::(send) Sending response Thread-801::DEBUG::2015-09-22 19:12:35,912::hba::62::Storage.HBA::(rescan) Scan finished Thread-801::DEBUG::2015-09-22 19:12:35,912::misc::743::Storage.SamplingMethod::(__call__) Returning last result Thread-801::DEBUG::2015-09-22 19:12:35,912::multipath::77::Storage.Misc.excCmd::(rescan) /usr/bin/sudo -n /usr/sbin/multipath (cwd None) Thread-801::DEBUG::2015-09-22 19:12:35,936::multipath::77::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 Thread-801::DEBUG::2015-09-22 19:12:35,936::utils::661::root::(execCmd) /sbin/udevadm settle --timeout=5 (cwd None) Thread-801::DEBUG::2015-09-22 19:12:35,946::utils::679::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 Thread-801::DEBUG::2015-09-22 19:12:35,947::lvm::498::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-801::DEBUG::2015-09-22 19:12:35,947::lvm::500::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-801::DEBUG::2015-09-22 19:12:35,947::lvm::509::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-801::DEBUG::2015-09-22 19:12:35,948::lvm::511::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-801::DEBUG::2015-09-22 19:12:35,948::lvm::529::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-801::DEBUG::2015-09-22 19:12:35,948::lvm::531::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-801::DEBUG::2015-09-22 19:12:35,948::misc::743::Storage.SamplingMethod::(__call__) Returning last result Thread-801::ERROR::2015-09-22 19:12:35,949::sdc::138::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c02fda97-62e3-40d3-9a6e-ac5d100f8ad3 Thread-801::ERROR::2015-09-22 19:12:35,949::sdc::155::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c02fda97-62e3-40d3-9a6e-ac5d100f8ad3 Thread-801::DEBUG::2015-09-22 19:12:35,949::lvm::371::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex Thread-801::DEBUG::2015-09-22 19:12:35,950::lvm::291::Storage.Misc.excCmd::(cmd) /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c02fda97-62e3-40d3-9a6e-ac5d100f8ad3 (cwd None) Thread-801::DEBUG::2015-09-22 19:12:35,985::lvm::291::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n Volume group "c02fda97-62e3-40d3-9a6e-ac5d100f8ad3" not found\n Cannot process volume group c02fda97-62e3-40d3-9a6e-ac5d100f8ad3\n'; <rc> = 5 Thread-801::WARNING::2015-09-22 19:12:35,986::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', ' Volume group "c02fda97-62e3-40d3-9a6e-ac5d100f8ad3" not found', ' Cannot process volume group c02fda97-62e3-40d3-9a6e-ac5d100f8ad3'] Thread-801::DEBUG::2015-09-22 19:12:35,987::lvm::416::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex Thread-801::ERROR::2015-09-22 19:12:35,997::sdc::144::Storage.StorageDomainCache::(_findDomain) domain c02fda97-62e3-40d3-9a6e-ac5d100f8ad3 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 142, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 172, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c02fda97-62e3-40d3-9a6e-ac5d100f8ad3',) Thread-801::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:35,998::nfsSD::69::Storage.StorageDomain::(create) sdUUID=c02fda97-62e3-40d3-9a6e-ac5d100f8ad3 domainName=sjcvmstore01 remotePath=sjcstorage01:/vmstore domClass=1 Thread-801::DEBUG::2015-09-22 19:12:36,015::__init__::298::IOProcessClient::(_run) Starting IOProcess... Thread-801::ERROR::2015-09-22 19:12:36,067::task::866::Storage.TaskManager.Task::(_setError) Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 49, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 2697, in createStorageDomain domVersion) File "/usr/share/vdsm/storage/nfsSD.py", line 84, in create remotePath, storageType, version) File "/usr/share/vdsm/storage/fileSD.py", line 264, in _prepareMetadata "create meta file '%s' failed: %s" % (metaFile, str(e))) StorageDomainMetadataCreationError: Error creating a storage domain's metadata: ("create meta file 'outbox' failed: [Errno 5] Input/output error",) Thread-801::DEBUG::2015-09-22 19:12:36,067::task::885::Storage.TaskManager.Task::(_run) Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::Task._run: d2d29352-8677-45cb-a4ab-06aa32cf1acb (7, u'c02fda97-62e3-40d3-9a6e-ac5d100f8ad3', u'sjcvmstore01', u'sjcstorage01:/vmstore', 1, u'3') {} failed - stopping task Thread-801::DEBUG::2015-09-22 19:12:36,067::task::1246::Storage.TaskManager.Task::(stop) Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::stopping in state preparing (force False) Thread-801::DEBUG::2015-09-22 19:12:36,067::task::993::Storage.TaskManager.Task::(_decref) Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::ref 1 aborting True Thread-801::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:36,067::task::1171::Storage.TaskManager.Task::(prepare) Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::aborting: Task is aborted: "Error creating a storage domain's metadata" - code 362 Thread-801::DEBUG::2015-09-22 19:12:36,068::task::1176::Storage.TaskManager.Task::(prepare) Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::Prepare: aborted: Error creating a storage domain's metadata Thread-801::DEBUG::2015-09-22 19:12:36,068::task::993::Storage.TaskManager.Task::(_decref) Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::ref 0 aborting True Thread-801::DEBUG::2015-09-22 19:12:36,068::task::928::Storage.TaskManager.Task::(_doAbort) Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::Task._doAbort: force False Thread-801::DEBUG::2015-09-22 19:12:36,068::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-801::DEBUG::2015-09-22 19:12:36,068::task::595::Storage.TaskManager.Task::(_updateState) Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::moving from state preparing -> state aborting Thread-801::DEBUG::2015-09-22 19:12:36,068::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::_aborting: recover policy none Thread-801::DEBUG::2015-09-22 19:12:36,068::task::595::Storage.TaskManager.Task::(_updateState) Task=`d2d29352-8677-45cb-a4ab-06aa32cf1acb`::moving from state aborting -> state failed Thread-801::DEBUG::2015-09-22 19:12:36,068::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-801::DEBUG::2015-09-22 19:12:36,068::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-801::ERROR::2015-09-22 19:12:36,068::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': 'Error creating a storage domain\'s metadata: ("create meta file \'outbox\' failed: [Errno 5] Input/output error",)', 'code': 362}} Thread-801::DEBUG::2015-09-22 19:12:36,069::stompreactor::304::yajsonrpc.StompServer::(send) Sending response Thread-807::DEBUG::2015-09-22 19:12:36,180::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.disconnectStorageServer' in bridge with {u'connectionParams': [{u'id': u'cd55e6a1-022a-4b32-8a94-cab506a9b054', u'connection': u'sjcstorage01:/vmstore', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 7} Thread-807::DEBUG::2015-09-22 19:12:36,181::task::595::Storage.TaskManager.Task::(_updateState) Task=`01af6594-9c7b-4ec7-b08f-02627db8f421`::moving from state init -> state preparing Thread-807::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:36,182::logUtils::48::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'id': u'cd55e6a1-022a-4b32-8a94-cab506a9b054', u'connection': u'sjcstorage01:/vmstore', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], options=None) Thread-807::DEBUG::2015-09-22 19:12:36,182::mount::229::Storage.Misc.excCmd::(_runcmd) /usr/bin/sudo -n /usr/bin/umount -f -l /rhev/data-center/mnt/glusterSD/sjcstorage01:_vmstore (cwd None) Thread-807::DEBUG::2015-09-22 19:12:36,222::misc::733::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) Thread-807::DEBUG::2015-09-22 19:12:36,222::misc::736::Storage.SamplingMethod::(__call__) Got in to sampling method Thread-807::DEBUG::2015-09-22 19:12:36,222::misc::733::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) Thread-807::DEBUG::2015-09-22 19:12:36,222::misc::736::Storage.SamplingMethod::(__call__) Got in to sampling method Thread-807::DEBUG::2015-09-22 19:12:36,223::iscsi::431::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds Thread-807::DEBUG::2015-09-22 19:12:36,223::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None) Thread-807::DEBUG::2015-09-22 19:12:36,258::misc::743::Storage.SamplingMethod::(__call__) Returning last result Thread-807::DEBUG::2015-09-22 19:12:36,258::misc::733::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.hba.rescan) Thread-807::DEBUG::2015-09-22 19:12:36,258::misc::736::Storage.SamplingMethod::(__call__) Got in to sampling method Thread-807::DEBUG::2015-09-22 19:12:36,258::hba::56::Storage.HBA::(rescan) Starting scan Thread-807::DEBUG::2015-09-22 19:12:36,350::hba::62::Storage.HBA::(rescan) Scan finished Thread-807::DEBUG::2015-09-22 19:12:36,350::misc::743::Storage.SamplingMethod::(__call__) Returning last result Thread-807::DEBUG::2015-09-22 19:12:36,350::multipath::77::Storage.Misc.excCmd::(rescan) /usr/bin/sudo -n /usr/sbin/multipath (cwd None) Thread-807::DEBUG::2015-09-22 19:12:36,374::multipath::77::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 Thread-807::DEBUG::2015-09-22 19:12:36,374::utils::661::root::(execCmd) /sbin/udevadm settle --timeout=5 (cwd None) Thread-807::DEBUG::2015-09-22 19:12:36,383::utils::679::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 Thread-807::DEBUG::2015-09-22 19:12:36,384::lvm::498::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-807::DEBUG::2015-09-22 19:12:36,385::lvm::500::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-807::DEBUG::2015-09-22 19:12:36,385::lvm::509::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-807::DEBUG::2015-09-22 19:12:36,385::lvm::511::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-807::DEBUG::2015-09-22 19:12:36,386::lvm::529::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-807::DEBUG::2015-09-22 19:12:36,386::lvm::531::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-807::DEBUG::2015-09-22 19:12:36,386::misc::743::Storage.SamplingMethod::(__call__) Returning last result Thread-807::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:36,386::logUtils::51::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': u'cd55e6a1-022a-4b32-8a94-cab506a9b054'}]} Thread-807::DEBUG::2015-09-22 19:12:36,387::task::1191::Storage.TaskManager.Task::(prepare) Task=`01af6594-9c7b-4ec7-b08f-02627db8f421`::finished: {'statuslist': [{'status': 0, 'id': u'cd55e6a1-022a-4b32-8a94-cab506a9b054'}]} Thread-807::DEBUG::2015-09-22 19:12:36,387::task::595::Storage.TaskManager.Task::(_updateState) Task=`01af6594-9c7b-4ec7-b08f-02627db8f421`::moving from state preparing -> state finished Thread-807::DEBUG::2015-09-22 19:12:36,387::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-807::DEBUG::2015-09-22 19:12:36,387::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-807::DEBUG::2015-09-22 19:12:36,387::task::993::Storage.TaskManager.Task::(_decref) Task=`01af6594-9c7b-4ec7-b08f-02627db8f421`::ref 0 aborting False Thread-807::DEBUG::2015-09-22 19:12:36,388::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.disconnectStorageServer' in bridge with [{'status': 0, 'id': u'cd55e6a1-022a-4b32-8a94-cab506a9b054'}] Thread-807::DEBUG::2015-09-22 19:12:36,388::stompreactor::304::yajsonrpc.StompServer::(send) Sending response Thread-808::DEBUG::2015-09-22 19:12:37,868::task::595::Storage.TaskManager.Task::(_updateState) Task=`adddaa68-dd1d-4d2e-9853-b7894ee4809c`::moving from state init -> state preparing Thread-808::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:37,868::logUtils::48::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-808::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:37,868::logUtils::51::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-808::DEBUG::2015-09-22 19:12:37,868::task::1191::Storage.TaskManager.Task::(prepare) Task=`adddaa68-dd1d-4d2e-9853-b7894ee4809c`::finished: {} Thread-808::DEBUG::2015-09-22 19:12:37,868::task::595::Storage.TaskManager.Task::(_updateState) Task=`adddaa68-dd1d-4d2e-9853-b7894ee4809c`::moving from state preparing -> state finished Thread-808::DEBUG::2015-09-22 19:12:37,868::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-808::DEBUG::2015-09-22 19:12:37,868::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-808::DEBUG::2015-09-22 19:12:37,868::task::993::Storage.TaskManager.Task::(_decref) Task=`adddaa68-dd1d-4d2e-9853-b7894ee4809c`::ref 0 aborting False Thread-808::DEBUG::2015-09-22 19:12:37,873::stompreactor::304::yajsonrpc.StompServer::(send) Sending response Reactor thread::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:44,867::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) Accepting connection from <a moz-do-not-send="true" href="http://127.0.0.1:52512">127.0.0.1:52512</a> Reactor thread::DEBUG::2015-09-22 19:12:44,874::protocoldetector::82::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:44,875::protocoldetector::118::ProtocolDetector.Detector::(handle_read) Detected protocol xml from <a moz-do-not-send="true" href="http://127.0.0.1:52512">127.0.0.1:52512</a> Reactor thread::DEBUG::2015-09-22 19:12:44,875::bindingxmlrpc::1297::XmlDetector::(handle_socket) xml over http detected from ('127.0.0.1', 52512) BindingXMLRPC::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:44,875::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for <a moz-do-not-send="true" href="http://127.0.0.1:52512">127.0.0.1:52512</a> Thread-809::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:44,876::xmlrpc::84::vds.XMLRPCServer::(_process_requests) Request handler for <a moz-do-not-send="true" href="http://127.0.0.1:52512">127.0.0.1:52512</a> started Thread-809::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:44,877::xmlrpc::92::vds.XMLRPCServer::(_process_requests) Request handler for <a moz-do-not-send="true" href="http://127.0.0.1:52512">127.0.0.1:52512</a> stopped Thread-810::DEBUG::2015-09-22 19:12:50,889::stompreactor::304::yajsonrpc.StompServer::(send) Sending response Thread-811::DEBUG::2015-09-22 19:12:52,902::task::595::Storage.TaskManager.Task::(_updateState) Task=`d9fb30bc-dff3-4df3-a25e-2ad09a940fde`::moving from state init -> state preparing Thread-811::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:52,902::logUtils::48::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-811::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:52,902::logUtils::51::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-811::DEBUG::2015-09-22 19:12:52,902::task::1191::Storage.TaskManager.Task::(prepare) Task=`d9fb30bc-dff3-4df3-a25e-2ad09a940fde`::finished: {} Thread-811::DEBUG::2015-09-22 19:12:52,903::task::595::Storage.TaskManager.Task::(_updateState) Task=`d9fb30bc-dff3-4df3-a25e-2ad09a940fde`::moving from state preparing -> state finished Thread-811::DEBUG::2015-09-22 19:12:52,903::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-811::DEBUG::2015-09-22 19:12:52,903::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-811::DEBUG::2015-09-22 19:12:52,903::task::993::Storage.TaskManager.Task::(_decref) Task=`d9fb30bc-dff3-4df3-a25e-2ad09a940fde`::ref 0 aborting False Thread-811::DEBUG::2015-09-22 19:12:52,908::stompreactor::304::yajsonrpc.StompServer::(send) Sending response Reactor thread::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:59,895::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) Accepting connection from <a moz-do-not-send="true" href="http://127.0.0.1:52513">127.0.0.1:52513</a> Reactor thread::DEBUG::2015-09-22 19:12:59,902::protocoldetector::82::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:59,902::protocoldetector::118::ProtocolDetector.Detector::(handle_read) Detected protocol xml from <a moz-do-not-send="true" href="http://127.0.0.1:52513">127.0.0.1:52513</a> Reactor thread::DEBUG::2015-09-22 19:12:59,902::bindingxmlrpc::1297::XmlDetector::(handle_socket) xml over http detected from ('127.0.0.1', 52513) BindingXMLRPC::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:59,903::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for <a moz-do-not-send="true" href="http://127.0.0.1:52513">127.0.0.1:52513</a> Thread-812::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:59,903::xmlrpc::84::vds.XMLRPCServer::(_process_requests) Request handler for <a moz-do-not-send="true" href="http://127.0.0.1:52513">127.0.0.1:52513</a> started Thread-812::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:12:59,904::xmlrpc::92::vds.XMLRPCServer::(_process_requests) Request handler for <a moz-do-not-send="true" href="http://127.0.0.1:52513">127.0.0.1:52513</a> stopped Thread-813::DEBUG::2015-09-22 19:13:05,898::stompreactor::304::yajsonrpc.StompServer::(send) Sending response Thread-814::DEBUG::2015-09-22 19:13:07,934::task::595::Storage.TaskManager.Task::(_updateState) Task=`c526c56c-6254-4497-9c3e-ffe09ed54af2`::moving from state init -> state preparing Thread-814::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:13:07,935::logUtils::48::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-814::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:13:07,935::logUtils::51::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-814::DEBUG::2015-09-22 19:13:07,935::task::1191::Storage.TaskManager.Task::(prepare) Task=`c526c56c-6254-4497-9c3e-ffe09ed54af2`::finished: {} Thread-814::DEBUG::2015-09-22 19:13:07,935::task::595::Storage.TaskManager.Task::(_updateState) Task=`c526c56c-6254-4497-9c3e-ffe09ed54af2`::moving from state preparing -> state finished Thread-814::DEBUG::2015-09-22 19:13:07,935::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-814::DEBUG::2015-09-22 19:13:07,935::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-814::DEBUG::2015-09-22 19:13:07,935::task::993::Storage.TaskManager.Task::(_decref) Task=`c526c56c-6254-4497-9c3e-ffe09ed54af2`::ref 0 aborting False Thread-814::DEBUG::2015-09-22 19:13:07,939::stompreactor::304::yajsonrpc.StompServer::(send) Sending response Reactor thread::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:13:14,921::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept) Accepting connection from <a moz-do-not-send="true" href="http://127.0.0.1:52515">127.0.0.1:52515</a> Reactor thread::DEBUG::2015-09-22 19:13:14,927::protocoldetector::82::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:13:14,928::protocoldetector::118::ProtocolDetector.Detector::(handle_read) Detected protocol xml from <a moz-do-not-send="true" href="http://127.0.0.1:52515">127.0.0.1:52515</a> Reactor thread::DEBUG::2015-09-22 19:13:14,928::bindingxmlrpc::1297::XmlDetector::(handle_socket) xml over http detected from ('127.0.0.1', 52515) BindingXMLRPC::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:13:14,928::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for <a moz-do-not-send="true" href="http://127.0.0.1:52515">127.0.0.1:52515</a> Thread-815::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:13:14,928::xmlrpc::84::vds.XMLRPCServer::(_process_requests) Request handler for <a moz-do-not-send="true" href="http://127.0.0.1:52515">127.0.0.1:52515</a> started Thread-815::<a class="moz-txt-link-freetext" href="INFO::2015-09-22">INFO::2015-09-22</a> 19:13:14,930::xmlrpc::92::vds.XMLRPCServer::(_process_requests) Request handler for <a moz-do-not-send="true" href="http://127.0.0.1:52515">127.0.0.1:52515</a> stopped Thread-816::DEBUG::2015-09-22 19:13:20,906::stompreactor::304::yajsonrpc.StompServer::(send) Sending response </div> <div> </div> <div> </div> <div>gluster logs</div> <div> </div> <div> +------------------------------------------------------------------------------+ 1: volume vmstore-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host sjcstorage01 5: option remote-subvolume /export/vmstore/brick01 6: option transport-type socket 7: option send-gids true 8: end-volume 9: 10: volume vmstore-client-1 11: type protocol/client 12: option ping-timeout 42 13: option remote-host sjcstorage02 14: option remote-subvolume /export/vmstore/brick01 15: option transport-type socket 16: option send-gids true 17: end-volume 18: 19: volume vmstore-client-2 20: type protocol/client 21: option ping-timeout 42 22: option remote-host sjcvhost02 23: option remote-subvolume /export/vmstore/brick01 24: option transport-type socket 25: option send-gids true 26: end-volume 27: 28: volume vmstore-replicate-0 29: type cluster/replicate 30: option arbiter-count 1 31: subvolumes vmstore-client-0 vmstore-client-1 vmstore-client-2 32: end-volume 33: 34: volume vmstore-dht 35: type cluster/distribute 36: subvolumes vmstore-replicate-0 37: end-volume 38: 39: volume vmstore-write-behind 40: type performance/write-behind 41: subvolumes vmstore-dht 42: end-volume 43: 44: volume vmstore-read-ahead 45: type performance/read-ahead 46: subvolumes vmstore-write-behind 47: end-volume 48: 49: volume vmstore-readdir-ahead 50: type performance/readdir-ahead 52: end-volume 53: 54: volume vmstore-io-cache 55: type performance/io-cache 56: subvolumes vmstore-readdir-ahead 57: end-volume 58: 59: volume vmstore-quick-read 60: type performance/quick-read 61: subvolumes vmstore-io-cache 62: end-volume 63: 64: volume vmstore-open-behind 65: type performance/open-behind 66: subvolumes vmstore-quick-read 67: end-volume 68: 69: volume vmstore-md-cache 70: type performance/md-cache 71: subvolumes vmstore-open-behind 72: end-volume 73: 74: volume vmstore 75: type debug/io-stats 76: option latency-measurement off 77: option count-fop-hits off 78: subvolumes vmstore-md-cache 79: end-volume 80: 81: volume meta-autoload 82: type meta 83: subvolumes vmstore 84: end-volume 85: +------------------------------------------------------------------------------+ [2015-09-22 05:29:07.586205] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-vmstore-client-0: changing port to 49153 (from 0) [2015-09-22 05:29:07.586325] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-vmstore-client-1: changing port to 49153 (from 0) [2015-09-22 05:29:07.586480] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-vmstore-client-2: changing port to 49153 (from 0) [2015-09-22 05:29:07.595052] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vmstore-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-09-22 05:29:07.595397] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vmstore-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-09-22 05:29:07.595576] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vmstore-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-09-22 05:29:07.595721] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-0: Connected to vmstore-client-0, attached to remote volume '/export/vmstore/brick01'. [2015-09-22 05:29:07.595738] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-09-22 05:29:07.596044] I [MSGID: 108005] [afr-common.c:3998:afr_notify] 0-vmstore-replicate-0: Subvolume 'vmstore-client-0' came back up; going online. [2015-09-22 05:29:07.596170] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-1: Connected to vmstore-client-1, attached to remote volume '/export/vmstore/brick01'. [2015-09-22 05:29:07.596189] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-09-22 05:29:07.596495] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-2: Connected to vmstore-client-2, attached to remote volume : [2015-09-22 05:29:07.596189] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-09-22 05:29:07.596495] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-2: Connected to vmstore-client-2, attached to remote volume '/export/vmstore/brick01'. [2015-09-22 05:29:07.596506] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-2: Server and Client lk-version numbers are not same, reopening the fds [2015-09-22 05:29:07.608758] I [fuse-bridge.c:5053:fuse_graph_setup] 0-fuse: switched to graph 0 [2015-09-22 05:29:07.608910] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-0: Server lk version = 1 [2015-09-22 05:29:07.608936] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-1: Server lk version = 1 [2015-09-22 05:29:07.608950] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-2: Server lk version = 1 [2015-09-22 05:29:07.609695] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2015-09-22 05:29:07.609868] I [fuse-bridge.c:3979:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22 [2015-09-22 05:29:07.616577] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-vmstore-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0 [2015-09-22 05:29:07.620230] I [MSGID: 109036] [dht-common.c:7754:dht_log_new_layout_for_dir_selfheal] 0-vmstore-dht: Setting layout of / with [Subvol_name: vmstore-replicate-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ], [2015-09-22 05:29:08.122415] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 26: REMOVEXATTR() /__DIRECT_IO_TEST__ => -1 (No data available) [2015-09-22 05:29:08.137359] I [MSGID: 109036] [dht-common.c:7754:dht_log_new_layout_for_dir_selfheal] 0-vmstore-dht: Setting layout of /061b73d5-ae59-462e-b674-ea9c60d436c2 with [Subvol_name: vmstore-replicate-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ], [2015-09-22 05:29:08.145835] I [MSGID: 109036] [dht-common.c:7754:dht_log_new_layout_for_dir_selfheal] 0-vmstore-dht: Setting layout of /061b73d5-ae59-462e-b674-ea9c60d436c2/dom_md with [Subvol_name: vmstore-replicate-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ], [2015-09-22 05:30:57.897819] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.4 (args: /usr/sbin/glusterfs --volfile-server=sjcvhost02 --volfile-server=sjcstorage01 --volfile-server=sjcstorage02 --volfile-id=/vmstore /rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore) [2015-09-22 05:30:57.909889] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-09-22 05:30:57.923087] I [MSGID: 114020] [client.c:2118:notify] 0-vmstore-client-0: parent translators are ready, attempting connect on transport [2015-09-22 05:30:57.925701] I [MSGID: 114020] [client.c:2118:notify] 0-vmstore-client-1: parent translators are ready, attempting connect on transport [2015-09-22 05:30:57.927984] I [MSGID: 114020] [client.c:2118:notify] 0-vmstore-client-2: parent translators are ready, attempting connect on transport Final graph: +------------------------------------------------------------------------------+ 1: volume vmstore-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host sjcstorage01 5: option remote-subvolume /export/vmstore/brick01 6: option transport-type socket 7: option send-gids true 8: end-volume 9: 10: volume vmstore-client-1 11: type protocol/client 12: option ping-timeout 42 13: option remote-host sjcstorage02 14: option remote-subvolume /export/vmstore/brick01 15: option transport-type socket 16: option send-gids true 17: end-volume 18: 19: volume vmstore-client-2 20: type protocol/client 21: option ping-timeout 42 22: option remote-host sjcvhost02 23: option remote-subvolume /export/vmstore/brick01 24: option transport-type socket 25: option send-gids true 26: end-volume 27: 28: volume vmstore-replicate-0 29: type cluster/replicate 30: option arbiter-count 1 31: subvolumes vmstore-client-0 vmstore-client-1 vmstore-client-2 32: end-volume 33: 34: volume vmstore-dht 35: type cluster/distribute 36: subvolumes vmstore-replicate-0 37: end-volume 38: 39: volume vmstore-write-behind 40: type performance/write-behind 41: subvolumes vmstore-dht 42: end-volume 43: 44: volume vmstore-read-ahead 45: type performance/read-ahead 46: subvolumes vmstore-write-behind 47: end-volume 48: 49: volume vmstore-readdir-ahead 50: type performance/readdir-ahead 51: subvolumes vmstore-read-ahead 52: end-volume 53: 54: volume vmstore-io-cache 55: type performance/io-cache 56: subvolumes vmstore-readdir-ahead 57: end-volume 58: 59: volume vmstore-quick-read 60: type performance/quick-read 61: subvolumes vmstore-io-cache 62: end-volume 63: 64: volume vmstore-open-behind 65: type performance/open-behind 66: subvolumes vmstore-quick-read 67: end-volume 68: 69: volume vmstore-md-cache 70: type performance/md-cache 71: subvolumes vmstore-open-behind 72: end-volume 73: 74: volume vmstore 75: type debug/io-stats 76: option latency-measurement off 77: option count-fop-hits off 78: subvolumes vmstore-md-cache 79: end-volume 80: 81: volume meta-autoload 82: type meta 83: subvolumes vmstore 84: end-volume 85: +------------------------------------------------------------------------------+ [2015-09-22 05:30:57.934021] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-vmstore-client-0: changing port to 49153 (from 0) [2015-09-22 05:30:57.934145] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-vmstore-client-1: changing port to 49153 (from 0) [2015-09-22 05:30:57.934491] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-vmstore-client-2: changing port to 49153 (from 0) [2015-09-22 05:30:57.942198] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vmstore-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-09-22 05:30:57.942545] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vmstore-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-09-22 05:30:57.942659] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vmstore-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-09-22 05:30:57.942797] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-0: Connected to vmstore-client-0, attached to remote volume '/export/vmstore/brick01'. [2015-09-22 05:30:57.942808] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-09-22 05:30:57.943036] I [MSGID: 108005] [afr-common.c:3998:afr_notify] 0-vmstore-replicate-0: Subvolume 'vmstore-client-0' came back up; going online. [2015-09-22 05:30:57.943078] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-1: Connected to vmstore-client-1, attached to remote volume '/export/vmstore/brick01'. [2015-09-22 05:30:57.943086] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-09-22 05:30:57.943292] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-2: Connected to vmstore-client-2, attached to remote volume '/export/vmstore/brick01'. [2015-09-22 05:30:57.943302] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-2: Server and Client lk-version numbers are not same, reopening the fds [2015-09-22 05:30:57.953887] I [fuse-bridge.c:5053:fuse_graph_setup] 0-fuse: switched to graph 0 [2015-09-22 05:30:57.954071] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-0: Server lk version = 1 [2015-09-22 05:30:57.954105] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-1: Server lk version = 1 [2015-09-22 05:30:57.954124] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-2: Server lk version = 1 [2015-09-22 05:30:57.955282] I [fuse-bridge.c:3979:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22 [2015-09-22 05:30:57.955738] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2015-09-22 05:30:57.970232] I [fuse-bridge.c:4900:fuse_thread_proc] 0-fuse: unmounting /rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore [2015-09-22 05:30:57.970834] W [glusterfsd.c:1219:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7df5) [0x7f187139fdf5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f1872a09785] -->/usr/sbin/glusterfs(cleanup_and_exit+0x69) [0x7f1872a09609] ) 0-: received signum (15), shutting down [2015-09-22 05:30:57.970848] I [fuse-bridge.c:5595:fini] 0-fuse: Unmounting '/rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore'. [2015-09-22 05:30:58.420973] I [fuse-bridge.c:4900:fuse_thread_proc] 0-fuse: unmounting /rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore [2015-09-22 05:30:58.421355] W [glusterfsd.c:1219:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7df5) [0x7f8267cd4df5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f826933e785] -->/usr/sbin/glusterfs(cleanup_and_exit+0x69) [0x7f826933e609] ) 0-: received signum (15), shutting down [2015-09-22 05:30:58.421369] I [fuse-bridge.c:5595:fini] 0-fuse: Unmounting '/rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore'. [2015-09-22 05:31:09.534410] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.4 (args: /usr/sbin/glusterfs --volfile-server=sjcvhost02 --volfile-server=sjcstorage01 --volfile-server=sjcstorage02 --volfile-id=/vmstore /rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore) [2015-09-22 05:31:09.545686] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-09-22 05:31:09.553019] I [MSGID: 114020] [client.c:2118:notify] 0-vmstore-client-0: parent translators are ready, attempting connect on transport [2015-09-22 05:31:09.555552] I [MSGID: 114020] [client.c:2118:notify] 0-vmstore-client-1: parent translators are ready, attempting connect on transport [2015-09-22 05:31:09.557989] I [MSGID: 114020] [client.c:2118:notify] 0-vmstore-client-2: parent translators are ready, attempting connect on transport Final graph: +------------------------------------------------------------------------------+ 1: volume vmstore-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host sjcstorage01 5: option remote-subvolume /export/vmstore/brick01 6: option transport-type socket 7: option send-gids true 8: end-volume 9: 10: volume vmstore-client-1 11: type protocol/client 12: option ping-timeout 42 13: option remote-host sjcstorage02 14: option remote-subvolume /export/vmstore/brick01 15: option transport-type socket 16: option send-gids true 17: end-volume 18: 19: volume vmstore-client-2 20: type protocol/client 21: option ping-timeout 42 22: option remote-host sjcvhost02 23: option remote-subvolume /export/vmstore/brick01 24: option transport-type socket 25: option send-gids true 26: end-volume 27: 28: volume vmstore-replicate-0 29: type cluster/replicate 30: option arbiter-count 1 31: subvolumes vmstore-client-0 vmstore-client-1 vmstore-client-2 32: end-volume 33: 34: volume vmstore-dht 35: type cluster/distribute 36: subvolumes vmstore-replicate-0 37: end-volume 38: 39: volume vmstore-write-behind 40: type performance/write-behind 41: subvolumes vmstore-dht 42: end-volume 43: 44: volume vmstore-read-ahead 45: type performance/read-ahead 46: subvolumes vmstore-write-behind 47: end-volume 48: 49: volume vmstore-readdir-ahead 50: type performance/readdir-ahead 51: subvolumes vmstore-read-ahead 52: end-volume 53: 54: volume vmstore-io-cache 55: type performance/io-cache 56: subvolumes vmstore-readdir-ahead 57: end-volume 58: 59: volume vmstore-quick-read 60: type performance/quick-read 61: subvolumes vmstore-io-cache 62: end-volume 63: 64: volume vmstore-open-behind 65: type performance/open-behind 66: subvolumes vmstore-quick-read 67: end-volume 68: 69: volume vmstore-md-cache 70: type performance/md-cache 71: subvolumes vmstore-open-behind 72: end-volume 73: 74: volume vmstore 75: type debug/io-stats 76: option latency-measurement off 77: option count-fop-hits off 78: subvolumes vmstore-md-cache 79: end-volume 80: 81: volume meta-autoload 82: type meta 83: subvolumes vmstore 84: end-volume 85: +------------------------------------------------------------------------------+ [2015-09-22 05:31:09.563262] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-vmstore-client-0: changing port to 49153 (from 0) [2015-09-22 05:31:09.563431] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-vmstore-client-1: changing port to 49153 (from 0) [2015-09-22 05:31:09.563877] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-vmstore-client-2: changing port to 49153 (from 0) [2015-09-22 05:31:09.572443] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vmstore-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-09-22 05:31:09.572599] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vmstore-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-09-22 05:31:09.572742] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vmstore-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-09-22 05:31:09.573165] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-1: Connected to vmstore-client-1, attached to remote volume '/export/vmstore/brick01'. [2015-09-22 05:31:09.573186] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-09-22 05:31:09.573395] I [MSGID: 108005] [afr-common.c:3998:afr_notify] 0-vmstore-replicate-0: Subvolume 'vmstore-client-1' came back up; going online. [2015-09-22 05:31:09.573427] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-0: Connected to vmstore-client-0, attached to remote volume '/export/vmstore/brick01'. [2015-09-22 05:31:09.573435] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-09-22 05:31:09.573754] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vmstore-client-2: Connected to vmstore-client-2, attached to remote volume '/export/vmstore/brick01'. [2015-09-22 05:31:09.573783] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vmstore-client-2: Server and Client lk-version numbers are not same, reopen: [2015-09-22 05:31:09.577192] I [fuse-bridge.c:5053:fuse_graph_setup] 0-fuse: switched to graph 0 [2015-09-22 05:31:09.577302] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-1: Server lk version = 1 [2015-09-22 05:31:09.577325] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-0: Server lk version = 1 [2015-09-22 05:31:09.577339] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vmstore-client-2: Server lk version = 1 [2015-09-22 05:31:09.578125] I [fuse-bridge.c:3979:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22 [2015-09-22 05:31:09.578636] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2015-09-22 05:31:10.073698] I [fuse-bridge.c:4900:fuse_thread_proc] 0-fuse: unmounting /rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore [2015-09-22 05:31:10.073977] W [glusterfsd.c:1219:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7df5) [0x7f6b9ba88df5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f6b9d0f2785] -->/usr/sbin/glusterfs(cleanup_and_exit+0x69) [0x7f6b9d0f2609] ) 0-: received signum (15), shutting down [2015-09-22 05:31:10.073993] I [fuse-bridge.c:5595:fini] 0-fuse: Unmounting '/rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore'. [2015-09-22 05:31:20.184700] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.4 (args: /usr/sbin/glusterfs --volfile-server=sjcvhost02 --volfile-server=sjcstorage01 --volfile-server=sjcstorage02 --volfile-id=/vmstore /rhev/data-center/mnt/glusterSD/sjcvhost02:_vmstore) [2015-09-22 05:31:20.194928] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-09-22 05:31:20.200701] I [MSGID: 114020] [client.c:2118:notify] 0-vmstore-client-0: parent translators are ready, attempting connect on transport [2015-09-22 05:31:20.203110] I [MSGID: 114020] [client.c:2118:notify] 0-vmstore-client-1: parent translators are ready, attempting connect on transport [2015-09-22 05:31:20.205708] I [MSGID: 114020] [client.c:2118:notify] 0-vmstore-client-2: parent translators are ready, attempting connect on transport Final graph: Hope this helps. thanks again Brett Stevens </div> </div> <div class="gmail_extra"> <div class="gmail_quote">On Tue, Sep 22, 2015 at 10:14 PM, Sahina Bose <<a moz-do-not-send="true" href="mailto:sabose@redhat.com" target="_blank">sabose(a)redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div text="#000000" bgcolor="#FFFFFF"> <div>On 09/22/2015 02:17 PM, Brett Stevens wrote: </div> <blockquote type="cite"> <div dir="ltr">Hi. First time on the lists. I've searched for this but no luck so sorry if this has been covered before. <div> </div> <div>Im working with the latest 3.6 beta with the following infrastructure. </div> <div> </div> <div>1 management host (to be used for a number of tasks so chose not to use self hosted, we are a school and will need to keep an eye on hardware costs)</div> <div>2 compute nodes</div> <div>2 gluster nodes</div> <div> </div> <div>so far built one gluster volume using the gluster cli to give me 2 nodes and one arbiter node (management host)</div> <div> </div> <div>so far, every time I create a volume, it shows up strait away on the ovirt gui. however no matter what I try, I cannot create or import it as a data domain. </div> <div> </div> <div>the current error in the ovirt gui is "Error while executing action AddGlusterFsStorageDomain: Error creating a storage domain's metadata"</div> </div> </blockquote> Please provide vdsm and gluster logs <blockquote type="cite"> <div dir="ltr"> <div> </div> <div>logs, continuously rolling the following errors around</div> <div> Scheduler_Worker-53) [] START, GlusterVolumesListVDSCommand(HostName = sjcstorage02, GlusterVolumesListVDSParameters:{runAsync='true', hostId='c75682ba-1e4c-42a3-85c7-16e4bb2ce5da'}), log id: 24198fbf 2015-09-22 03:57:29,903 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-53) [] Could not associate brick 'sjcstorage01:/export/vmstore/brick01' of volume '878a316d-2394-4aae-bdf8-e10eea38225e' with correct network as no gluster network found in cluster 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' </div> </div> </blockquote> What is the hostname provided in ovirt engine for sjcstorage01 ? Does this host have multiple nics? Could you provide output of gluster volume info? Please note, that these errors are not related to error in creating storage domain. However, these errors could prevent you from monitoring the state of gluster volume from oVirt <blockquote type="cite"> <div dir="ltr"> <div> 2015-09-22 03:57:29,905 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-53) [] Could not associate brick 'sjcstorage02:/export/vmstore/brick01' of volume '878a316d-2394-4aae-bdf8-e10eea38225e' with correct network as no gluster network found in cluster 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' 2015-09-22 03:57:29,905 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-53) [] Could not add brick 'sjcvhost02:/export/vmstore/brick01' to volume '878a316d-2394-4aae-bdf8-e10eea38225e' - server uuid '29b58278-9aa3-47c5-bfb4-1948ef7fdbba' not found in cluster 'b00d3c6d-fdfb-49e8-9f1a-f749c3d42486' 2015-09-22 03:57:29,905 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-53) [] FINISH, GlusterVolumesListVDSCommand, return: {878a316d-2394-4aae-bdf8-e10eea38225e=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@41e93fb1}, log id: 24198fbf I'm new to ovirt and gluster, so any help would be great thanks Brett Stevens </div> </div> <fieldset></fieldset> <pre>_______________________________________________ Users mailing list <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users(a)ovirt.org</a> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> </div> </blockquote> </div> </div> </blockquote> </body> </html> --------------090502070204020205010802--

1 0