Re: [ovirt-users] Failed to create volume in OVirt with gluster

From: "Punit Dambiwal" <hypunit(a)gmail.com&gt; To: "Kanagaraj" <kmayilsa(a)redhat.com&gt; Cc: "Martin Pavlík" <mpavlik(a)redhat.com&gt;, "Vijay Bellur" <vbellur(a)redhat.com&gt;, "Kaushal M" <kshlmster(a)gmail.com&gt;, users(a)ovirt.org, gluster-users(a)gluster.org Sent: Monday, January 12, 2015 3:36:43 PM Subject: Re: Failed to create volume in OVirt with gluster Hi Kanagaraj, Please find the logs from here :- http://ur1.ca/jeszc [image: Inline image 1] [image: Inline image 2] On Mon, Jan 12, 2015 at 1:02 PM, Kanagaraj <kmayilsa(a)redhat.com&gt; wrote: > Looks like there are some failures in gluster. > Can you send the log output from glusterd log file from the relevant hosts? > > Thanks, > Kanagaraj > > > On 01/12/2015 10:24 AM, Punit Dambiwal wrote: > > Hi, > > Is there any one from gluster can help me here :- > > Engine logs :- > > 2015-01-12 12:50:33,841 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (DefaultQuartzScheduler_Worker-12) Failed to acquire lock and wait lock > EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 > value: GLUSTER > , sharedLocks= ] > 2015-01-12 12:50:34,725 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (DefaultQuartzScheduler_Worker-12) Failed to acquire lock and wait lock > EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 > value: GLUSTER > , sharedLocks= ] > 2015-01-12 12:50:36,824 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (DefaultQuartzScheduler_Worker-12) Failed to acquire lock and wait lock > EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 > value: GLUSTER > , sharedLocks= ] > 2015-01-12 12:50:36,853 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (DefaultQuartzScheduler_Worker-12) Failed to acquire lock and wait lock > EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 > value: GLUSTER > , sharedLocks= ] > 2015-01-12 12:50:36,866 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (DefaultQuartzScheduler_Worker-12) Failed to acquire lock and wait lock > EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 > value: GLUSTER > , sharedLocks= ] > 2015-01-12 12:50:37,751 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (DefaultQuartzScheduler_Worker-12) Failed to acquire lock and wait lock > EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 > value: GLUSTER > , sharedLocks= ] > 2015-01-12 12:50:39,849 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (DefaultQuartzScheduler_Worker-12) Failed to acquire lock and wait lock > EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 > value: GLUSTER > , sharedLocks= ] > 2015-01-12 12:50:39,878 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (DefaultQuartzScheduler_Worker-12) Failed to acquire lock and wait lock > EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 > value: GLUSTER > , sharedLocks= ] > 2015-01-12 12:50:39,890 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (DefaultQuartzScheduler_Worker-12) Failed to acquire lock and wait lock > EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 > value: GLUSTER > , sharedLocks= ] > 2015-01-12 12:50:40,776 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (DefaultQuartzScheduler_Worker-12) Failed to acquire lock and wait lock > EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 > value: GLUSTER > , sharedLocks= ] > 2015-01-12 12:50:42,878 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (DefaultQuartzScheduler_Worker-12) Failed to acquire lock and wait lock > EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 > value: GLUSTER > , sharedLocks= ] > 2015-01-12 12:50:42,903 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (DefaultQuartzScheduler_Worker-12) Failed to acquire lock and wait lock > EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 > value: GLUSTER > , sharedLocks= ] > 2015-01-12 12:50:42,916 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (DefaultQuartzScheduler_Worker-12) Failed to acquire lock and wait lock > EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 > value: GLUSTER > , sharedLocks= ] > 2015-01-12 12:50:43,771 INFO > [org.ovirt.engine.core.vdsbroker.gluster.CreateGlusterVolumeVDSCommand] > (ajp--127.0.0.1-8702-1) [330ace48] FINISH, CreateGlusterVolumeVDSCommand, > log id: 303e70a4 > 2015-01-12 12:50:43,780 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (ajp--127.0.0.1-8702-1) [330ace48] Correlation ID: 330ace48, Job ID: > 896a69b3-a678-40a7-bceb-3635e4062aa0, Call Stack: null, Custom Event ID: > -1, Message: Creation of Gluster Volume vol01 failed. > 2015-01-12 12:50:43,785 INFO > [org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand] > (ajp--127.0.0.1-8702-1) [330ace48] Lock freed to object EngineLock > [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 value: GLUSTER > , sharedLocks= ] > > [image: Inline image 2] > > > On Sun, Jan 11, 2015 at 6:48 PM, Martin Pavlík <mpavlik(a)redhat.com&gt; wrote: > >> Hi Punit, >> >> unfortunately I’am not that good with the gluster, I was just following >> the obvious clue from the log. Could you try on the nodes if the packages >> are even available for installation >> >> yum install gluster-swift gluster-swift-object gluster-swift-plugin >> gluster-swift-account >> gluster-swift-proxy gluster-swift-doc gluster-swift-container >> glusterfs-geo-replication >> >> if not you could try to get them in official gluster repo. >> http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs... >> >> HTH >> >> M. >> >> >> >> >> On 10 Jan 2015, at 04:35, Punit Dambiwal <hypunit(a)gmail.com&gt; wrote: >> >> Hi Martin, >> >> I installed gluster from ovirt repo....is it require to install those >> packages manually ?? >> >> On Fri, Jan 9, 2015 at 7:19 PM, Martin Pavlík <mpavlik(a)redhat.com&gt; wrote: >> >>> Hi Punit, >>> >>> can you verify that nodes contain cluster packages from the following >>> log? >>> >>> Thread-14::DEBUG::2015-01-09 >>> 18:06:28,823::caps::716::root::(_getKeyPackages) rpm package >>> ('gluster-swift',) not found >>> Thread-14::DEBUG::2015-01-09 >>> 18:06:28,825::caps::716::root::(_getKeyPackages) rpm package >>> ('gluster-swift-object',) not found >>> Thread-14::DEBUG::2015-01-09 >>> 18:06:28,826::caps::716::root::(_getKeyPackages) rpm package >>> ('gluster-swift-plugin',) not found >>> Thread-14::DEBUG::2015-01-09 >>> 18:06:28,829::caps::716::root::(_getKeyPackages) rpm package >>> ('gluster-swift-account',) not found >>> Thread-14::DEBUG::2015-01-09 >>> 18:06:28,829::caps::716::root::(_getKeyPackages) rpm package >>> ('gluster-swift-proxy',) not found >>> Thread-14::DEBUG::2015-01-09 >>> 18:06:28,829::caps::716::root::(_getKeyPackages) rpm package >>> ('gluster-swift-doc',) not found >>> Thread-14::DEBUG::2015-01-09 >>> 18:06:28,830::caps::716::root::(_getKeyPackages) rpm package >>> ('gluster-swift-container',) not found >>> Thread-14::DEBUG::2015-01-09 >>> 18:06:28,830::caps::716::root::(_getKeyPackages) rpm package >>> ('glusterfs-geo-replication',) not found >>> >>> >>> M. >>> >>> On 09 Jan 2015, at 11:13, Punit Dambiwal <hypunit(a)gmail.com&gt; wrote: >>> >>> Hi Kanagaraj, >>> >>> Please find the attached logs :- >>> >>> Engine Logs :- http://ur1.ca/jdopt >>> VDSM Logs :- http://ur1.ca/jdoq9 >>> >>> >>> >>> On Thu, Jan 8, 2015 at 6:05 PM, Kanagaraj <kmayilsa(a)redhat.com&gt; wrote: >>> >>>> Do you see any errors in the UI? >>>> >>>> Also please provide the engine.log and vdsm.log when the failure >>>> occured. >>>> >>>> Thanks, >>>> Kanagaraj >>>> >>>> >>>> On 01/08/2015 02:25 PM, Punit Dambiwal wrote: >>>> >>>> Hi Martin, >>>> >>>> The steps are below :- >>>> >>>> 1. Step the ovirt engine on the one server... >>>> 2. Installed centos 7 on 4 host node servers.. >>>> 3. I am using host node (compute+storage)....now i have added all 4 >>>> nodes to engine... >>>> 4. Create the gluster volume from GUI... >>>> >>>> Network :- >>>> eth0 :- public network (1G) >>>> eth1+eth2=bond0= VM public network (1G) >>>> eth3+eth4=bond1=ovirtmgmt+storage (10G private network) >>>> >>>> every hostnode has 24 bricks=24*4(distributed replicated) >>>> >>>> Thanks, >>>> Punit >>>> >>>> >>>> On Thu, Jan 8, 2015 at 3:20 PM, Martin Pavlík <mpavlik(a)redhat.com&gt; >>>> wrote: >>>> >>>>> Hi Punit, >>>>> >>>>> can you please provide also errors from /var/log/vdsm/vdsm.log and >>>>> /var/log/vdsm/vdsmd.log >>>>> >>>>> it would be really helpful if you provided exact steps how to >>>>> reproduce the problem. >>>>> >>>>> regards >>>>> >>>>> Martin Pavlik - rhev QE >>>>> > On 08 Jan 2015, at 03:06, Punit Dambiwal <hypunit(a)gmail.com&gt; wrote: >>>>> > >>>>> > Hi, >>>>> > >>>>> > I try to add gluster volume but it failed... >>>>> > >>>>> > Ovirt :- 3.5 >>>>> > VDSM :- vdsm-4.16.7-1.gitdb83943.el7 >>>>> > KVM :- 1.5.3 - 60.el7_0.2 >>>>> > libvirt-1.1.1-29.el7_0.4 >>>>> > Glusterfs :- glusterfs-3.5.3-1.el7 >>>>> > >>>>> > Engine Logs :- >>>>> > >>>>> > 2015-01-08 09:57:52,569 INFO >>>>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >>>>> (DefaultQuartzScheduler_Worker-16) Failed to acquire lock and wait lock >>>>> EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 >>>>> value: GLUSTER >>>>> > , sharedLocks= ] >>>>> > 2015-01-08 09:57:52,609 INFO >>>>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >>>>> (DefaultQuartzScheduler_Worker-16) Failed to acquire lock and wait lock >>>>> EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 >>>>> value: GLUSTER >>>>> > , sharedLocks= ] >>>>> > 2015-01-08 09:57:55,582 INFO >>>>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >>>>> (DefaultQuartzScheduler_Worker-16) Failed to acquire lock and wait lock >>>>> EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 >>>>> value: GLUSTER >>>>> > , sharedLocks= ] >>>>> > 2015-01-08 09:57:55,591 INFO >>>>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >>>>> (DefaultQuartzScheduler_Worker-16) Failed to acquire lock and wait lock >>>>> EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 >>>>> value: GLUSTER >>>>> > , sharedLocks= ] >>>>> > 2015-01-08 09:57:55,596 INFO >>>>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >>>>> (DefaultQuartzScheduler_Worker-16) Failed to acquire lock and wait lock >>>>> EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 >>>>> value: GLUSTER >>>>> > , sharedLocks= ] >>>>> > 2015-01-08 09:57:55,633 INFO >>>>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >>>>> (DefaultQuartzScheduler_Worker-16) Failed to acquire lock and wait lock >>>>> EngineLock [exclusiveLocks= key: 00000001-0001-0001-0001-000000000300 >>>>> value: GLUSTER >>>>> > , sharedLocks= ] >>>>> > ^C >>>>> > >>>>> > >>>>> >>>>> >>>> >>>> >>> <216 09-Jan-15.jpg><217 09-Jan-15.jpg> >>> >>> >>> >> >> > >