</div><div class=3D"yahoo_quoted" id=3D"yui_3_16_0_ym19_1_1463183424286_15= 425" style=3D"display: block;"><div style=3D"font-family: Courier New, cour= ier, monaco, monospace, sans-serif; font-size: 13px;" id=3D"yui_3_16_0_ym19= _1_1463183424286_15424"><div style=3D"font-family: HelveticaNeue, Helvetica= Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;" id=3D= "yui_3_16_0_ym19_1_1463183424286_15423"><div class=3D"y_msg_container" id= =3D"yui_3_16_0_ym19_1_1463183424286_15422"><div id=3D"yiv1436813381"><div i= d=3D"yui_3_16_0_ym19_1_1463183424286_15421"><div style=3D"font-family: 'Cou= rier New', courier, monaco, monospace, sans-serif; font-size: 13px;" id=3D"= yui_3_16_0_ym19_1_1463183424286_15420"><div class=3D"yiv1436813381yqt809646= 6043" id=3D"yiv1436813381yqt34582"><div id=3D"yui_3_16_0_ym19_1_14631834242= 86_15426"><div id=3D"yiv1436813381yui_3_16_0_ym19_1_1463183424286_7642" sty= le=3D"font-family:Courier New, courier, monaco, monospace, sans-serif;font-= size:13px;"><div id=3D"yiv1436813381yui_3_16_0_ym19_1_1463183424286_7641" s= tyle=3D"font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida= Grande, sans-serif;font-size:16px;"><div class=3D"yiv1436813381y_msg_conta= iner" id=3D"yiv1436813381yui_3_16_0_ym19_1_1463183424286_7640"><br></div><d= iv class=3D"yiv1436813381y_msg_container" id=3D"yiv1436813381yui_3_16_0_ym1= 9_1_1463183424286_7640">San lock.log:</div><div class=3D"yiv1436813381y_msg= _container" id=3D"yiv1436813381yui_3_16_0_ym19_1_1463183424286_7640"><div s= tyle=3D"font-size: 12px; line-height: normal; font-family: Menlo;" id=3D"yu= i_3_16_0_ym19_1_1463183424286_15920"><span style=3D"font-variant-ligatures:= no-common-ligatures" id=3D"yui_3_16_0_ym19_1_1463183424286_15921">2016-05-= 13 22:09:58-0400 4283 [16627]: cmd 9 target pid 17793 not found</span></div= <div style=3D"font-size: 12px; line-height: normal; font-family: Menlo;" i= d=3D"yui_3_16_0_ym19_1_1463183424286_15843"> </div><div style=3D"font-size: 12px; line-height: normal; font-family: Menl= o;" dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_1463183424286_15922"><span style=3D= "font-variant-ligatures: no-common-ligatures" id=3D"yui_3_16_0_ym19_1_14631= 83424286_15923">2016-05-13 22:09:58-0400 4284 [16632]: r4 cmd_acquire 2,10,= 17793 <span style=3D"background-color: rgb(253, 239, 43);" id=3D"yui_3_16_0= _ym19_1_1463183424286_16445">invalid lockspace found</span> -1 failed 0 nam= e 50816073-c2eb-4d9d-be72-30ad62bf85f2</span></div></div></div></div></div>= </div></div></div></div><br></div><div class=3D"y_msg_container" id=3D"yui_= 3_16_0_ym19_1_1463183424286_15422" dir=3D"ltr">I did cluster heal. Fi= le systems seems to be fine. I see this lockspace file with hosted-en= gine. Does it have anything do with the issue?</div><div class=3D"y_m= sg_container" id=3D"yui_3_16_0_ym19_1_1463183424286_15422" dir=3D"ltr"><br = id=3D"yui_3_16_0_ym19_1_1463183424286_16510"></div><div class=3D"y_msg_cont= ainer" id=3D"yui_3_16_0_ym19_1_1463183424286_15422" dir=3D"ltr"><div style= =3D"font-size: 12px; line-height: normal; font-family: Menlo;" id=3D"yui_3_= 16_0_ym19_1_1463183424286_16502"><span style=3D"font-variant-ligatures: no-= common-ligatures" id=3D"yui_3_16_0_ym19_1_1463183424286_16503"># find /glus= ter -name *lock*</span></div><div style=3D"font-size: 12px; line-height: no= rmal; font-family: Menlo;" id=3D"yui_3_16_0_ym19_1_1463183424286_16062"> </div><div style=3D"font-size: 12px; line-height: normal; font-family: Menl= o;" id=3D"yui_3_16_0_ym19_1_1463183424286_16504"><span style=3D"font-varian= t-ligatures: no-common-ligatures" id=3D"yui_3_16_0_ym19_1_1463183424286_165= 05">/gluster/engine/brick1/50816073-c2eb-4d9d-be72-30ad62bf85f2/ha_agent/<s=
------=_Part_1864187_991623925.1463192625760 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi all, There was an issue with sudoers syntax. =C2=A0Vdsm didn't start correctly s= o 54321 wasn't available.After fixing it, I got some errors on locks. =C2= =A0I'm not sure how this should be fixed correctly. Thread-49::DEBUG::2016-05-13 22:02:49,689::libvirtconnection::143::root::(w= rapper) Unknown libvirterror: ecode: 38 edom: 42 level: 2 message: Failed t= o acquire lock: No space left on deviceThread-49::DEBUG::2016-05-13 22:02:4= 9,690::vm::2287::vm.Vm::(_startUnderlyingVm) vmId=3D`bd48bba5-4109-4a58-ac8= 9-832add1f4de4`::_ongoingCreations releasedThread-49::ERROR::2016-05-13 22:= 02:49,690::vm::2324::vm.Vm::(_startUnderlyingVm) vmId=3D`bd48bba5-4109-4a58= -ac89-832add1f4de4`::The vm start process failedTraceback (most recent call= last):=C2=A0 File "/usr/share/vdsm/virt/vm.py", line 2264, in _startUnderl= yingVm=C2=A0 =C2=A0 self._run()=C2=A0 File "/usr/share/vdsm/virt/vm.py", li= ne 3328, in _run=C2=A0 =C2=A0 self._connection.createXML(domxml, flags),=C2= =A0 File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line= 111, in wrapper=C2=A0 =C2=A0 ret =3D f(*args, **kwargs)=C2=A0 File "/usr/l= ib64/python2.6/site-packages/libvirt.py", line 2709, in createXML=C2=A0 =C2= =A0 if ret is None:raise libvirtError('virDomainCreateXML() failed', conn= =3Dself)libvirtError: Failed to acquire lock: No space left on deviceThread= -49::DEBUG::2016-05-13 22:02:49,693::vm::2779::vm.Vm::(setDownStatus) vmId= =3D`bd48bba5-4109-4a58-ac89-832add1f4de4`::Changed state to Down: Failed to= acquire lock: No space left on device (code=3D1) San lock.log:2016-05-13 22:09:58-0400 4283 [16627]: cmd 9 target pid 17793 = not found2016-05-13 22:09:58-0400 4284 [16632]: r4 cmd_acquire 2,10,17793 i= nvalid lockspace found -1 failed 0 name 50816073-c2eb-4d9d-be72-30ad62bf85f= 2 I did cluster heal. =C2=A0File systems seems to be fine. =C2=A0I see this l= ockspace file with hosted-engine. =C2=A0Does it have anything do with the i= ssue? # find /gluster -name *lock*/gluster/engine/brick1/50816073-c2eb-4d9d-be72-= 30ad62bf85f2/ha_agent/hosted-engine.lockspace ThanksWill ------=_Part_1864187_991623925.1463192625760 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <html><head></head><body><div style=3D"color:#000; background-color:#fff; f= ont-family:Courier New, courier, monaco, monospace, sans-serif;font-size:13= px"><div id=3D"yui_3_16_0_ym19_1_1463183424286_15400">Hi all,</div><div id= =3D"yui_3_16_0_ym19_1_1463183424286_15400"><br></div><div id=3D"yui_3_16_0_= ym19_1_1463183424286_15400">There was an issue with sudoers syntax. V= dsm didn't start correctly so 54321 wasn't available.</div><div id=3D"yui_3= _16_0_ym19_1_1463183424286_15400">After fixing it, I got some errors on loc= ks. I'm not sure how this should be fixed correctly.</div><div id=3D"= yui_3_16_0_ym19_1_1463183424286_15400"><br></div><div style=3D"margin-top: = 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family:= Menlo;" id=3D"yui_3_16_0_ym19_1_1463183424286_15694"><span style=3D"font-v= ariant-ligatures: no-common-ligatures" id=3D"yui_3_16_0_ym19_1_146318342428= 6_15695">Thread-49::DEBUG::2016-05-13 22:02:49,689::libvirtconnection::143:= :root::(wrapper) Unknown libvirterror: ecode: 38 edom: 42 level: 2 message:= <font id=3D"yui_3_16_0_ym19_1_1463183424286_16364" style=3D"background-col= or: rgb(253, 239, 43);">Failed to acquire lock: No space left on device</fo= nt></span></div><div style=3D"margin-top: 0px; margin-bottom: 0px; font-siz= e: 12px; line-height: normal; font-family: Menlo;" id=3D"yui_3_16_0_ym19_1_= 1463183424286_15696"><span style=3D"font-variant-ligatures: no-common-ligat= ures" id=3D"yui_3_16_0_ym19_1_1463183424286_15697">Thread-49::DEBUG::2016-0= 5-13 22:02:49,690::vm::2287::vm.Vm::(_startUnderlyingVm) vmId=3D`bd48bba5-4= 109-4a58-ac89-832add1f4de4`::_ongoingCreations released</span></div><div st= yle=3D"margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: n= ormal; font-family: Menlo;" id=3D"yui_3_16_0_ym19_1_1463183424286_15698"><s= pan style=3D"font-variant-ligatures: no-common-ligatures" id=3D"yui_3_16_0_= ym19_1_1463183424286_15699">Thread-49::ERROR::2016-05-13 22:02:49,690::vm::= 2324::vm.Vm::(_startUnderlyingVm) vmId=3D`bd48bba5-4109-4a58-ac89-832add1f4= de4`::The vm start process failed</span></div><div style=3D"margin-top: 0px= ; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Me= nlo;" id=3D"yui_3_16_0_ym19_1_1463183424286_15700"><span style=3D"font-vari= ant-ligatures: no-common-ligatures" id=3D"yui_3_16_0_ym19_1_1463183424286_1= 5701">Traceback (most recent call last):</span></div><div style=3D"margin-t= op: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-fam= ily: Menlo;" id=3D"yui_3_16_0_ym19_1_1463183424286_15702"><span style=3D"fo= nt-variant-ligatures: no-common-ligatures" id=3D"yui_3_16_0_ym19_1_14631834= 24286_15703"> File "/usr/share/vdsm/virt/vm.py", line 2264, in _start= UnderlyingVm</span></div><div style=3D"margin-top: 0px; margin-bottom: 0px;= font-size: 12px; line-height: normal; font-family: Menlo;" id=3D"yui_3_16_= 0_ym19_1_1463183424286_15704"><span style=3D"font-variant-ligatures: no-com= mon-ligatures" id=3D"yui_3_16_0_ym19_1_1463183424286_15705"> s= elf._run()</span></div><div style=3D"margin-top: 0px; margin-bottom: 0px; f= ont-size: 12px; line-height: normal; font-family: Menlo;" id=3D"yui_3_16_0_= ym19_1_1463183424286_15706"><span style=3D"font-variant-ligatures: no-commo= n-ligatures" id=3D"yui_3_16_0_ym19_1_1463183424286_15707"> File "/usr= /share/vdsm/virt/vm.py", line 3328, in _run</span></div><div style=3D"margi= n-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-= family: Menlo;" id=3D"yui_3_16_0_ym19_1_1463183424286_15708"><span style=3D= "font-variant-ligatures: no-common-ligatures" id=3D"yui_3_16_0_ym19_1_14631= 83424286_15709"> self._connection.createXML(domxml, flags),</s= pan></div><div style=3D"margin-top: 0px; margin-bottom: 0px; font-size: 12p= x; line-height: normal; font-family: Menlo;" id=3D"yui_3_16_0_ym19_1_146318= 3424286_15710"><span style=3D"font-variant-ligatures: no-common-ligatures" = id=3D"yui_3_16_0_ym19_1_1463183424286_15711"> File "/usr/lib/python2.= 6/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper</span></di= v><div style=3D"margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-= height: normal; font-family: Menlo;" id=3D"yui_3_16_0_ym19_1_1463183424286_= 15712"><span style=3D"font-variant-ligatures: no-common-ligatures" id=3D"yu= i_3_16_0_ym19_1_1463183424286_15713"> ret =3D f(*args, **kwarg= s)</span></div><div style=3D"margin-top: 0px; margin-bottom: 0px; font-size= : 12px; line-height: normal; font-family: Menlo;" id=3D"yui_3_16_0_ym19_1_1= 463183424286_15714"><span style=3D"font-variant-ligatures: no-common-ligatu= res" id=3D"yui_3_16_0_ym19_1_1463183424286_15715"> File "/usr/lib64/p= ython2.6/site-packages/libvirt.py", line 2709, in createXML</span></div><di= v style=3D"margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-heigh= t: normal; font-family: Menlo;" id=3D"yui_3_16_0_ym19_1_1463183424286_15716= "><span style=3D"font-variant-ligatures: no-common-ligatures" id=3D"yui_3_1= 6_0_ym19_1_1463183424286_15717"> if ret is None:raise libvirtE= rror('virDomainCreateXML() failed', conn=3Dself)</span></div><div style=3D"= margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; = font-family: Menlo;" id=3D"yui_3_16_0_ym19_1_1463183424286_15718"><span sty= le=3D"font-variant-ligatures: no-common-ligatures" id=3D"yui_3_16_0_ym19_1_= 1463183424286_15719">libvirtError: Failed to acquire lock: No space left on= device</span></div><div id=3D"yui_3_16_0_ym19_1_1463183424286_15400"> </div><div style=3D"margin-top: 0px; margin-bottom: 0px; font-size: 12px; l= ine-height: normal; font-family: Menlo;" id=3D"yui_3_16_0_ym19_1_1463183424= 286_15720"><span style=3D"font-variant-ligatures: no-common-ligatures" id= =3D"yui_3_16_0_ym19_1_1463183424286_15721">Thread-49::DEBUG::2016-05-13 22:= 02:49,693::vm::2779::vm.Vm::(setDownStatus) vmId=3D`bd48bba5-4109-4a58-ac89= -832add1f4de4`::Changed state to Down: Failed to acquire lock: No space lef= t on device (code=3D1)</span></div><div dir=3D"ltr" id=3D"yui_3_16_0_ym19_1= _1463183424286_15722"><span style=3D"font-variant-ligatures: no-common-liga= tures" id=3D"yui_3_16_0_ym19_1_1463183424286_15723"><br id=3D"yui_3_16_0_ym= 19_1_1463183424286_15724"></span></div><div class=3D"qtdSeparateBR"><br><br= pan style=3D"background-color: rgb(253, 239, 43);" id=3D"yui_3_16_0_ym19_1_= 1463183424286_16581">hosted-engine.lockspace</span></span></div><div dir=3D= "ltr" id=3D"yui_3_16_0_ym19_1_1463183424286_16506"><span style=3D"font-vari= ant-ligatures: no-common-ligatures" id=3D"yui_3_16_0_ym19_1_1463183424286_1= 6507"><br id=3D"yui_3_16_0_ym19_1_1463183424286_16508"></span></div><div st= yle=3D"font-size: 12px; line-height: normal; font-family: Menlo;" dir=3D"lt= r" id=3D"yui_3_16_0_ym19_1_1463183424286_16064"><span style=3D"font-variant= -ligatures: no-common-ligatures"><br></span></div><div style=3D"font-size: = 12px; line-height: normal; font-family: Menlo;" dir=3D"ltr" id=3D"yui_3_16_= 0_ym19_1_1463183424286_16064"><span style=3D"font-variant-ligatures: no-com= mon-ligatures"><br></span></div><div style=3D"font-size: 12px; line-height:= normal; font-family: Menlo;" dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_146318342= 4286_16064"><span style=3D"font-variant-ligatures: no-common-ligatures" id= =3D"yui_3_16_0_ym19_1_1463183424286_16067">Thanks</span></div><div style=3D= "font-size: 12px; line-height: normal; font-family: Menlo;" dir=3D"ltr" id= =3D"yui_3_16_0_ym19_1_1463183424286_16064"><span style=3D"font-variant-liga= tures: no-common-ligatures;" id=3D"yui_3_16_0_ym19_1_1463183424286_16283">W= ill</span></div></div> </div> </div> </div></div></body></html> ------=_Part_1864187_991623925.1463192625760--