<html><head></head><body><div style="color:#000; background-color:#fff; font-family:Courier New, courier, monaco, monospace, sans-serif;font-size:13px"><div id="yui_3_16_0_ym19_1_1463183424286_15400">Hi all,</div><div id="yui_3_16_0_ym19_1_1463183424286_15400"><br></div><div id="yui_3_16_0_ym19_1_1463183424286_15400">There was an issue with sudoers syntax. Vdsm didn't start correctly so 54321 wasn't available.</div><div id="yui_3_16_0_ym19_1_1463183424286_15400">After fixing it, I got some errors on locks. I'm not sure how this should be fixed correctly.</div><div id="yui_3_16_0_ym19_1_1463183424286_15400"><br></div><div style="margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_15694"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15695">Thread-49::DEBUG::2016-05-13 22:02:49,689::libvirtconnection::143::root::(wrapper) Unknown libvirterror: ecode: 38 edom: 42 level: 2 message: <font id="yui_3_16_0_ym19_1_1463183424286_16364" style="background-color: rgb(253, 239, 43);">Failed to acquire lock: No space left on device</font></span></div><div style="margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_15696"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15697">Thread-49::DEBUG::2016-05-13 22:02:49,690::vm::2287::vm.Vm::(_startUnderlyingVm) vmId=`bd48bba5-4109-4a58-ac89-832add1f4de4`::_ongoingCreations released</span></div><div style="margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_15698"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15699">Thread-49::ERROR::2016-05-13 22:02:49,690::vm::2324::vm.Vm::(_startUnderlyingVm) vmId=`bd48bba5-4109-4a58-ac89-832add1f4de4`::The vm start process failed</span></div><div style="margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_15700"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15701">Traceback (most recent call last):</span></div><div style="margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_15702"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15703"> File "/usr/share/vdsm/virt/vm.py", line 2264, in _startUnderlyingVm</span></div><div style="margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_15704"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15705"> self._run()</span></div><div style="margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_15706"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15707"> File "/usr/share/vdsm/virt/vm.py", line 3328, in _run</span></div><div style="margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_15708"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15709"> self._connection.createXML(domxml, flags),</span></div><div style="margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_15710"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15711"> File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper</span></div><div style="margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_15712"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15713"> ret = f(*args, **kwargs)</span></div><div style="margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_15714"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15715"> File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2709, in createXML</span></div><div style="margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_15716"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15717"> if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)</span></div><div style="margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_15718"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15719">libvirtError: Failed to acquire lock: No space left on device</span></div><div id="yui_3_16_0_ym19_1_1463183424286_15400">
</div><div style="margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_15720"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15721">Thread-49::DEBUG::2016-05-13 22:02:49,693::vm::2779::vm.Vm::(setDownStatus) vmId=`bd48bba5-4109-4a58-ac89-832add1f4de4`::Changed state to Down: Failed to acquire lock: No space left on device (code=1)</span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1463183424286_15722"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15723"><br id="yui_3_16_0_ym19_1_1463183424286_15724"></span></div><div class="qtdSeparateBR"><br><br></div><div class="yahoo_quoted" id="yui_3_16_0_ym19_1_1463183424286_15425" style="display: block;"><div style="font-family: Courier New, courier, monaco, monospace, sans-serif; font-size: 13px;" id="yui_3_16_0_ym19_1_1463183424286_15424"><div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;" id="yui_3_16_0_ym19_1_1463183424286_15423"><div class="y_msg_container" id="yui_3_16_0_ym19_1_1463183424286_15422"><div id="yiv1436813381"><div id="yui_3_16_0_ym19_1_1463183424286_15421"><div style="font-family: 'Courier New', courier, monaco, monospace, sans-serif; font-size: 13px;" id="yui_3_16_0_ym19_1_1463183424286_15420"><div class="yiv1436813381yqt8096466043" id="yiv1436813381yqt34582"><div id="yui_3_16_0_ym19_1_1463183424286_15426"><div id="yiv1436813381yui_3_16_0_ym19_1_1463183424286_7642" style="font-family:Courier New, courier, monaco, monospace, sans-serif;font-size:13px;"><div id="yiv1436813381yui_3_16_0_ym19_1_1463183424286_7641" style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;"><div class="yiv1436813381y_msg_container" id="yiv1436813381yui_3_16_0_ym19_1_1463183424286_7640"><br></div><div class="yiv1436813381y_msg_container" id="yiv1436813381yui_3_16_0_ym19_1_1463183424286_7640">San lock.log:</div><div class="yiv1436813381y_msg_container" id="yiv1436813381yui_3_16_0_ym19_1_1463183424286_7640"><div style="font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_15920"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15921">2016-05-13 22:09:58-0400 4283 [16627]: cmd 9 target pid 17793 not found</span></div><div style="font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_15843">
</div><div style="font-size: 12px; line-height: normal; font-family: Menlo;" dir="ltr" id="yui_3_16_0_ym19_1_1463183424286_15922"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_15923">2016-05-13 22:09:58-0400 4284 [16632]: r4 cmd_acquire 2,10,17793 <span style="background-color: rgb(253, 239, 43);" id="yui_3_16_0_ym19_1_1463183424286_16445">invalid lockspace found</span> -1 failed 0 name 50816073-c2eb-4d9d-be72-30ad62bf85f2</span></div></div></div></div></div></div></div></div></div><br></div><div class="y_msg_container" id="yui_3_16_0_ym19_1_1463183424286_15422" dir="ltr">I did cluster heal. File systems seems to be fine. I see this lockspace file with hosted-engine. Does it have anything do with the issue?</div><div class="y_msg_container" id="yui_3_16_0_ym19_1_1463183424286_15422" dir="ltr"><br id="yui_3_16_0_ym19_1_1463183424286_16510"></div><div class="y_msg_container" id="yui_3_16_0_ym19_1_1463183424286_15422" dir="ltr"><div style="font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_16502"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_16503"># find /gluster -name *lock*</span></div><div style="font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_16062">
</div><div style="font-size: 12px; line-height: normal; font-family: Menlo;" id="yui_3_16_0_ym19_1_1463183424286_16504"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_16505">/gluster/engine/brick1/50816073-c2eb-4d9d-be72-30ad62bf85f2/ha_agent/<span style="background-color: rgb(253, 239, 43);" id="yui_3_16_0_ym19_1_1463183424286_16581">hosted-engine.lockspace</span></span></div><div dir="ltr" id="yui_3_16_0_ym19_1_1463183424286_16506"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_16507"><br id="yui_3_16_0_ym19_1_1463183424286_16508"></span></div><div style="font-size: 12px; line-height: normal; font-family: Menlo;" dir="ltr" id="yui_3_16_0_ym19_1_1463183424286_16064"><span style="font-variant-ligatures: no-common-ligatures"><br></span></div><div style="font-size: 12px; line-height: normal; font-family: Menlo;" dir="ltr" id="yui_3_16_0_ym19_1_1463183424286_16064"><span style="font-variant-ligatures: no-common-ligatures"><br></span></div><div style="font-size: 12px; line-height: normal; font-family: Menlo;" dir="ltr" id="yui_3_16_0_ym19_1_1463183424286_16064"><span style="font-variant-ligatures: no-common-ligatures" id="yui_3_16_0_ym19_1_1463183424286_16067">Thanks</span></div><div style="font-size: 12px; line-height: normal; font-family: Menlo;" dir="ltr" id="yui_3_16_0_ym19_1_1463183424286_16064"><span style="font-variant-ligatures: no-common-ligatures;" id="yui_3_16_0_ym19_1_1463183424286_16283">Will</span></div></div> </div> </div> </div></div></body></html>