5:23 a.m.
------=_Part_1864187_991623925.1463192625760
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Hi all,
There was an issue with sudoers syntax. =C2=A0Vdsm didn't start correctly s=
o 54321 wasn't available.After fixing it, I got some errors on locks. =C2=
=A0I'm not sure how this should be fixed correctly.
Thread-49::DEBUG::2016-05-13 22:02:49,689::libvirtconnection::143::root::(w=
rapper) Unknown libvirterror: ecode: 38 edom: 42 level: 2 message: Failed t=
o acquire lock: No space left on deviceThread-49::DEBUG::2016-05-13 22:02:4=
9,690::vm::2287::vm.Vm::(_startUnderlyingVm) vmId=3D`bd48bba5-4109-4a58-ac8=
9-832add1f4de4`::_ongoingCreations releasedThread-49::ERROR::2016-05-13 22:=
02:49,690::vm::2324::vm.Vm::(_startUnderlyingVm) vmId=3D`bd48bba5-4109-4a58=
-ac89-832add1f4de4`::The vm start process failedTraceback (most recent call=
last):=C2=A0 File "/usr/share/vdsm/virt/vm.py", line 2264, in _startUnderl=
yingVm=C2=A0 =C2=A0 self._run()=C2=A0 File "/usr/share/vdsm/virt/vm.py", li=
ne 3328, in _run=C2=A0 =C2=A0 self._connection.createXML(domxml, flags),=C2=
=A0 File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line=
111, in wrapper=C2=A0 =C2=A0 ret =3D f(*args, **kwargs)=C2=A0 File "/usr/l=
ib64/python2.6/site-packages/libvirt.py", line 2709, in createXML=C2=A0 =C2=
=A0 if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=
=3Dself)libvirtError: Failed to acquire lock: No space left on deviceThread=
-49::DEBUG::2016-05-13 22:02:49,693::vm::2779::vm.Vm::(setDownStatus) vmId=
=3D`bd48bba5-4109-4a58-ac89-832add1f4de4`::Changed state to Down: Failed to=
acquire lock: No space left on device (code=3D1)
San lock.log:2016-05-13 22:09:58-0400 4283 [16627]: cmd 9 target pid 17793 =
not found2016-05-13 22:09:58-0400 4284 [16632]: r4 cmd_acquire 2,10,17793 i=
nvalid lockspace found -1 failed 0 name 50816073-c2eb-4d9d-be72-30ad62bf85f=
2
I did cluster heal. =C2=A0File systems seems to be fine. =C2=A0I see this l=
ockspace file with hosted-engine. =C2=A0Does it have anything do with the i=
ssue?
# find /gluster -name *lock*/gluster/engine/brick1/50816073-c2eb-4d9d-be72-=
30ad62bf85f2/ha_agent/hosted-engine.lockspace
ThanksWill
------=_Part_1864187_991623925.1463192625760
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<html><head></head><body><div style=3D"color:#000;
background-color:#fff; f=
ont-family:Courier New, courier, monaco, monospace, sans-serif;font-size:13=
px"><div id=3D"yui_3_16_0_ym19_1_1463183424286_15400">Hi
all,</div><div id=
=3D"yui_3_16_0_ym19_1_1463183424286_15400"><br></div><div
id=3D"yui_3_16_0_=
ym19_1_1463183424286_15400">There was an issue with sudoers syntax. V=
dsm didn't start correctly so 54321 wasn't available.</div><div
id=3D"yui_3=
_16_0_ym19_1_1463183424286_15400">After fixing it, I got some errors on loc=
ks. I'm not sure how this should be fixed correctly.</div><div
id=3D"=
yui_3_16_0_ym19_1_1463183424286_15400"><br></div><div
style=3D"margin-top: =
0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family:=
Menlo;" id=3D"yui_3_16_0_ym19_1_1463183424286_15694"><span
style=3D"font-v=
ariant-ligatures: no-common-ligatures" id=3D"yui_3_16_0_ym19_1_146318342428=
6_15695">Thread-49::DEBUG::2016-05-13 22:02:49,689::libvirtconnection::143:=
:root::(wrapper) Unknown libvirterror: ecode: 38 edom: 42 level: 2 message:=
<font id=3D"yui_3_16_0_ym19_1_1463183424286_16364"
style=3D"background-col=
or: rgb(253, 239, 43);">Failed to acquire lock: No space left on device</fo=
nt></span></div><div style=3D"margin-top: 0px; margin-bottom: 0px;
font-siz=
e: 12px; line-height: normal; font-family: Menlo;" id=3D"yui_3_16_0_ym19_1_=
1463183424286_15696"><span style=3D"font-variant-ligatures:
no-common-ligat=
ures"
id=3D"yui_3_16_0_ym19_1_1463183424286_15697">Thread-49::DEBUG::2016-0=
5-13 22:02:49,690::vm::2287::vm.Vm::(_startUnderlyingVm) vmId=3D`bd48bba5-4=
109-4a58-ac89-832add1f4de4`::_ongoingCreations released</span></div><div
st=
yle=3D"margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: n=
ormal; font-family: Menlo;"
id=3D"yui_3_16_0_ym19_1_1463183424286_15698"><s=
pan style=3D"font-variant-ligatures: no-common-ligatures"
id=3D"yui_3_16_0_=
ym19_1_1463183424286_15699">Thread-49::ERROR::2016-05-13 22:02:49,690::vm::=
2324::vm.Vm::(_startUnderlyingVm) vmId=3D`bd48bba5-4109-4a58-ac89-832add1f4=
de4`::The vm start process failed</span></div><div
style=3D"margin-top: 0px=
; margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Me=
nlo;" id=3D"yui_3_16_0_ym19_1_1463183424286_15700"><span
style=3D"font-vari=
ant-ligatures: no-common-ligatures" id=3D"yui_3_16_0_ym19_1_1463183424286_1=
5701">Traceback (most recent call last):</span></div><div
style=3D"margin-t=
op: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-fam=
ily: Menlo;" id=3D"yui_3_16_0_ym19_1_1463183424286_15702"><span
style=3D"fo=
nt-variant-ligatures: no-common-ligatures" id=3D"yui_3_16_0_ym19_1_14631834=
24286_15703"> File "/usr/share/vdsm/virt/vm.py", line 2264, in
_start=
UnderlyingVm</span></div><div style=3D"margin-top: 0px; margin-bottom:
0px;=
font-size: 12px; line-height: normal; font-family: Menlo;" id=3D"yui_3_16_=
0_ym19_1_1463183424286_15704"><span style=3D"font-variant-ligatures:
no-com=
mon-ligatures" id=3D"yui_3_16_0_ym19_1_1463183424286_15705">
s=
elf._run()</span></div><div style=3D"margin-top: 0px; margin-bottom:
0px; f=
ont-size: 12px; line-height: normal; font-family: Menlo;" id=3D"yui_3_16_0_=
ym19_1_1463183424286_15706"><span style=3D"font-variant-ligatures:
no-commo=
n-ligatures" id=3D"yui_3_16_0_ym19_1_1463183424286_15707">
File "/usr=
/share/vdsm/virt/vm.py", line 3328, in _run</span></div><div
style=3D"margi=
n-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; font-=
family: Menlo;" id=3D"yui_3_16_0_ym19_1_1463183424286_15708"><span
style=3D=
"font-variant-ligatures: no-common-ligatures"
id=3D"yui_3_16_0_ym19_1_14631=
83424286_15709"> self._connection.createXML(domxml,
flags),</s=
pan></div><div style=3D"margin-top: 0px; margin-bottom: 0px; font-size:
12p=
x; line-height: normal; font-family: Menlo;" id=3D"yui_3_16_0_ym19_1_146318=
3424286_15710"><span style=3D"font-variant-ligatures:
no-common-ligatures" =
id=3D"yui_3_16_0_ym19_1_1463183424286_15711"> File
"/usr/lib/python2.=
6/site-packages/vdsm/libvirtconnection.py", line 111, in
wrapper</span></di=
v><div style=3D"margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-=
height: normal; font-family: Menlo;" id=3D"yui_3_16_0_ym19_1_1463183424286_=
15712"><span style=3D"font-variant-ligatures: no-common-ligatures"
id=3D"yu=
i_3_16_0_ym19_1_1463183424286_15713"> ret =3D f(*args,
**kwarg=
s)</span></div><div style=3D"margin-top: 0px; margin-bottom: 0px;
font-size=
: 12px; line-height: normal; font-family: Menlo;" id=3D"yui_3_16_0_ym19_1_1=
463183424286_15714"><span style=3D"font-variant-ligatures:
no-common-ligatu=
res" id=3D"yui_3_16_0_ym19_1_1463183424286_15715"> File
"/usr/lib64/p=
ython2.6/site-packages/libvirt.py", line 2709, in
createXML</span></div><di=
v style=3D"margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-heigh=
t: normal; font-family: Menlo;" id=3D"yui_3_16_0_ym19_1_1463183424286_15716=
"><span style=3D"font-variant-ligatures: no-common-ligatures"
id=3D"yui_3_1=
6_0_ym19_1_1463183424286_15717"> if ret is None:raise
libvirtE=
rror('virDomainCreateXML() failed', conn=3Dself)</span></div><div
style=3D"=
margin-top: 0px; margin-bottom: 0px; font-size: 12px; line-height: normal; =
font-family: Menlo;"
id=3D"yui_3_16_0_ym19_1_1463183424286_15718"><span sty=
le=3D"font-variant-ligatures: no-common-ligatures"
id=3D"yui_3_16_0_ym19_1_=
1463183424286_15719">libvirtError: Failed to acquire lock: No space left on=
device</span></div><div
id=3D"yui_3_16_0_ym19_1_1463183424286_15400">
</div><div style=3D"margin-top: 0px; margin-bottom: 0px; font-size: 12px;
l=
ine-height: normal; font-family: Menlo;" id=3D"yui_3_16_0_ym19_1_1463183424=
286_15720"><span style=3D"font-variant-ligatures:
no-common-ligatures" id=
=3D"yui_3_16_0_ym19_1_1463183424286_15721">Thread-49::DEBUG::2016-05-13 22:=
02:49,693::vm::2779::vm.Vm::(setDownStatus) vmId=3D`bd48bba5-4109-4a58-ac89=
-832add1f4de4`::Changed state to Down: Failed to acquire lock: No space lef=
t on device (code=3D1)</span></div><div dir=3D"ltr"
id=3D"yui_3_16_0_ym19_1=
_1463183424286_15722"><span style=3D"font-variant-ligatures:
no-common-liga=
tures" id=3D"yui_3_16_0_ym19_1_1463183424286_15723"><br
id=3D"yui_3_16_0_ym=
19_1_1463183424286_15724"></span></div><div
class=3D"qtdSeparateBR"><br><br=
</div><div class=3D"yahoo_quoted"
id=3D"yui_3_16_0_ym19_1_1463183424286_15=
425" style=3D"display:
block;"><div style=3D"font-family: Courier New, cour=
ier, monaco, monospace, sans-serif; font-size: 13px;" id=3D"yui_3_16_0_ym19=
_1_1463183424286_15424"><div style=3D"font-family: HelveticaNeue,
Helvetica=
Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;" id=3D=
"yui_3_16_0_ym19_1_1463183424286_15423"><div
class=3D"y_msg_container" id=
=3D"yui_3_16_0_ym19_1_1463183424286_15422"><div
id=3D"yiv1436813381"><div i=
d=3D"yui_3_16_0_ym19_1_1463183424286_15421"><div
style=3D"font-family: 'Cou=
rier New', courier, monaco, monospace, sans-serif; font-size: 13px;"
id=3D"=
yui_3_16_0_ym19_1_1463183424286_15420"><div
class=3D"yiv1436813381yqt809646=
6043" id=3D"yiv1436813381yqt34582"><div
id=3D"yui_3_16_0_ym19_1_14631834242=
86_15426"><div
id=3D"yiv1436813381yui_3_16_0_ym19_1_1463183424286_7642" sty=
le=3D"font-family:Courier New, courier, monaco, monospace, sans-serif;font-=
size:13px;"><div
id=3D"yiv1436813381yui_3_16_0_ym19_1_1463183424286_7641" s=
tyle=3D"font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida=
Grande, sans-serif;font-size:16px;"><div
class=3D"yiv1436813381y_msg_conta=
iner"
id=3D"yiv1436813381yui_3_16_0_ym19_1_1463183424286_7640"><br></div><d=
iv class=3D"yiv1436813381y_msg_container"
id=3D"yiv1436813381yui_3_16_0_ym1=
9_1_1463183424286_7640">San lock.log:</div><div
class=3D"yiv1436813381y_msg=
_container"
id=3D"yiv1436813381yui_3_16_0_ym19_1_1463183424286_7640"><div s=
tyle=3D"font-size: 12px; line-height: normal; font-family: Menlo;"
id=3D"yu=
i_3_16_0_ym19_1_1463183424286_15920"><span
style=3D"font-variant-ligatures:=
no-common-ligatures"
id=3D"yui_3_16_0_ym19_1_1463183424286_15921">2016-05-=
13 22:09:58-0400 4283 [16627]: cmd 9 target pid 17793 not found</span></div=
<div style=3D"font-size: 12px; line-height: normal;
font-family: Menlo;" i=
d=3D"yui_3_16_0_ym19_1_1463183424286_15843">
</div><div style=3D"font-size: 12px; line-height: normal; font-family:
Menl=
o;" dir=3D"ltr"
id=3D"yui_3_16_0_ym19_1_1463183424286_15922"><span style=3D=
"font-variant-ligatures: no-common-ligatures"
id=3D"yui_3_16_0_ym19_1_14631=
83424286_15923">2016-05-13 22:09:58-0400 4284 [16632]: r4 cmd_acquire 2,10,=
17793 <span style=3D"background-color: rgb(253, 239, 43);"
id=3D"yui_3_16_0=
_ym19_1_1463183424286_16445">invalid lockspace found</span> -1 failed 0
nam=
e
50816073-c2eb-4d9d-be72-30ad62bf85f2</span></div></div></div></div></div>=
</div></div></div></div><br></div><div
class=3D"y_msg_container" id=3D"yui_=
3_16_0_ym19_1_1463183424286_15422" dir=3D"ltr">I did cluster heal.
Fi=
le systems seems to be fine. I see this lockspace file with hosted-en=
gine. Does it have anything do with the issue?</div><div
class=3D"y_m=
sg_container" id=3D"yui_3_16_0_ym19_1_1463183424286_15422"
dir=3D"ltr"><br =
id=3D"yui_3_16_0_ym19_1_1463183424286_16510"></div><div
class=3D"y_msg_cont=
ainer" id=3D"yui_3_16_0_ym19_1_1463183424286_15422"
dir=3D"ltr"><div style=
=3D"font-size: 12px; line-height: normal; font-family: Menlo;"
id=3D"yui_3_=
16_0_ym19_1_1463183424286_16502"><span style=3D"font-variant-ligatures:
no-=
common-ligatures" id=3D"yui_3_16_0_ym19_1_1463183424286_16503"># find
/glus=
ter -name *lock*</span></div><div style=3D"font-size: 12px;
line-height: no=
rmal; font-family: Menlo;"
id=3D"yui_3_16_0_ym19_1_1463183424286_16062">
</div><div style=3D"font-size: 12px; line-height: normal; font-family:
Menl=
o;" id=3D"yui_3_16_0_ym19_1_1463183424286_16504"><span
style=3D"font-varian=
t-ligatures: no-common-ligatures" id=3D"yui_3_16_0_ym19_1_1463183424286_165=
05">/gluster/engine/brick1/50816073-c2eb-4d9d-be72-30ad62bf85f2/ha_agent/<s=
pan style=3D"background-color: rgb(253, 239, 43);"
id=3D"yui_3_16_0_ym19_1_=
1463183424286_16581">hosted-engine.lockspace</span></span></div><div
dir=3D=
"ltr" id=3D"yui_3_16_0_ym19_1_1463183424286_16506"><span
style=3D"font-vari=
ant-ligatures: no-common-ligatures" id=3D"yui_3_16_0_ym19_1_1463183424286_1=
6507"><br
id=3D"yui_3_16_0_ym19_1_1463183424286_16508"></span></div><div
st=
yle=3D"font-size: 12px; line-height: normal; font-family: Menlo;"
dir=3D"lt=
r" id=3D"yui_3_16_0_ym19_1_1463183424286_16064"><span
style=3D"font-variant=
-ligatures: no-common-ligatures"><br></span></div><div
style=3D"font-size: =
12px; line-height: normal; font-family: Menlo;" dir=3D"ltr"
id=3D"yui_3_16_=
0_ym19_1_1463183424286_16064"><span style=3D"font-variant-ligatures:
no-com=
mon-ligatures"><br></span></div><div style=3D"font-size:
12px; line-height:=
normal; font-family: Menlo;" dir=3D"ltr"
id=3D"yui_3_16_0_ym19_1_146318342=
4286_16064"><span style=3D"font-variant-ligatures:
no-common-ligatures" id=
=3D"yui_3_16_0_ym19_1_1463183424286_16067">Thanks</span></div><div
style=3D=
"font-size: 12px; line-height: normal; font-family: Menlo;"
dir=3D"ltr" id=
=3D"yui_3_16_0_ym19_1_1463183424286_16064"><span
style=3D"font-variant-liga=
tures: no-common-ligatures;"
id=3D"yui_3_16_0_ym19_1_1463183424286_16283">W=
ill</span></div></div> </div> </div>
</div></div></body></html>
------=_Part_1864187_991623925.1463192625760--