3:29 a.m.
--_000_8569C17008504C42643F5ABB0F2D445E86650D736F998CDCmailout_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Hello Didi,
this is a clean install of the first host. (I am trying a disaster scenario=
and will later restore the backup)
--
Daniel Helgenberger
m box bewegtbild GmbH
P: +49/30/2408781-22<tel:+49/30/2408781-22>
F: +49/30/2408781-10<tel:+49/30/2408781-10>
ACKERSTR. 19
D-10115 BERLIN
www.m-box.de<http://www.m-box.de> www.monkeymen.tv<http://www.monkeymen.tv>
Gesch=E4ftsf=FChrer: Martin Retschitzegger / Michaela G=F6llner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767
On Wed, May 13, 2015 at 12:22 AM -0700, "Yedidyah Bar David" <didi(a)redhat.c=
om<mailto:didi@redhat.com>> wrote:
----- Original Message -----
From: "Daniel Helgenberger"
<daniel.helgenberger(a)m-box.de>
To: users(a)ovirt.org
Sent: Tuesday, May 12, 2015 8:48:42 PM
Subject: [ovirt-users] HE setup failing: Cannot set temporary password fo=
r
console connection
Hello,
a week ago I set up a HE (3.5.2) with no problems; now this fails.
You mean you re-deploy the same host (clean setup), or adding another host?
I
suspect storage trouble but cannot pinpoint it. Did someth. change there
with the recent async release?
The storage domain is NFS3 via a 1G link to a ZFS storage appliance. The
export is working quite fast; tested with iozone running for over 24h
without problems.
It starts with the "Verifying sanlock lockspace initialization" phase;
which lasts about 7min (!).
In the end it seems sanlock is locking the domain down. I can only get
it back by stopping sanlock and vdsm.
Attached is the full vdsm lock.
Thanks!
vdsm:
Thread-27::ERROR::2015-05-12
19:17:51,470::domainMonitor::256::Storage.DomainMonitorThread::(_monitorD=
omain)
Error while collecting domain 2f83ea82-e110-465d-a2a6-ac30534a9f6a
monitoring information
Traceback (most recent call last):
File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in
_monitorDomain
self.domain.selftest()
File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
return getattr(self.getRealDomain(), attrName)
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain =3D self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom =3D findMethod(sdUUID)
File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain
return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
('2f83ea82-e110-465d-a2a6-ac30534a9f6a',)
Detector thread::DEBUG::2015-05-12
19:24:33,163::BindingXMLRPC::1173::XmlDetector::(handleSocket) xml over h=
ttp
detected from ('127.0.0.1', 47076)
Thread-93::DEBUG::2015-05-12
19:24:33,166::BindingXMLRPC::1133::vds::(wrapper) client [127.0.0.1]::cal=
l
vmSetTicket with ('e3d350c5-6254-4908-99e9-d60ca2e046f7',
'5435uOVP',
'10800', 'disconnect', {}) {}
Thread-93::ERROR::2015-05-12
19:24:33,166::BindingXMLRPC::1152::vds::(wrapper) unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1136, in wrapper
res =3D f(*args, **kwargs)
File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 383, in vmSetTicket
return vm.setTicket(password, ttl, existingConnAction, params)
File "/usr/share/vdsm/API.py", line 642, in setTicket
return v.setTicket(password, ttl, existingConnAction, params)
File "/usr/share/vdsm/virt/vm.py", line 4834, in setTicket
graphics =3D _domParseStr(self._dom.XMLDesc(0)).childNodes[0]. \
AttributeError: 'NoneType' object has no attribute 'XMLDesc'
and:
Thread-91::DEBUG::2015-05-12
19:25:15,152::libvirtconnection::143::root::(wrapper) Unknown libvirterro=
r:
ecode: 38 edom: 42 level: 2 message: Failed to acquire lock: No space
lef=
t
on device
Thread-91::DEBUG::2015-05-12
19:25:15,154::vm::2294::vm.Vm::(_startUnderlyingVm)
vmId=3D`e3d350c5-6254-4908-99e9-d60ca2e046f7`::_ongoingCreations released
Thread-91::ERROR::2015-05-12
19:25:15,154::vm::2331::vm.Vm::(_startUnderlyingVm)
vmId=3D`e3d350c5-6254-4908-99e9-d60ca2e046f7`::The vm start process faile=
d
Traceback (most recent call last):
File "/usr/share/vdsm/virt/vm.py", line 2271, in _startUnderlyingVm
self._run()
File "/usr/share/vdsm/virt/vm.py", line 3335, in _run
self._connection.createXML(domxml, flags),
File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line
111, in wrapper
ret =3D f(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3427, in
createXML
if ret is None:raise libvirtError('virDomainCreateXML() failed',
conn=3Dself)
libvirtError: Failed to acquire lock: No space left on device
sanlock:
2015-05-12 19:11:27+0200 4338 [708]: s2 lockspace
2f83ea82-e110-465d-a2a6-ac30534a9f6a:1:/rhev/data-center/mnt/10.11.0.30:_=
volumes_ovirt_engine/2f83ea82-e110-465d-a2a6-ac30534a9f6a/dom_md/ids:0
2015-05-12 19:11:48+0200 4359 [702]: s2 host 1 1 4338
9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i
2015-05-12 19:11:48+0200 4359 [702]: s2 host 250 1 0
9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i
2015-05-12 19:11:48+0200 4359 [707]: s2:r2 resource
2f83ea82-e110-465d-a2a6-ac30534a9f6a:SDM:/rhev/data-center/mnt/10.11.0.30=
:_volumes_ovirt_engine/2f83ea82-e110-465d-a2a6-ac30534a9f6a/dom_md/leases:1=
048576
for 3,11,4167
2015-05-12 19:23:26+0200 5057 [708]: s3 lockspace
2f83ea82-e110-465d-a2a6-ac30534a9f6a:1:/rhev/data-center/mnt/10.11.0.30:_=
volumes_ovirt_engine/2f83ea82-e110-465d-a2a6-ac30534a9f6a/dom_md/ids:0
2015-05-12 19:23:48+0200 5079 [702]: s3 host 1 2 5058
9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i
2015-05-12 19:23:48+0200 5079 [702]: s3 host 250 1 0
9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i
2015-05-12 19:24:53+0200 5144 [702]: cmd 9 target pid 4893 not found
2015-05-12 19:25:14+0200 5165 [708]: r3 cmd_acquire 2,9,4893 invalid
lockspace found -1 failed 0 name 2f83ea82-e110-465d-a2a6-ac30534a9f6a
--
Daniel Helgenberger
m box bewegtbild GmbH
P: +49/30/2408781-22
F: +49/30/2408781-10
ACKERSTR. 19
D-10115 BERLIN
www.m-box.de<http://www.m-box.de> www.monkeymen.tv<http://www.monkeymen.=
tv>
Gesch=E4ftsf=FChrer: Martin Retschitzegger / Michaela G=F6llner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
Didi
--_000_8569C17008504C42643F5ABB0F2D445E86650D736F998CDCmailout_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html;
charset=3Diso-8859-=
1">
<meta name=3D"Generator" content=3D"Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt;
pad=
ding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<div>
<div>Hello Didi,</div>
<div>this is a clean install of the first host. (I am trying a disaster sce=
nario and will later restore the backup)</div>
<div>
<div class=3D"x_acompli_signature"><br>
-- <br>
Daniel Helgenberger <br>
m box bewegtbild GmbH <br>
<br>
P: <a
href=3D"tel:+49/30/2408781-22">+49/30/2408781-22</a>
<br>
F: <a
href=3D"tel:+49/30/2408781-10">+49/30/2408781-10</a>
<br>
<br>
ACKERSTR. 19 <br>
D-10115 BERLIN <br>
<br>
<br>
<a href=3D"http://www.m-box.de">www.m-box.de</a> <a
href=3D"http://www.monk=
eymen.tv">
www.monkeymen.tv</a> <br>
<br>
Gesch=E4ftsf=FChrer: Martin Retschitzegger / Michaela G=F6llner <br>
Handeslregister: Amtsgericht Charlottenburg / HRB 112767 </div>
<br>
</div>
<br>
<br>
<br>
<div class=3D"x_gmail_quote">On Wed, May 13, 2015 at 12:22 AM -0700,
"=
Yedidyah Bar David"
<span dir=3D"ltr"><<a href=3D"mailto:didi@redhat.com"
target=3D"_blank">=
didi(a)redhat.com</a>&gt;</span> wrote:<br>
<br>
</div>
</div>
<font size=3D"2"><span style=3D"font-size:10pt;">
<div class=3D"PlainText">----- Original Message -----<br>
> From: "Daniel Helgenberger"
&lt;daniel.helgenberger(a)m-box.de=
><br>
> To: users(a)ovirt.org<br>
> Sent: Tuesday, May 12, 2015 8:48:42 PM<br>
> Subject: [ovirt-users] HE setup failing: Cannot set temporary password=
for console connection<br>
> <br>
> Hello,<br>
> <br>
> a week ago I set up a HE (3.5.2) with no problems; now this fails.<br>
<br>
You mean you re-deploy the same host (clean setup), or adding another host?=
<br>
<br>
> I<br>
> suspect storage trouble but cannot pinpoint it. Did someth. change the=
re<br>
> with the recent async release?<br>
> <br>
> The storage domain is NFS3 via a 1G link to a ZFS storage appliance. T=
he<br>
> export is working quite fast; tested with iozone running for over 24h<=
br>
> without problems.<br>
> <br>
> It starts with the "Verifying sanlock lockspace initialization&qu=
ot; phase;<br>
> which lasts about 7min (!).<br>
> <br>
> In the end it seems sanlock is locking the domain down. I can only get=
<br>
> it back by stopping sanlock and vdsm.<br>
> <br>
> Attached is the full vdsm lock.<br>
> <br>
> <br>
> Thanks!<br>
> <br>
> vdsm:<br>
> <br>
> Thread-27::ERROR::2015-05-12<br>
> 19:17:51,470::domainMonitor::256::Storage.DomainMonitorThread::(_monit=
orDomain)<br>
> Error while collecting domain 2f83ea82-e110-465d-a2a6-ac30534a9f6a<br>
> monitoring information<br>
> Traceback (most recent call last):<br>
> File
"/usr/share/vdsm/storage/domainMonitor.py",=
line 232, in<br>
> _monitorDomain<br>
> self.domain.selftest()<br>
> File "/usr/share/vdsm/storage/sdc.py", line
49, =
in __getattr__<br>
> return getattr(self.getRealDomain(),
attrName)=
<br>
> File "/usr/share/vdsm/storage/sdc.py", line
52, =
in getRealDomain<br>
> return
self._cache._realProduce(self._sdUUID)<=
br>
> File "/usr/share/vdsm/storage/sdc.py", line
122,=
in _realProduce<br>
> domain =3D
self._findDomain(sdUUID)<br>
> File "/usr/share/vdsm/storage/sdc.py", line
141,=
in _findDomain<br>
> dom =3D findMethod(sdUUID)<br>
> File "/usr/share/vdsm/storage/nfsSD.py",
line 12=
2, in findDomain<br>
> return
NfsStorageDomain(NfsStorageDomain.findD=
omainPath(sdUUID))<br>
> File "/usr/share/vdsm/storage/nfsSD.py",
line 11=
2, in findDomainPath<br>
> raise
se.StorageDomainDoesNotExist(sdUUID)<br>
> StorageDomainDoesNotExist: Storage domain does not exist:<br>
> ('2f83ea82-e110-465d-a2a6-ac30534a9f6a',)<br>
> <br>
> <br>
> Detector thread::DEBUG::2015-05-12<br>
> 19:24:33,163::BindingXMLRPC::1173::XmlDetector::(handleSocket) xml ove=
r http<br>
> detected from ('127.0.0.1', 47076)<br>
> Thread-93::DEBUG::2015-05-12<br>
> 19:24:33,166::BindingXMLRPC::1133::vds::(wrapper) client [127.0.0.1]::=
call<br>
> vmSetTicket with ('e3d350c5-6254-4908-99e9-d60ca2e046f7',
'5435uOVP',<=
br>
> '10800', 'disconnect', {}) {}<br>
> Thread-93::ERROR::2015-05-12<br>
> 19:24:33,166::BindingXMLRPC::1152::vds::(wrapper) unexpected error<br>
> Traceback (most recent call last):<br>
> File
"/usr/share/vdsm/rpc/BindingXMLRPC.py", lin=
e 1136, in wrapper<br>
> res =3D f(*args, **kwargs)<br>
> File
"/usr/share/vdsm/rpc/BindingXMLRPC.py", lin=
e 383, in vmSetTicket<br>
> return vm.setTicket(password, ttl,
existingCon=
nAction, params)<br>
> File "/usr/share/vdsm/API.py", line 642, in
setT=
icket<br>
> return v.setTicket(password, ttl,
existingConn=
Action, params)<br>
> File "/usr/share/vdsm/virt/vm.py", line
4834, in=
setTicket<br>
> graphics =3D
_domParseStr(self._dom.XMLDesc(0)=
).childNodes[0]. \<br>
> AttributeError: 'NoneType' object has no attribute
'XMLDesc'<br>
> <br>
> and:<br>
> <br>
> Thread-91::DEBUG::2015-05-12<br>
> 19:25:15,152::libvirtconnection::143::root::(wrapper) Unknown libvirte=
rror:<br>
> ecode: 38 edom: 42 level: 2 message: Failed to acquire lock: No space =
left<br>
> on device<br>
> Thread-91::DEBUG::2015-05-12<br>
> 19:25:15,154::vm::2294::vm.Vm::(_startUnderlyingVm)<br>
> vmId=3D`e3d350c5-6254-4908-99e9-d60ca2e046f7`::_ongoingCreations relea=
sed<br>
> Thread-91::ERROR::2015-05-12<br>
> 19:25:15,154::vm::2331::vm.Vm::(_startUnderlyingVm)<br>
> vmId=3D`e3d350c5-6254-4908-99e9-d60ca2e046f7`::The vm start process fa=
iled<br>
> Traceback (most recent call last):<br>
> File "/usr/share/vdsm/virt/vm.py", line
2271, in=
_startUnderlyingVm<br>
> self._run()<br>
> File "/usr/share/vdsm/virt/vm.py", line
3335, in=
_run<br>
> self._connection.createXML(domxml,
flags),<br>
> File
"/usr/lib/python2.7/site-packages/vdsm/libvirtco=
nnection.py", line<br>
> 111, in wrapper<br>
> ret =3D f(*args, **kwargs)<br>
> File
"/usr/lib64/python2.7/site-packages/libvirt.py&q=
uot;, line 3427, in<br>
> createXML<br>
> if ret is None:raise
libvirtError('virDomainCr=
eateXML() failed',<br>
> conn=3Dself)<br>
> libvirtError: Failed to acquire lock: No space left on device<br>
> <br>
> sanlock:<br>
> 2015-05-12 19:11:27+0200 4338 [708]: s2 lockspace<br>
> 2f83ea82-e110-465d-a2a6-ac30534a9f6a:1:/rhev/data-center/mnt/10.11.0.3=
0:_volumes_ovirt_engine/2f83ea82-e110-465d-a2a6-ac30534a9f6a/dom_md/ids:0<b=
r>
> 2015-05-12 19:11:48+0200 4359 [702]: s2 host 1 1 4338<br>
> 9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i<br>
> 2015-05-12 19:11:48+0200 4359 [702]: s2 host 250 1 0<br>
> 9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i<br>
> 2015-05-12 19:11:48+0200 4359 [707]: s2:r2 resource<br>
> 2f83ea82-e110-465d-a2a6-ac30534a9f6a:SDM:/rhev/data-center/mnt/10.11.0=
.30:_volumes_ovirt_engine/2f83ea82-e110-465d-a2a6-ac30534a9f6a/dom_md/lease=
s:1048576<br>
> for 3,11,4167<br>
> 2015-05-12 19:23:26+0200 5057 [708]: s3 lockspace<br>
> 2f83ea82-e110-465d-a2a6-ac30534a9f6a:1:/rhev/data-center/mnt/10.11.0.3=
0:_volumes_ovirt_engine/2f83ea82-e110-465d-a2a6-ac30534a9f6a/dom_md/ids:0<b=
r>
> 2015-05-12 19:23:48+0200 5079 [702]: s3 host 1 2 5058<br>
> 9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i<br>
> 2015-05-12 19:23:48+0200 5079 [702]: s3 host 250 1 0<br>
> 9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i<br>
> 2015-05-12 19:24:53+0200 5144 [702]: cmd 9 target pid 4893 not fou=
nd<br>
> 2015-05-12 19:25:14+0200 5165 [708]: r3 cmd_acquire 2,9,4893 inval=
id<br>
> lockspace found -1 failed 0 name 2f83ea82-e110-465d-a2a6-ac30534a9f6a<=
br>
> <br>
> --<br>
> Daniel Helgenberger<br>
> m box bewegtbild GmbH<br>
> <br>
> P: +49/30/2408781-22<br>
> F: +49/30/2408781-10<br>
> <br>
> ACKERSTR. 19<br>
> D-10115 BERLIN<br>
> <br>
> <br>
> <a href=3D"http://www.m-box.de">www.m-box.de</a>
<a href=3D"http=
://www.monkeymen.tv">
www.monkeymen.tv</a><br>
> <br>
> Gesch=E4ftsf=FChrer: Martin Retschitzegger / Michaela G=F6llner<br>
> Handeslregister: Amtsgericht Charlottenburg / HRB 112767<br>
> <br>
> _______________________________________________<br>
> Users mailing list<br>
> Users(a)ovirt.org<br>
> <a
href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
.ovirt.org/mailman/listinfo/users</a><br>
> <br>
<br>
-- <br>
Didi<br>
<br>
</div>
</span></font>
</body>
</html>
--_000_8569C17008504C42643F5ABB0F2D445E86650D736F998CDCmailout_--