--_000_8569C17008504C42643F5ABB0F2D445E86650D736F998CDCmailout_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hello Didi, this is a clean install of the first host. (I am trying a disaster scenario= and will later restore the backup) -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22<tel:+49/30/2408781-22> F: +49/30/2408781-10<tel:+49/30/2408781-10> ACKERSTR. 19 D-10115 BERLIN www.m-box.de<http://www.m-box.de> www.monkeymen.tv<http://www.monkeymen.tv> Gesch=E4ftsf=FChrer: Martin Retschitzegger / Michaela G=F6llner Handeslregister: Amtsgericht Charlottenburg / HRB 112767 On Wed, May 13, 2015 at 12:22 AM -0700, "Yedidyah Bar David" <didi@redhat.c= om<mailto:didi@redhat.com>> wrote: ----- Original Message -----
From: "Daniel Helgenberger" <daniel.helgenberger@m-box.de> To: users@ovirt.org Sent: Tuesday, May 12, 2015 8:48:42 PM Subject: [ovirt-users] HE setup failing: Cannot set temporary password fo= r console connection
Hello,
a week ago I set up a HE (3.5.2) with no problems; now this fails.
You mean you re-deploy the same host (clean setup), or adding another host?
I suspect storage trouble but cannot pinpoint it. Did someth. change there with the recent async release?
The storage domain is NFS3 via a 1G link to a ZFS storage appliance. The export is working quite fast; tested with iozone running for over 24h without problems.
It starts with the "Verifying sanlock lockspace initialization" phase; which lasts about 7min (!).
In the end it seems sanlock is locking the domain down. I can only get it back by stopping sanlock and vdsm.
Attached is the full vdsm lock.
Thanks!
vdsm:
Thread-27::ERROR::2015-05-12 19:17:51,470::domainMonitor::256::Storage.DomainMonitorThread::(_monitorD= omain) Error while collecting domain 2f83ea82-e110-465d-a2a6-ac30534a9f6a monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain self.domain.selftest() File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain =3D self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom =3D findMethod(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('2f83ea82-e110-465d-a2a6-ac30534a9f6a',)
Detector thread::DEBUG::2015-05-12 19:24:33,163::BindingXMLRPC::1173::XmlDetector::(handleSocket) xml over h= ttp detected from ('127.0.0.1', 47076) Thread-93::DEBUG::2015-05-12 19:24:33,166::BindingXMLRPC::1133::vds::(wrapper) client [127.0.0.1]::cal= l vmSetTicket with ('e3d350c5-6254-4908-99e9-d60ca2e046f7', '5435uOVP', '10800', 'disconnect', {}) {} Thread-93::ERROR::2015-05-12 19:24:33,166::BindingXMLRPC::1152::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1136, in wrapper res =3D f(*args, **kwargs) File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 383, in vmSetTicket return vm.setTicket(password, ttl, existingConnAction, params) File "/usr/share/vdsm/API.py", line 642, in setTicket return v.setTicket(password, ttl, existingConnAction, params) File "/usr/share/vdsm/virt/vm.py", line 4834, in setTicket graphics =3D _domParseStr(self._dom.XMLDesc(0)).childNodes[0]. \ AttributeError: 'NoneType' object has no attribute 'XMLDesc'
and:
Thread-91::DEBUG::2015-05-12 19:25:15,152::libvirtconnection::143::root::(wrapper) Unknown libvirterro= r: ecode: 38 edom: 42 level: 2 message: Failed to acquire lock: No space lef= t on device Thread-91::DEBUG::2015-05-12 19:25:15,154::vm::2294::vm.Vm::(_startUnderlyingVm) vmId=3D`e3d350c5-6254-4908-99e9-d60ca2e046f7`::_ongoingCreations released Thread-91::ERROR::2015-05-12 19:25:15,154::vm::2331::vm.Vm::(_startUnderlyingVm) vmId=3D`e3d350c5-6254-4908-99e9-d60ca2e046f7`::The vm start process faile= d Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 2271, in _startUnderlyingVm self._run() File "/usr/share/vdsm/virt/vm.py", line 3335, in _run self._connection.createXML(domxml, flags), File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper ret =3D f(*args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3427, in createXML if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=3Dself) libvirtError: Failed to acquire lock: No space left on device
sanlock: 2015-05-12 19:11:27+0200 4338 [708]: s2 lockspace 2f83ea82-e110-465d-a2a6-ac30534a9f6a:1:/rhev/data-center/mnt/10.11.0.30:_= volumes_ovirt_engine/2f83ea82-e110-465d-a2a6-ac30534a9f6a/dom_md/ids:0 2015-05-12 19:11:48+0200 4359 [702]: s2 host 1 1 4338 9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i 2015-05-12 19:11:48+0200 4359 [702]: s2 host 250 1 0 9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i 2015-05-12 19:11:48+0200 4359 [707]: s2:r2 resource 2f83ea82-e110-465d-a2a6-ac30534a9f6a:SDM:/rhev/data-center/mnt/10.11.0.30= :_volumes_ovirt_engine/2f83ea82-e110-465d-a2a6-ac30534a9f6a/dom_md/leases:1= 048576 for 3,11,4167 2015-05-12 19:23:26+0200 5057 [708]: s3 lockspace 2f83ea82-e110-465d-a2a6-ac30534a9f6a:1:/rhev/data-center/mnt/10.11.0.30:_= volumes_ovirt_engine/2f83ea82-e110-465d-a2a6-ac30534a9f6a/dom_md/ids:0 2015-05-12 19:23:48+0200 5079 [702]: s3 host 1 2 5058 9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i 2015-05-12 19:23:48+0200 5079 [702]: s3 host 250 1 0 9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i 2015-05-12 19:24:53+0200 5144 [702]: cmd 9 target pid 4893 not found 2015-05-12 19:25:14+0200 5165 [708]: r3 cmd_acquire 2,9,4893 invalid lockspace found -1 failed 0 name 2f83ea82-e110-465d-a2a6-ac30534a9f6a
-- Daniel Helgenberger m box bewegtbild GmbH
P: +49/30/2408781-22 F: +49/30/2408781-10
ACKERSTR. 19 D-10115 BERLIN
www.m-box.de<http://www.m-box.de> www.monkeymen.tv<http://www.monkeymen.= tv>
Gesch=E4ftsf=FChrer: Martin Retschitzegger / Michaela G=F6llner Handeslregister: Amtsgericht Charlottenburg / HRB 112767
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Didi --_000_8569C17008504C42643F5ABB0F2D445E86650D736F998CDCmailout_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <meta name=3D"Generator" content=3D"Microsoft Exchange Server"> <!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; pad= ding-left: 4pt; border-left: #800000 2px solid; } --></style> </head> <body> <div> <div>Hello Didi,</div> <div>this is a clean install of the first host. (I am trying a disaster sce= nario and will later restore the backup)</div> <div> <div class=3D"x_acompli_signature"><br> -- <br> Daniel Helgenberger <br> m box bewegtbild GmbH <br> <br> P: <a href=3D"tel:+49/30/2408781-22">+49/30/2408781-22</a> <br> F: <a href=3D"tel:+49/30/2408781-10">+49/30/2408781-10</a> <br> <br> ACKERSTR. 19 <br> D-10115 BERLIN <br> <br> <br> <a href=3D"http://www.m-box.de">www.m-box.de</a> <a href=3D"http://www.monk= eymen.tv"> www.monkeymen.tv</a> <br> <br> Gesch=E4ftsf=FChrer: Martin Retschitzegger / Michaela G=F6llner <br> Handeslregister: Amtsgericht Charlottenburg / HRB 112767 </div> <br> </div> <br> <br> <br> <div class=3D"x_gmail_quote">On Wed, May 13, 2015 at 12:22 AM -0700, "= Yedidyah Bar David" <span dir=3D"ltr"><<a href=3D"mailto:didi@redhat.com" target=3D"_blank">= didi@redhat.com</a>></span> wrote:<br> <br> </div> </div> <font size=3D"2"><span style=3D"font-size:10pt;"> <div class=3D"PlainText">----- Original Message -----<br> > From: "Daniel Helgenberger" <daniel.helgenberger@m-box.de= ><br> > To: users@ovirt.org<br> > Sent: Tuesday, May 12, 2015 8:48:42 PM<br> > Subject: [ovirt-users] HE setup failing: Cannot set temporary password= for console connection<br> > <br> > Hello,<br> > <br> > a week ago I set up a HE (3.5.2) with no problems; now this fails.<br> <br> You mean you re-deploy the same host (clean setup), or adding another host?= <br> <br> > I<br> > suspect storage trouble but cannot pinpoint it. Did someth. change the= re<br> > with the recent async release?<br> > <br> > The storage domain is NFS3 via a 1G link to a ZFS storage appliance. T= he<br> > export is working quite fast; tested with iozone running for over 24h<= br> > without problems.<br> > <br> > It starts with the "Verifying sanlock lockspace initialization&qu= ot; phase;<br> > which lasts about 7min (!).<br> > <br> > In the end it seems sanlock is locking the domain down. I can only get= <br> > it back by stopping sanlock and vdsm.<br> > <br> > Attached is the full vdsm lock.<br> > <br> > <br> > Thanks!<br> > <br> > vdsm:<br> > <br> > Thread-27::ERROR::2015-05-12<br> > 19:17:51,470::domainMonitor::256::Storage.DomainMonitorThread::(_monit= orDomain)<br> > Error while collecting domain 2f83ea82-e110-465d-a2a6-ac30534a9f6a<br> > monitoring information<br> > Traceback (most recent call last):<br> > File "/usr/share/vdsm/storage/domainMonitor.py",= line 232, in<br> > _monitorDomain<br> > self.domain.selftest()<br> > File "/usr/share/vdsm/storage/sdc.py", line 49, = in __getattr__<br> > return getattr(self.getRealDomain(), attrName)= <br> > File "/usr/share/vdsm/storage/sdc.py", line 52, = in getRealDomain<br> > return self._cache._realProduce(self._sdUUID)<= br> > File "/usr/share/vdsm/storage/sdc.py", line 122,= in _realProduce<br> > domain =3D self._findDomain(sdUUID)<br> > File "/usr/share/vdsm/storage/sdc.py", line 141,= in _findDomain<br> > dom =3D findMethod(sdUUID)<br> > File "/usr/share/vdsm/storage/nfsSD.py", line 12= 2, in findDomain<br> > return NfsStorageDomain(NfsStorageDomain.findD= omainPath(sdUUID))<br> > File "/usr/share/vdsm/storage/nfsSD.py", line 11= 2, in findDomainPath<br> > raise se.StorageDomainDoesNotExist(sdUUID)<br> > StorageDomainDoesNotExist: Storage domain does not exist:<br> > ('2f83ea82-e110-465d-a2a6-ac30534a9f6a',)<br> > <br> > <br> > Detector thread::DEBUG::2015-05-12<br> > 19:24:33,163::BindingXMLRPC::1173::XmlDetector::(handleSocket) xml ove= r http<br> > detected from ('127.0.0.1', 47076)<br> > Thread-93::DEBUG::2015-05-12<br> > 19:24:33,166::BindingXMLRPC::1133::vds::(wrapper) client [127.0.0.1]::= call<br> > vmSetTicket with ('e3d350c5-6254-4908-99e9-d60ca2e046f7', '5435uOVP',<= br> > '10800', 'disconnect', {}) {}<br> > Thread-93::ERROR::2015-05-12<br> > 19:24:33,166::BindingXMLRPC::1152::vds::(wrapper) unexpected error<br> > Traceback (most recent call last):<br> > File "/usr/share/vdsm/rpc/BindingXMLRPC.py", lin= e 1136, in wrapper<br> > res =3D f(*args, **kwargs)<br> > File "/usr/share/vdsm/rpc/BindingXMLRPC.py", lin= e 383, in vmSetTicket<br> > return vm.setTicket(password, ttl, existingCon= nAction, params)<br> > File "/usr/share/vdsm/API.py", line 642, in setT= icket<br> > return v.setTicket(password, ttl, existingConn= Action, params)<br> > File "/usr/share/vdsm/virt/vm.py", line 4834, in= setTicket<br> > graphics =3D _domParseStr(self._dom.XMLDesc(0)= ).childNodes[0]. \<br> > AttributeError: 'NoneType' object has no attribute 'XMLDesc'<br> > <br> > and:<br> > <br> > Thread-91::DEBUG::2015-05-12<br> > 19:25:15,152::libvirtconnection::143::root::(wrapper) Unknown libvirte= rror:<br> > ecode: 38 edom: 42 level: 2 message: Failed to acquire lock: No space = left<br> > on device<br> > Thread-91::DEBUG::2015-05-12<br> > 19:25:15,154::vm::2294::vm.Vm::(_startUnderlyingVm)<br> > vmId=3D`e3d350c5-6254-4908-99e9-d60ca2e046f7`::_ongoingCreations relea= sed<br> > Thread-91::ERROR::2015-05-12<br> > 19:25:15,154::vm::2331::vm.Vm::(_startUnderlyingVm)<br> > vmId=3D`e3d350c5-6254-4908-99e9-d60ca2e046f7`::The vm start process fa= iled<br> > Traceback (most recent call last):<br> > File "/usr/share/vdsm/virt/vm.py", line 2271, in= _startUnderlyingVm<br> > self._run()<br> > File "/usr/share/vdsm/virt/vm.py", line 3335, in= _run<br> > self._connection.createXML(domxml, flags),<br> > File "/usr/lib/python2.7/site-packages/vdsm/libvirtco= nnection.py", line<br> > 111, in wrapper<br> > ret =3D f(*args, **kwargs)<br> > File "/usr/lib64/python2.7/site-packages/libvirt.py&q= uot;, line 3427, in<br> > createXML<br> > if ret is None:raise libvirtError('virDomainCr= eateXML() failed',<br> > conn=3Dself)<br> > libvirtError: Failed to acquire lock: No space left on device<br> > <br> > sanlock:<br> > 2015-05-12 19:11:27+0200 4338 [708]: s2 lockspace<br> > 2f83ea82-e110-465d-a2a6-ac30534a9f6a:1:/rhev/data-center/mnt/10.11.0.3= 0:_volumes_ovirt_engine/2f83ea82-e110-465d-a2a6-ac30534a9f6a/dom_md/ids:0<b= r> > 2015-05-12 19:11:48+0200 4359 [702]: s2 host 1 1 4338<br> > 9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i<br> > 2015-05-12 19:11:48+0200 4359 [702]: s2 host 250 1 0<br> > 9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i<br> > 2015-05-12 19:11:48+0200 4359 [707]: s2:r2 resource<br> > 2f83ea82-e110-465d-a2a6-ac30534a9f6a:SDM:/rhev/data-center/mnt/10.11.0= .30:_volumes_ovirt_engine/2f83ea82-e110-465d-a2a6-ac30534a9f6a/dom_md/lease= s:1048576<br> > for 3,11,4167<br> > 2015-05-12 19:23:26+0200 5057 [708]: s3 lockspace<br> > 2f83ea82-e110-465d-a2a6-ac30534a9f6a:1:/rhev/data-center/mnt/10.11.0.3= 0:_volumes_ovirt_engine/2f83ea82-e110-465d-a2a6-ac30534a9f6a/dom_md/ids:0<b= r> > 2015-05-12 19:23:48+0200 5079 [702]: s3 host 1 2 5058<br> > 9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i<br> > 2015-05-12 19:23:48+0200 5079 [702]: s3 host 250 1 0<br> > 9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i<br> > 2015-05-12 19:24:53+0200 5144 [702]: cmd 9 target pid 4893 not fou= nd<br> > 2015-05-12 19:25:14+0200 5165 [708]: r3 cmd_acquire 2,9,4893 inval= id<br> > lockspace found -1 failed 0 name 2f83ea82-e110-465d-a2a6-ac30534a9f6a<= br> > <br> > --<br> > Daniel Helgenberger<br> > m box bewegtbild GmbH<br> > <br> > P: +49/30/2408781-22<br> > F: +49/30/2408781-10<br> > <br> > ACKERSTR. 19<br> > D-10115 BERLIN<br> > <br> > <br> > <a href=3D"http://www.m-box.de">www.m-box.de</a> ; <a href=3D"http= ://www.monkeymen.tv"> www.monkeymen.tv</a><br> > <br> > Gesch=E4ftsf=FChrer: Martin Retschitzegger / Michaela G=F6llner<br> > Handeslregister: Amtsgericht Charlottenburg / HRB 112767<br> > <br> > _______________________________________________<br> > Users mailing list<br> > Users@ovirt.org<br> > <a href=3D"http://lists.ovirt.org/mailman/listinfo/users">http://lists= .ovirt.org/mailman/listinfo/users</a><br> > <br> <br> -- <br> Didi<br> <br> </div> </span></font> </body> </html> --_000_8569C17008504C42643F5ABB0F2D445E86650D736F998CDCmailout_--