------=_NextPartTM-000-161201e6-5459-4708-99a4-b65c101b4bd9 Content-Type: multipart/alternative; boundary="_000_12EF8D94C6F8734FB2FF37B9FBEDD17358578085EXCHANGEcollogi_" --_000_12EF8D94C6F8734FB2FF37B9FBEDD17358578085EXCHANGEcollogi_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hello, maybe a stupid one but ... When I create a (NFS) storage domain I have to provide a node host that mak= es the inital contact. All other node hosts wil directly connect to that domai= n. So no bottlenecks. Today I stopped one of my two nodes outside ovirt-engine. For simplicity we assume the node crashed. The machine is up right now but VDSM is down. All domains that where setup with this host are "down" now (red arrow down)= . After searching the web interface I found "Data center" -> "Select your DC"= -> "Storage" -> "Activate". Trying to activate only results in a failure messa= ge. To ensure that I can recover those situations in the future I'd like to kno= w what this node binding is all about and what to do next. Logs attached & thanks in advance Markus 2013-09-17 08:54:54,985 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.Ir= sBrokerCommand] (pool-6-thread-50) [1be325f3] spm vds is non responsive, st= opping spm selection. 2013-09-17 08:54:54,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.Ac= tivateStorageDomainVDSCommand] (pool-6-thread-50) [1be325f3] FINISH, Activa= teStorageDomainVDSCommand, log id: 4c38e98 2013-09-17 08:54:54,987 ERROR [org.ovirt.engine.core.bll.storage.ActivateSt= orageDomainCommand] (pool-6-thread-50) [1be325f3] Command org.ovirt.engine.= core.bll.storage.ActivateStorageDomainCommand throw Vdc Bll exception. With= error message VdcBLLException: Cannot allocate IRS server (Failed with VDS= M error IRS_REPOSITORY_NOT_FOUND and code 5009) 2013-09-17 08:54:54,989 INFO [org.ovirt.engine.core.bll.storage.ActivateSt= orageDomainCommand] (pool-6-thread-50) [1be325f3] Command [id=3Da0dbe909-fb= b1-40ff-b77a-8e43bd075ace]: Compensating CHANGED_STATUS_ONLY of org.ovirt.e= ngine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatu= sSnapshot [id=3DstoragePoolId =3D b054727d-fe4a-41ed-8393-a81e36b8a1af, sto= rageId =3D ecf7f507-b0fa-47ee-a8b2-d621fbd7b8bf, status=3DUnknown]. 2013-09-17 08:54:55,004 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapa= bilitiesVDS execution failed. Exception: VDSNetworkException: java.net.Conn= ectException: Connection refused 2013-09-17 08:54:55,008 INFO [org.ovirt.engine.core.dal.dbbroker.auditlogh= andling.AuditLogDirector] (pool-6-thread-50) [1be325f3] Correlation ID: 1be= 325f3, Job ID: c88c00ba-0298-4f42-bc0e-a720d79c5f49, Call Stack: null, Cust= om Event ID: -1, Message: Failed to activate Storage Domain NAS5_IB (Data C= enter Collogia) by admin@internal 2013-09-17 08:54:56,263 INFO [org.ovirt.engine.core.bll.storage.ActivateSt= orageDomainCommand] (ajp--127.0.0.1-8702-2) [5c6218c1] Lock Acquired to obj= ect EngineLock [exclusiveLocks=3D key: ecf7f507-b0fa-47ee-a8b2-d621fbd7b8bf= value: STORAGE , sharedLocks=3D ] 2013-09-17 08:54:56,272 INFO [org.ovirt.engine.core.bll.storage.ActivateSt= orageDomainCommand] (pool-6-thread-50) [5c6218c1] Running command: Activate= StorageDomainCommand internal: false. Entities affected : ID: ecf7f507-b0f= a-47ee-a8b2-d621fbd7b8bf Type: Storage 2013-09-17 08:54:56,291 INFO [org.ovirt.engine.core.bll.storage.ActivateSt= orageDomainCommand] (pool-6-thread-50) [5c6218c1] Lock freed to object Engi= neLock [exclusiveLocks=3D key: ecf7f507-b0fa-47ee-a8b2-d621fbd7b8bf value: = STORAGE , sharedLocks=3D ] 2013-09-17 08:54:56,292 INFO [org.ovirt.engine.core.bll.storage.ActivateSt= orageDomainCommand] (pool-6-thread-50) [5c6218c1] ActivateStorage Domain. B= efore Connect all hosts to pool. Time:9/17/13 8:54 AM 2013-09-17 08:54:56,296 INFO [org.ovirt.engine.core.bll.storage.ConnectSto= rageToVdsCommand] (pool-6-thread-47) Running command: ConnectStorageToVdsCo= mmand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-1234= 56789aaa Type: System 2013-09-17 08:54:56,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Co= nnectStorageServerVDSCommand] (pool-6-thread-47) START, ConnectStorageServe= rVDSCommand(HostName =3D colovn3, HostId =3D 0fdccd63-f5d7-41e4-8350-5941bb= c29270, storagePoolId =3D 00000000-0000-0000-0000-000000000000, storageType= =3D NFS, connectionList =3D [{ id: 68c31a49-0e37-4438-a8fe-fc28be62cd3f, c= onnection: 10.10.30.251:/var/nas5/ovirt, iqn: null, vfsType: null, mountOpt= ions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id= : 75a9c6a0 2013-09-17 08:54:56,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Co= nnectStorageServerVDSCommand] (pool-6-thread-47) FINISH, ConnectStorageServ= erVDSCommand, return: {68c31a49-0e37-4438-a8fe-fc28be62cd3f=3D0}, log id: 7= 5a9c6a0 --_000_12EF8D94C6F8734FB2FF37B9FBEDD17358578085EXCHANGEcollogi_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html dir=3D"ltr"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <style id=3D"owaParaStyle" type=3D"text/css">P {margin-top:0;margin-bottom:= 0;}</style> </head> <body ocsi=3D"0" fpstyle=3D"1"> <div style=3D"direction: ltr;font-family: Tahoma;color: #000000;font-size: = 10pt;">Hello,<br> <br> maybe a stupid one but ...<br> <br> When I create a (NFS) storage domain I have to provide a node host that mak= es<br> the inital contact. All other node hosts wil directly connect to that domai= n. So<br> no bottlenecks.<br> <br> Today I stopped one of my two nodes outside ovirt-engine. For simplicity we= <br> assume the node crashed. The machine is up right now but VDSM is down.<br> <br> All domains that where setup with this host are "down" now (red a= rrow down). <br> After searching the web interface I found "Data center" -> &qu= ot;Select your DC" -> <br> "Storage" -> "Activate". Trying to activate only res= ults in a failure message.<br> To ensure that I can recover those situations in the future I'd like to kno= w what<br> this node binding is all about and what to do next.<br> <br> Logs attached & thanks in advance<br> <br> Markus<br> <br> 2013-09-17 08:54:54,985 WARN [org.ovirt.engine.core.vdsbroker.irsbrok= er.IrsBrokerCommand] (pool-6-thread-50) [1be325f3] spm vds is non responsiv= e, stopping spm selection.<br> 2013-09-17 08:54:54,986 INFO [org.ovirt.engine.core.vdsbroker.irsbrok= er.ActivateStorageDomainVDSCommand] (pool-6-thread-50) [1be325f3] FINISH, A= ctivateStorageDomainVDSCommand, log id: 4c38e98<br> 2013-09-17 08:54:54,987 ERROR [org.ovirt.engine.core.bll.storage.ActivateSt= orageDomainCommand] (pool-6-thread-50) [1be325f3] Command org.ovirt.engine.= core.bll.storage.ActivateStorageDomainCommand throw Vdc Bll exception. With= error message VdcBLLException: Cannot allocate IRS server (Failed with VDSM error IRS_REPOSITORY_NOT_FOUN= D and code 5009)<br> 2013-09-17 08:54:54,989 INFO [org.ovirt.engine.core.bll.storage.Activ= ateStorageDomainCommand] (pool-6-thread-50) [1be325f3] Command [id=3Da0dbe9= 09-fbb1-40ff-b77a-8e43bd075ace]: Compensating CHANGED_STATUS_ONLY of org.ov= irt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot [id=3DstoragePoolId =3D b054727d-fe4a-41ed-= 8393-a81e36b8a1af, storageId =3D ecf7f507-b0fa-47ee-a8b2-d621fbd7b8bf, stat= us=3DUnknown].<br> 2013-09-17 08:54:55,004 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapa= bilitiesVDS execution failed. Exception: VDSNetworkException: java.net.Conn= ectException: Connection refused<br> 2013-09-17 08:54:55,008 INFO [org.ovirt.engine.core.dal.dbbroker.audi= tloghandling.AuditLogDirector] (pool-6-thread-50) [1be325f3] Correlation ID= : 1be325f3, Job ID: c88c00ba-0298-4f42-bc0e-a720d79c5f49, Call Stack: null,= Custom Event ID: -1, Message: Failed to activate Storage Domain NAS5_IB (Data Center Collogia) by admin@interna= l<br> 2013-09-17 08:54:56,263 INFO [org.ovirt.engine.core.bll.storage.Activ= ateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [5c6218c1] Lock Acquired t= o object EngineLock [exclusiveLocks=3D key: ecf7f507-b0fa-47ee-a8b2-d621fbd= 7b8bf value: STORAGE<br> , sharedLocks=3D ]<br> 2013-09-17 08:54:56,272 INFO [org.ovirt.engine.core.bll.storage.Activ= ateStorageDomainCommand] (pool-6-thread-50) [5c6218c1] Running command: Act= ivateStorageDomainCommand internal: false. Entities affected : ID: ec= f7f507-b0fa-47ee-a8b2-d621fbd7b8bf Type: Storage<br> 2013-09-17 08:54:56,291 INFO [org.ovirt.engine.core.bll.storage.Activ= ateStorageDomainCommand] (pool-6-thread-50) [5c6218c1] Lock freed to object= EngineLock [exclusiveLocks=3D key: ecf7f507-b0fa-47ee-a8b2-d621fbd7b8bf va= lue: STORAGE<br> , sharedLocks=3D ]<br> 2013-09-17 08:54:56,292 INFO [org.ovirt.engine.core.bll.storage.Activ= ateStorageDomainCommand] (pool-6-thread-50) [5c6218c1] ActivateStorage Doma= in. Before Connect all hosts to pool. Time:9/17/13 8:54 AM<br> 2013-09-17 08:54:56,296 INFO [org.ovirt.engine.core.bll.storage.Conne= ctStorageToVdsCommand] (pool-6-thread-47) Running command: ConnectStorageTo= VdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000= -0000-123456789aaa Type: System<br> 2013-09-17 08:54:56,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbrok= er.ConnectStorageServerVDSCommand] (pool-6-thread-47) START, ConnectStorage= ServerVDSCommand(HostName =3D colovn3, HostId =3D 0fdccd63-f5d7-41e4-8350-5= 941bbc29270, storagePoolId =3D 00000000-0000-0000-0000-000000000000, storageType =3D NFS, connectionList =3D [{ id: 68c31a49-0e37-4438-a8fe-fc2= 8be62cd3f, connection: 10.10.30.251:/var/nas5/ovirt, iqn: null, vfsType: nu= ll, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null = };]), log id: 75a9c6a0<br> 2013-09-17 08:54:56,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbrok= er.ConnectStorageServerVDSCommand] (pool-6-thread-47) FINISH, ConnectStorag= eServerVDSCommand, return: {68c31a49-0e37-4438-a8fe-fc28be62cd3f=3D0}, log = id: 75a9c6a0<br> <br> </div> </body> </html> --_000_12EF8D94C6F8734FB2FF37B9FBEDD17358578085EXCHANGEcollogi_-- ------=_NextPartTM-000-161201e6-5459-4708-99a4-b65c101b4bd9 Content-Type: text/plain; name="InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="InterScan_Disclaimer.txt" **************************************************************************** Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Über das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=_NextPartTM-000-161201e6-5459-4708-99a4-b65c101b4bd9--