Re: [Users] NFS Domains down because of single node failure

Hello, maybe a stupid one but ... When I create a (NFS) storage domain I have to provide a node host that makes the inital contact. All other node hosts wil directly connect to that domain. So no bottlenecks. Today I stopped one of my two nodes outside ovirt-engine. For simplicity we assume the node crashed. The machine is up right now but VDSM is down. All domains that where setup with this host are "down" now (red arrow down). After searching the web interface I found "Data center" -> "Select your DC" -> "Storage" -> "Activate". Trying to activate only results in a failure message. To ensure that I can recover those situations in the future I'd like to know what this node binding is all about and what to do next. Logs attached & thanks in advance Markus 2013-09-17 08:54:54,985 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-6-thread-50) [1be325f3] spm vds is non responsive, stopping spm selection. 2013-09-17 08:54:54,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-6-thread-50) [1be325f3] FINISH, ActivateStorageDomainVDSCommand, log id: 4c38e98 2013-09-17 08:54:54,987 ERROR [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-6-thread-50) [1be325f3] Command org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: Cannot allocate IRS server (Failed with VDSM error IRS_REPOSITORY_NOT_FOUND and code 5009) 2013-09-17 08:54:54,989 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-6-thread-50) [1be325f3] Command [id=a0dbe909-fbb1-40ff-b77a-8e43bd075ace]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot [id=storagePoolId = b054727d-fe4a-41ed-8393-a81e36b8a1af, storageId = ecf7f507-b0fa-47ee-a8b2-d621fbd7b8bf, status=Unknown]. 2013-09-17 08:54:55,004 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDS execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused 2013-09-17 08:54:55,008 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-6-thread-50) [1be325f3] Correlation ID: 1be325f3, Job ID: c88c00ba-0298-4f42-bc0e-a720d79c5f49, Call Stack: null, Custom Event ID: -1, Message: Failed to activate Storage Domain NAS5_IB (Data Center Collogia) by admin@internal 2013-09-17 08:54:56,263 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-2) [5c6218c1] Lock Acquired to object EngineLock [exclusiveLocks= key: ecf7f507-b0fa-47ee-a8b2-d621fbd7b8bf value: STORAGE , sharedLocks= ] 2013-09-17 08:54:56,272 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-6-thread-50) [5c6218c1] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: ecf7f507-b0fa-47ee-a8b2-d621fbd7b8bf Type: Storage 2013-09-17 08:54:56,291 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-6-thread-50) [5c6218c1] Lock freed to object EngineLock [exclusiveLocks= key: ecf7f507-b0fa-47ee-a8b2-d621fbd7b8bf value: STORAGE , sharedLocks= ] 2013-09-17 08:54:56,292 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-6-thread-50) [5c6218c1] ActivateStorage Domain. Before Connect all hosts to pool. Time:9/17/13 8:54 AM 2013-09-17 08:54:56,296 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-47) Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2013-09-17 08:54:56,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-47) START, ConnectStorageServerVDSCommand(HostName = colovn3, HostId = 0fdccd63-f5d7-41e4-8350-5941bbc29270, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 68c31a49-0e37-4438-a8fe-fc28be62cd3f, connection: 10.10.30.251:/var/nas5/ovirt, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 75a9c6a0 2013-09-17 08:54:56,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-47) FINISH, ConnectStorageServerVDSCommand, return: {68c31a49-0e37-4438-a8fe-fc28be62cd3f=0}, log id: 75a9c6a0 _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users