Re: [ovirt-users] NFS storage domain fail after engine upgrade [SOLVED]

Hi, I migrated my engine from CentOS 6 to CentOS 7 by taking an engine-backup on the CentOS 6 install and running the restore on a CentOS 7.1 install. This worked rather well. I can log into the admin webinterface and see my still running VM's. The only issue I'm facing is that the hosts can no longer access the export and ISO domain (which are nfs exports on my engine). When I try to activate the storage domain on a host I get the following message in the engine log (see below). It seems the engine thinks the storage domain does not exist. I copied the files from the old installation into the same directories on the new installation and I can nfs mount them manually from the hosts. I can also nfs mount it on the engine itself. Any idea on how to debug this? My engine is running 3.5.1 (actually 3.5.2 now as it just got upgraded, but the upgrade did not change anything regarding this bug). Is there a way to remove the export/iso domain? I can not detach it from my data centers using the web interface. Regards, Rik 2015-04-28 12:59:19,271 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-8) [450f2bb2] Lock Acquired to object EngineLock [exclusiveLocks= key: 31ba6486-d4ef-45ae-a184-8296185ef79b value: STORAGE , sharedLocks= ] 2015-04-28 12:59:19,330 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-25) [450f2bb2] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: 31ba6486-d4ef-45ae-a184-8296185ef79b Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2015-04-28 12:59:19,360 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-25) [450f2bb2] Lock freed to object EngineLock [exclusiveLocks= key: 31ba6486-d4ef-45ae-a184-8296185ef79b value: STORAGE , sharedLocks= ] 2015-04-28 12:59:19,362 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-25) [450f2bb2] ActivateStorage Domain. Before Connect all hosts to pool. Time:4/28/15 12:59 PM 2015-04-28 12:59:19,383 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-28) [3e09aa16] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2015-04-28 12:59:19,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-28) [3e09aa16] START, ConnectStorageServerVDSCommand(HostName = stadius-virt2, HostId = 7212971a-d38a-42e7-8e6a-24d3396dfa6a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 5f18ed21-8c71-4e71-874a-a6a8594c3138, connection: iron:/var/lib/exports/export-domain, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 13d9ec07 2015-04-28 12:59:19,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-28) [3e09aa16] FINISH, ConnectStorageServerVDSCommand, return: {5f18ed21-8c71-4e71-874a-a6a8594c3138=0}, log id: 13d9ec07 2015-04-28 12:59:19,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-25) [450f2bb2] START, ActivateStorageDomainVDSCommand( storagePoolId = e7bdba88-e718-41a9-8d2b-0ca79c517630, ignoreFailoverLimit = false, storageDomainId = 31ba6486-d4ef-45ae-a184-8296185ef79b), log id: 1da2c4c4 2015-04-28 12:59:19,774 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-25) [450f2bb2] Failed in ActivateStorageDomainVDS method 2015-04-28 12:59:19,781 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-8-thread-25) [450f2bb2] IrsBroker::Failed::ActivateStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: ('31ba6486-d4ef-45ae-a184-8296185ef79b',), code = 358 2015-04-28 12:59:19,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-25) [450f2bb2] FINISH, ActivateStorageDomainVDSCommand, log id: 1da2c4c4 2015-04-28 12:59:19,794 ERROR [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-25) [450f2bb2] Command org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: ('31ba6486-d4ef-45ae-a184-8296185ef79b',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358) 2015-04-28 12:59:19,819 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-25) [450f2bb2] Command [id=4e3a7128-ad1f-48fd-b871-ebf89d6cfb87]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot [id=storagePoolId = e7bdba88-e718-41a9-8d2b-0ca79c517630, storageId = 31ba6486-d4ef-45ae-a184-8296185ef79b, status=Inactive]. 2015-04-28 12:59:19,832 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-25) [450f2bb2] Correlation ID: 450f2bb2, Job ID: 982d9fcc-7b9c-4159-b8c5-ef9414c76dee, Call Stack: null, Custom Event ID: -1, Message: Failed to activate Storage Domain iron-export (Data Center SENSORED) by XXXXX@XXXX