[ovirt-users] NFS storage domain fail after engine upgrade [SOLVED]

Tue Apr 28 12:22:18 UTC 2015

Hi,

The root cause of my problem was that there was a stale NFS mount on my 
hosts. They still had an nfs mount active.

Killing the ioprocess processes that were keeping the mount busy and 
then unmounting the nfs mounts allowed me to activate the iso and export 
domain again.

Regards,

Rik

On 04/28/2015 01:48 PM, Rik Theys wrote:
> Hi,
>
> I migrated my engine from CentOS 6 to CentOS 7 by taking an
> engine-backup on the CentOS 6 install and running the restore on a
> CentOS 7.1 install.
>
> This worked rather well. I can log into the admin webinterface and see
> my still running VM's.
>
> The only issue I'm facing is that the hosts can no longer access the
> export and ISO domain (which are nfs exports on my engine).
>
> When I try to activate the storage domain on a host I get the following
> message in the engine log (see below).
>
> It seems the engine thinks the storage domain does not exist. I copied
> the files from the old installation into the same directories on the new
> installation and I can nfs mount them manually from the hosts. I can
> also nfs mount it on the engine itself.
>
> Any idea on how to debug this?
>
> My engine is running 3.5.1 (actually 3.5.2 now as it just got upgraded,
> but the upgrade did not change anything regarding this bug).
>
> Is there a way to remove the export/iso domain? I can not detach it from
> my data centers using the web interface.
>
> Regards,
>
> Rik
>
>
> 2015-04-28 12:59:19,271 INFO
> [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
> (ajp--127.0.0.1-8702-8) [450f2bb2] Lock Acquired to object EngineLock
> [exclusiveLocks= key: 31ba6486-d4ef-45ae-a184-8296185ef79b value: STORAGE
> , sharedLocks= ]
> 2015-04-28 12:59:19,330 INFO
> [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
> (org.ovirt.thread.pool-8-thread-25) [450f2bb2] Running command:
> ActivateStorageDomainCommand internal: false. Entities affected :  ID:
> 31ba6486-d4ef-45ae-a184-8296185ef79b Type: StorageAction group
> MANIPULATE_STORAGE_DOMAIN with role type ADMIN
> 2015-04-28 12:59:19,360 INFO
> [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
> (org.ovirt.thread.pool-8-thread-25) [450f2bb2] Lock freed to object
> EngineLock [exclusiveLocks= key: 31ba6486-d4ef-45ae-a184-8296185ef79b
> value: STORAGE
> , sharedLocks= ]
> 2015-04-28 12:59:19,362 INFO
> [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
> (org.ovirt.thread.pool-8-thread-25) [450f2bb2] ActivateStorage Domain.
> Before Connect all hosts to pool. Time:4/28/15 12:59 PM
> 2015-04-28 12:59:19,383 INFO
> [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand]
> (org.ovirt.thread.pool-8-thread-28) [3e09aa16] Running command:
> ConnectStorageToVdsCommand internal: true. Entities affected :  ID:
> aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group
> CREATE_STORAGE_DOMAIN with role type ADMIN
> 2015-04-28 12:59:19,388 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand]
> (org.ovirt.thread.pool-8-thread-28) [3e09aa16] START,
> ConnectStorageServerVDSCommand(HostName = stadius-virt2, HostId =
> 7212971a-d38a-42e7-8e6a-24d3396dfa6a, storagePoolId =
> 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList
> = [{ id: 5f18ed21-8c71-4e71-874a-a6a8594c3138, connection:
> iron:/var/lib/exports/export-domain, iqn: null, vfsType: null,
> mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null
> };]), log id: 13d9ec07
> 2015-04-28 12:59:19,409 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand]
> (org.ovirt.thread.pool-8-thread-28) [3e09aa16] FINISH,
> ConnectStorageServerVDSCommand, return:
> {5f18ed21-8c71-4e71-874a-a6a8594c3138=0}, log id: 13d9ec07
> 2015-04-28 12:59:19,417 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
> (org.ovirt.thread.pool-8-thread-25) [450f2bb2] START,
> ActivateStorageDomainVDSCommand( storagePoolId =
> e7bdba88-e718-41a9-8d2b-0ca79c517630, ignoreFailoverLimit = false,
> storageDomainId = 31ba6486-d4ef-45ae-a184-8296185ef79b), log id: 1da2c4c4
> 2015-04-28 12:59:19,774 ERROR
> [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
> (org.ovirt.thread.pool-8-thread-25) [450f2bb2] Failed in
> ActivateStorageDomainVDS method
> 2015-04-28 12:59:19,781 ERROR
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
> (org.ovirt.thread.pool-8-thread-25) [450f2bb2]
> IrsBroker::Failed::ActivateStorageDomainVDS due to: IRSErrorException:
> IRSGenericException: IRSErrorException: Failed to
> ActivateStorageDomainVDS, error = Storage domain does not exist:
> ('31ba6486-d4ef-45ae-a184-8296185ef79b',), code = 358
> 2015-04-28 12:59:19,793 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
> (org.ovirt.thread.pool-8-thread-25) [450f2bb2] FINISH,
> ActivateStorageDomainVDSCommand, log id: 1da2c4c4
> 2015-04-28 12:59:19,794 ERROR
> [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
> (org.ovirt.thread.pool-8-thread-25) [450f2bb2] Command
> org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand throw Vdc
> Bll exception. With error message VdcBLLException:
> org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException:
> IRSGenericException: IRSErrorException: Failed to
> ActivateStorageDomainVDS, error = Storage domain does not exist:
> ('31ba6486-d4ef-45ae-a184-8296185ef79b',), code = 358 (Failed with error
> StorageDomainDoesNotExist and code 358)
> 2015-04-28 12:59:19,819 INFO
> [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
> (org.ovirt.thread.pool-8-thread-25) [450f2bb2] Command
> [id=4e3a7128-ad1f-48fd-b871-ebf89d6cfb87]: Compensating
> CHANGED_STATUS_ONLY of
> org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap;
> snapshot: EntityStatusSnapshot [id=storagePoolId =
> e7bdba88-e718-41a9-8d2b-0ca79c517630, storageId =
> 31ba6486-d4ef-45ae-a184-8296185ef79b, status=Inactive].
> 2015-04-28 12:59:19,832 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (org.ovirt.thread.pool-8-thread-25) [450f2bb2] Correlation ID: 450f2bb2,
> Job ID: 982d9fcc-7b9c-4159-b8c5-ef9414c76dee, Call Stack: null, Custom
> Event ID: -1, Message: Failed to activate Storage Domain iron-export
> (Data Center SENSORED) by XXXXX at XXXX
>
>
>

-- 
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440  - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>