[Users] Urgent: Export NFS Migration issue oVirt 3.0 -> 3.2.1
Sven Knohsalla
s.knohsalla at netbiscuits.com
Fri Nov 8 09:32:32 UTC 2013
Hi,
I could eliminate this issue to our oVirt 3.0 instance, as the pool_uuid & SHA checksum in metadata on NFS Export wasn't cleared properly from engine 3.0.
(/NFSmountpoint/<storage-pool-id>/dom_md/metadata)
Main issue:
When activating an imported NFS Export Domain, SPM function from hypervisor will be dropped
and data-center status changes to "inactive". NFS Export Domain cannot be attached successfully an so I currently have no option to export VMs
from oVirt 3.0 to newer oVirt infrastructure !!
HV VDSM version (ovirt-node 16-1 2.3):
3.0.0.0.0
engine.log during action:
2013-11-08 08:12:49,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-5-thr nectionVDSCommand, return: {2a84acc3-1700-45c4-bbf7-a3305b338f83=0}, log id: 207c684e
2013-11-08 08:12:49,075 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-5-thread-39) Running command rue. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2013-11-08 08:12:49,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-39) STA = 66b546c2-ae62-11e1-b734-5254005cbe44, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionLis b338f83, connection: 172.16.101.95:/ovirtmig02 };]), log id: 2482b112
2013-11-08 08:12:52,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-50) FIN urn: {2a84acc3-1700-45c4-bbf7-a3305b338f83=451}, log id: 7dcfb51f
2013-11-08 08:12:52,099 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (pool-5-thread-50) The connection with detai use of error code 451 and error message is: error storage server connection
2013-11-08 08:12:52,105 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-5-thread-50) Transaction rol ore.bll.storage.ConnectStorageToVdsCommand.
2013-11-08 08:12:59,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-5-thr nectionVDSCommand, return: {2a84acc3-1700-45c4-bbf7-a3305b338f83=0}, log id: 1ea6131f
2013-11-08 08:12:59,049 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-5-thread-36) Running command rue. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2013-11-08 08:12:59,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-36) STA = b1be5126-e61c-11e1-b794-5254005cbe44, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionLis b338f83, connection: 172.16.101.95:/ovirtmig02 };]), log id: 50f00f9d
2013-11-08 08:13:09,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-5-thr nectionVDSCommand, return: {2a84acc3-1700-45c4-bbf7-a3305b338f83=453}, log id: 5a39c96b
2013-11-08 08:13:09,108 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (pool-5-thread-48) The connection with detai use of error code 453 and error message is: the specified path does not exist or cannot be reached.
verify the path is correct, and for remote storage,
check the connection to your storage
2013-11-08 08:13:09,108 WARN [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-5-thread-48) CanDoAction of ns:ACTION_TYPE_FAILED_STORAGE_CONNECTION
2013-11-08 08:13:09,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-5-thr nectionVDSCommand, return: {2a84acc3-1700-45c4-bbf7-a3305b338f83=453}, log id: ffa5456
2013-11-08 08:13:09,159 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (pool-5-thread-49) The connection with detai use of error code 453 and error message is: the specified path does not exist or cannot be reached.
verify the path is correct, and for remote storage,
check the connection to your storage
2013-11-08 08:13:09,159 WARN [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-5-thread-49) CanDoAction of ns:ACTION_TYPE_FAILED_STORAGE_CONNECTION
2013-11-08 08:13:12,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-36) FIN urn: {2a84acc3-1700-45c4-bbf7-a3305b338f83=451}, log id: 50f00f9d
2013-11-08 08:13:12,142 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (pool-5-thread-36) The connection with detai use of error code 451 and error message is: error storage server connection
2013-11-08 08:13:12,145 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-5-thread-36) Transaction rol ore.bll.storage.ConnectStorageToVdsCommand.
2013-11-08 08:13:29,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-39) FINISH, ConnectStorageServerVDSCommand, return: {2a84acc3-1700-45c4-bbf7-a3305b338f83=0}, log id: 2482b112
2013-11-08 08:13:29,033 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-5-thread-47) ActivateStorage Domain. After Connect all hosts to pool. Time:11/8/13 8:13 AM
2013-11-08 08:13:29,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-5-thread-47) START, ActivateStorageDomainVDSCommand(storagePoolId = 4e37e18a-ae32-41a3-a558-af2495d64da8, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 7f975678-5bf4-4de2-b3a1-505307fdb695), log id: 6c3aaff5
2013-11-08 08:13:34,327 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Failed in ActivateStorageDomainVDS method
2013-11-08 08:13:34,328 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Error code StorageDomainDoesNotExist and error message IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: ('7f975678-5bf4-4de2-b3a1-505307fdb695',)
2013-11-08 08:13:34,328 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-5-thread-47) IrsBroker::Failed::ActivateStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: ('7f975678-5bf4-4de2-b3a1-505307fdb695',)
2013-11-08 08:13:34,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-5-thread-47) START, SpmStopVDSCommand(vdsId = b1be5126-e61c-11e1-b794-5254005cbe44, storagePoolId = 4e37e18a-ae32-41a3-a558-af2495d64da8), log id: b69d65b
2013-11-08 08:13:34,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-5-thread-47) SpmStopVDSCommand::Stopping SPM on vds deovn-i03, pool id 4e37e18a-ae32-41a3-a558-af2495d64da8
2013-11-08 08:13:35,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-5-thread-47) FINISH, SpmStopVDSCommand, log id: b69d65b
2013-11-08 08:13:35,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-5-thread-47) Irs placed on server null failed. Proceed Failover
2013-11-08 08:13:35,587 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (pool-5-thread-47) Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 4e37e18a-ae32-41a3-a558-af2495d64da8 Type: StoragePool
2013-11-08 08:13:35,615 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-5-thread-47) hostFromVds::selectedVds - deovn-a01, spmStatus Free, storage pool DE-KL-DC01
2013-11-08 08:13:35,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-5-thread-47) SpmStatus on vds b1be5126-e61c-11e1-b794-5254005cbe44: Free
WebGui error message:
The error code for connection 172.16.101.95:/ovirtmig02 returned by VDSM was following The specified path does not exist or cannot be reached. Verify the path is correct, and for remote storage, check the connection to your storage
The error code for connection 172.16.101.95:/ovirtmig02 returned by VDSM was following The specified path does not exist or cannot be reached. Verify the path is correct, and for remote storage, check the connection to your storage
Data Center is being initialized, please wait for initialization to complete.
VDSM log snippet from HV:
Thread-53::ERROR::2013-11-08 08:13:44,031::storage_connection::133::Storage.ServerConnection::(__processConnections) Error during storage connection operation:
Thread-53::ERROR::2013-11-08 08:13:49,343::storage_connection::133::Storage.ServerConnection::(__processConnections) Error during storage connection operation:
Thread-53::ERROR::2013-11-08 08:13:54,410::storage_connection::133::Storage.ServerConnection::(__processConnections) Error during storage connection operation:
Thread-53::ERROR::2013-11-08 08:13:59,700::storage_connection::133::Storage.ServerConnection::(__processConnections) Error during storage connection operation:
After this issue SPM status for all hypervisors is "None" and there's no SPM election triggered, I need to restart vdsmd on former SPM HV to force restarting SPM election again!
even if I create a new NFS share and connect it to oVirt 3.0, I receive the same result.
I need to migrate ~90 productive VMs to our oVirt 3.2 cluster + GlusterFS :/
Is there a workaround / alternative to export VMs or
does anybody has a glue where I may have a look at ?
Thanks in advance four your help!
Regards,
Sven.
Sven Knohsalla | Sr. IT Administrator
Office +49 631 68036 XXX | E-Mail s.knohsalla at netbiscuits.com | Skype: sven.knohsalla
Netbiscuits GmbH | Europaallee 10 | 67657 Kaiserslautern | GERMANY
Register Court: Local Court Kaiserslautern | Commercial Register ID: HR B 3604
Management Board: Guido Moggert, Michael Neidhöfer, Christian Reitz, Martin Süß
This message and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. It may also be privileged or otherwise protected by work product immunity or other legal rules. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
Warning: Although Netbiscuits has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
Please consider the environment before printing
-----Original Message-----
From: Eli Mesika [mailto:emesika at redhat.com]
Sent: Montag, 4. November 2013 12:04
To: Sven Knohsalla
Cc: users at ovirt.org
Subject: Re: [Users] Urgent: Export NFS Migration issue oVirt 3.0 -> 3.2.1
----- Original Message -----
> From: "Sven Knohsalla" <s.knohsalla at netbiscuits.com>
> To: users at ovirt.org
> Sent: Monday, November 4, 2013 9:48:00 AM
> Subject: [Users] Urgent: Export NFS Migration issue oVirt 3.0 -> 3.2.1
>
>
>
> Hi,
>
>
>
> we are currently running in the following issue when trying to migrate
> VMs from oVirt 3.0 -> oVirt 3.2.1 :
>
> NFS Exports are mounted separately to both oVirt instances, both are working.
>
>
> When re-attaching NFS Export with exported VMs from oVirt 3.0 to oVirt
> 3.2.1 the following error message occurs:
> “Failed to attach Storage Domain DE-SD-Export-NFS to Data Center Default.
> (User: X)”
>
> engine log:
> 2013-11-04 08:33:46,763 ERROR
> [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand]
> (pool-3-thread-49) [6348553f] Command
> org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand
> throw Vdc Bll exception. With error message VdcBLLException:
> org.ovirt.engine.core.vdsbroker.irsbroker.IrsOperationFailedNoFailoverException:
> IRSGenericException: IRSErrorException: Storage domain already
> attached to
> pool: 'domain=5c1dec62-3144-4fc4-8ac5-be4e8c04c3ae,
> pool=4e37e18a-ae32-41a3-a558-af2495d64da8'
Are you sure that this is the correct portion of the log? , you say that you are doing a DETACH while the log clearly shows that ATTACH was called.
Can you please attach the full engine + vdsm logs
Thanks
>
> 2013-11-04 08:33:46,772 INFO
> [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand]
> (pool-3-thread-49) [6348553f] Command
> [id=9095ca2b-f875-42f4-8861-1c89805e6b74]: Compensating NEW_ENTITY_ID
> of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot:
> storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, storageId =
> 5c1dec62-3144-4fc4-8ac5-be4e8c04c3ae.
>
>
>
> This issue occurred first when exporting a oVirt 3.0 VM to NFS export
> share of oVirt 3.2.1.
> The opposite way ( attach NFS Export from oVirt 3.2.1 to 3.0) did work
> with no issue.
>
>
> Is this a known bug?
>
> Thanks in advance for your help!
>
> Regards,
>
> Sven.
>
>
>
> Sven Knohsalla | Sr. IT Systems Administrator
>
>
>
> Office +49 631 68036 433 | Fax +49 631 68036 111 |E-Mail
> s.knohsalla at netbiscuits.com | Skype: netbiscuits.admin
>
> Netbiscuits GmbH | Europaallee 10 | 67657 | GERMANY
>
>
>
>
>
>
>
> Register Court: Local Court Kaiserslautern | Commercial Register ID:
> HR B
> 3604
> Management Board : Guido Moggert, Michael Neidhöfer, Christian Reitz,
> Martin Süß
>
>
>
> This message and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they are addressed.
> It may also be privileged or otherwise protected by work product
> immunity or other legal rules. Please notify the sender immediately by
> e-mail if you have received this e-mail by mistake and delete this
> e-mail from your system. If you are not the intended recipient you are
> notified that disclosing, copying, distributing or taking any action
> in reliance on the contents of this information is strictly prohibited.
>
> Warning: Although Netbiscuits has taken reasonable precautions to
> ensure no viruses are present in this email, the company cannot accept
> responsibility for any loss or damage arising from the use of this email or attachments.
>
>
>
> Please consider the environment before printing
>
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: engine.log
Type: application/octet-stream
Size: 926401 bytes
Desc: engine.log
URL: <http://lists.ovirt.org/pipermail/users/attachments/20131108/e69996a1/attachment-0001.obj>
More information about the Users
mailing list