This is a multi-part message in MIME format. --------------000906040806080001080808 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi list, I had to do some fsck-related things last week on my ovirt box (centos, single node and NFS on localhost). Afterwards ovirt refused to start VMs according to the webinterface because it can't connect host to Data Center. On my OS it works fine (I just replaced the IP by 'IP'): [root@server ovirt-engine]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg_leiden-lv_root 51606140 6872592 42112108 15% / tmpfs 3978120 0 3978120 0% /dev/shm /dev/sdb1 495844 99542 370702 22% /boot /dev/mapper/vg_server-lv_home 62990260 2584272 57206196 5% /home /dev/sda1 5814366992 4629615452 1184751540 80% /raid IP:/raid/ovirt/data 5814367232 4629615616 1184751616 80% /rhev/data-center/mnt/IP:_raid_ovirt_data IP:/raid/ovirt/iso 5814367232 4629615616 1184751616 80% /rhev/data-center/mnt/IP:_raid_ovirt_iso in my event log in the webinterface I found correlation ID 7a735111 . So grepping my logs for that one: engine.log:2014-04-30 15:59:22,851 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-8) [7a735111] Lock Acquired to object EngineLock [exclusiveLocks= key: 6bee0e2d-961c-453d-a266-e4623f91e162 value: STORAGE engine.log:2014-04-30 15:59:22,888 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: 6bee0e2d-961c-453d-a266-e4623f91e162 Type: Storage engine.log:2014-04-30 15:59:22,894 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] Lock freed to object EngineLock [exclusiveLocks= key: 6bee0e2d-961c-453d-a266-e4623f91e162 value: STORAGE engine.log:2014-04-30 15:59:22,895 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] ActivateStorage Domain. Before Connect all hosts to pool. Time:4/30/14 3:59 PM engine.log:2014-04-30 15:59:22,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000000ec, ignoreFailoverLimit = false, storageDomainId = 6bee0e2d-961c-453d-a266-e4623f91e162), log id: 5fecb439 engine.log:2014-04-30 15:59:23,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] hostFromVds::selectedVds - server, spmStatus Unknown_Pool, storage pool Default engine.log:2014-04-30 15:59:23,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] START, ConnectStoragePoolVDSCommand(HostName = server, HostId = ff23de79-f17c-439d-939e-d8f3d9672367, storagePoolId = 00000002-0002-0002-0002-0000000000ec, vds_spm_id = 1, masterDomainId = 6bee0e2d-961c-453d-a266-e4623f91e162, masterVersion = 1), log id: 15866a95 engine.log:2014-04-30 15:59:23,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] FINISH, ConnectStoragePoolVDSCommand, log id: 15866a95 engine.log:2014-04-30 15:59:23,152 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] IrsBroker::Failed::ActivateStorageDomainVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) engine.log:2014-04-30 15:59:23,156 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] FINISH, ActivateStorageDomainVDSCommand, log id: 5fecb439 engine.log:2014-04-30 15:59:23,157 ERROR [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] Command org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) (Failed with error ENGINE and code 5001) engine.log:2014-04-30 15:59:23,162 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] Command [id=c2d8b433-f203-4d9e-b241-222eebf3dbae]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot [id=storagePoolId = 00000002-0002-0002-0002-0000000000ec, storageId = 6bee0e2d-961c-453d-a266-e4623f91e162, status=InActive]. engine.log:2014-04-30 15:59:23,170 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [7a735111] Correlation ID: 7a735111, Job ID: 75dcda45-e0a3-46e3-b79f-8df9f0ed9d85, Call Stack: null, Custom Event ID: -1, Message: Failed to activate Storage Domain data (Data Center Default) by admin I guess the most relevant error is: engine.log:2014-04-30 15:59:23,152 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] IrsBroker::Failed::ActivateStorageDomainVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) Okay so it can't connect to the data center. Well the NFS storage is mounted and looks fine. The directory is indeed looking fine: [root@leiden data]# ls -al total 12 drwxr-xr-x. 3 vdsm kvm 4096 Mar 30 03:27 . drwxr-xr-x. 4 vdsm kvm 4096 Mar 30 03:26 .. drwxr-xr-x. 5 vdsm kvm 4096 Mar 30 03:27 6bee0e2d-961c-453d-a266-e4623f91e162 -rwxr-xr-x. 1 vdsm kvm 0 Mar 30 03:27 __DIRECT_IO_TEST__ (and in the uuid there are indeed my VMs). How should I solve this? To me everything looks just fine. Cheers Boudewijn Ector --------------000906040806080001080808 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-text-plain" wrap="true" graphical-quote="true" style="font-family: -moz-fixed; font-size: 14px;" lang="x-western"> <pre wrap="">Hi list, I had to do some fsck-related things last week on my ovirt box (centos, single node and NFS on localhost). Afterwards ovirt refused to start VMs according to the webinterface because it can't connect host to Data Center. On my OS it works fine (I just replaced the IP by 'IP'): [root@server ovirt-engine]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg_leiden-lv_root 51606140 6872592 42112108 15% / tmpfs 3978120 0 3978120 0% /dev/shm /dev/sdb1 495844 99542 370702 22% /boot /dev/mapper/vg_server-lv_home 62990260 2584272 57206196 5% /home /dev/sda1 5814366992 4629615452 1184751540 80% /raid IP:/raid/ovirt/data 5814367232 4629615616 1184751616 80% /rhev/data-center/mnt/IP:_raid_ovirt_data IP:/raid/ovirt/iso 5814367232 4629615616 1184751616 80% /rhev/data-center/mnt/IP:_raid_ovirt_iso in my event log in the webinterface I found correlation ID 7a735111 . So grepping my logs for that one: engine.log:2014-04-30 15:59:22,851 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-8) [7a735111] Lock Acquired to object EngineLock [exclusiveLocks= key: 6bee0e2d-961c-453d-a266-e4623f91e162 value: STORAGE engine.log:2014-04-30 15:59:22,888 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: 6bee0e2d-961c-453d-a266-e4623f91e162 Type: Storage engine.log:2014-04-30 15:59:22,894 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] Lock freed to object EngineLock [exclusiveLocks= key: 6bee0e2d-961c-453d-a266-e4623f91e162 value: STORAGE engine.log:2014-04-30 15:59:22,895 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] ActivateStorage Domain. Before Connect all hosts to pool. Time:4/30/14 3:59 PM engine.log:2014-04-30 15:59:22,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] START, ActivateStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000000ec, ignoreFailoverLimit = false, storageDomainId = 6bee0e2d-961c-453d-a266-e4623f91e162), log id: 5fecb439 engine.log:2014-04-30 15:59:23,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] hostFromVds::selectedVds - server, spmStatus Unknown_Pool, storage pool Default engine.log:2014-04-30 15:59:23,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] START, ConnectStoragePoolVDSCommand(HostName = server, HostId = ff23de79-f17c-439d-939e-d8f3d9672367, storagePoolId = 00000002-0002-0002-0002-0000000000ec, vds_spm_id = 1, masterDomainId = 6bee0e2d-961c-453d-a266-e4623f91e162, masterVersion = 1), log id: 15866a95 engine.log:2014-04-30 15:59:23,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] FINISH, ConnectStoragePoolVDSCommand, log id: 15866a95 engine.log:2014-04-30 15:59:23,152 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] IrsBroker::Failed::ActivateStorageDomainVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) engine.log:2014-04-30 15:59:23,156 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] FINISH, ActivateStorageDomainVDSCommand, log id: 5fecb439 engine.log:2014-04-30 15:59:23,157 ERROR [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] Command org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) (Failed with error ENGINE and code 5001) engine.log:2014-04-30 15:59:23,162 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] Command [id=c2d8b433-f203-4d9e-b241-222eebf3dbae]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot [id=storagePoolId = 00000002-0002-0002-0002-0000000000ec, storageId = 6bee0e2d-961c-453d-a266-e4623f91e162, status=InActive]. engine.log:2014-04-30 15:59:23,170 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [7a735111] Correlation ID: 7a735111, Job ID: 75dcda45-e0a3-46e3-b79f-8df9f0ed9d85, Call Stack: null, Custom Event ID: -1, Message: Failed to activate Storage Domain data (Data Center Default) by admin I guess the most relevant error is: engine.log:2014-04-30 15:59:23,152 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-49) [7a735111] IrsBroker::Failed::ActivateStorageDomainVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) Okay so it can't connect to the data center. Well the NFS storage is mounted and looks fine. The directory is indeed looking fine: [root@leiden data]# ls -al total 12 drwxr-xr-x. 3 vdsm kvm 4096 Mar 30 03:27 . drwxr-xr-x. 4 vdsm kvm 4096 Mar 30 03:26 .. drwxr-xr-x. 5 vdsm kvm 4096 Mar 30 03:27 6bee0e2d-961c-453d-a266-e4623f91e162 -rwxr-xr-x. 1 vdsm kvm 0 Mar 30 03:27 __DIRECT_IO_TEST__ (and in the uuid there are indeed my VMs). How should I solve this? To me everything looks just fine. Cheers Boudewijn Ector </pre> </div> </body> </html> --------------000906040806080001080808--