2016-08-14 17:49:32,414 INFO [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (ajp--127.0.0.1-8702-9) [446f3b0e] Lock Acquired to object EngineLock [exclus iveLocks= key: Hertz value: VM_NAME , sharedLocks= key: 49950ad2-05cd-4c5a-bde7-b702199fa250 value: REMOTE_VM ] 2016-08-14 17:49:32,446 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] (ajp--127.0.0.1-8702-9) [446f3b0e] START, DoesImageExistVDSCommand( s toragePoolId = 00000002-0002-0002-0002-00000000009c, ignoreFailoverLimit = false, storageDomainId = 2103720d-5821-482e-a265-2152ec97eb0f, imageGroupId = 29b97d45-e481-4 35d-a50a-f2a890a678bf, imageId = 0225fa05-be1e-44a6-9d9e-1203b8144933), log id: 3d5b900c 2016-08-14 17:49:32,604 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] (ajp--127.0.0.1-8702-9) [446f3b0e] FINISH, DoesImageExistVDSCommand, return: true, log id: 3d5b900c 2016-08-14 17:49:32,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] (ajp--127.0.0.1-8702-9) [446f3b0e] START, DoesImageExistVDSCommand( s toragePoolId = 00000002-0002-0002-0002-00000000009c, ignoreFailoverLimit = false, storageDomainId = a074b2a8-0cd3-4d28-bc29-ad37587a8777, imageGroupId = bba3bd45-d02d-4 032-8bb0-e723b7acf23b, imageId = 20f9d5eb-b4d9-4a5d-b75e-3af013c56604), log id: 464e234c 2016-08-14 17:49:32,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] (ajp--127.0.0.1-8702-9) [446f3b0e] FINISH, DoesImageExistVDSCommand, return: true, log id: 464e234c 2016-08-14 17:49:32,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] (ajp--127.0.0.1-8702-9) [446f3b0e] START, DoesImageExistVDSCommand( s toragePoolId = 00000002-0002-0002-0002-00000000009c, ignoreFailoverLimit = false, storageDomainId = 2103720d-5821-482e-a265-2152ec97eb0f, imageGroupId = 6804229e-68f7-4 6a8-90d3-c065529f9624, imageId = 76b4f302-5182-478d-a7d3-cf4c8b6abe40), log id: 254ca74a 2016-08-14 17:49:32,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] (ajp--127.0.0.1-8702-9) [446f3b0e] FINISH, DoesImageExistVDSCommand, return: true, log id: 254ca74a 2016-08-14 17:49:32,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] (ajp--127.0.0.1-8702-9) [446f3b0e] START, DoesImageExistVDSCommand( s toragePoolId = 00000002-0002-0002-0002-00000000009c, ignoreFailoverLimit = false, storageDomainId = 2103720d-5821-482e-a265-2152ec97eb0f, imageGroupId = 217a34f9-ca8c-4a13-a2a2-9dde4c329c55, imageId = 621c0b0c-c041-46e0-8c00-e69896fcf554), log id: 4b7afe67 2016-08-14 17:49:32,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] (ajp--127.0.0.1-8702-9) [446f3b0e] FINISH, DoesImageExistVDSCommand, return: true, log id: 4b7afe67 2016-08-14 17:49:32,843 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] (ajp--127.0.0.1-8702-9) [446f3b0e] START, DoesImageExistVDSCommand( storagePoolId = 00000002-0002-0002-0002-00000000009c, ignoreFailoverLimit = false, storageDomainId = 2103720d-5821-482e-a265-2152ec97eb0f, imageGroupId = efb67e0b-c592-406a-949c-121102eef796, imageId = 44420cae-7c90-461a-bee1-7552758ecee9), log id: 77d442e7 2016-08-14 17:49:32,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] (ajp--127.0.0.1-8702-9) [446f3b0e] FINISH, DoesImageExistVDSCommand, 2016-08-14 17:49:32,908 INFO [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (org.ovirt.thread.pool-8-thread-42) [446f3b0e] Running command: ImportVmFromConfigurationCommand internal: false. Entities affected : ID: a074b2a8-0cd3-4d28-bc29-ad37587a8777 Type: StorageAction group IMPORT_EXPORT_VM with role type ADMIN, ID: 2103720d-5821-482e-a265-2152ec97eb0f Type: StorageAction group IMPORT_EXPORT_VM with role type ADMIN, ID: 2103720d-5821-482e-a265-2152ec97eb0f Type: StorageAction group IMPORT_EXPORT_VM with role type ADMIN, ID: 2103720d-5821-482e-a265-2152ec97eb0f Type: StorageAction group IMPORT_EXPORT_VM with role type ADMIN, ID: 2103720d-5821-482e-a265-2152ec97eb0f Type: StorageAction group IMPORT_EXPORT_VM with role type ADMIN 2016-08-14 17:49:33,079 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (org.ovirt.thread.pool-8-thread-42) [446f3b0e] transaction rolled back 2016-08-14 17:49:33,080 ERROR [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (org.ovirt.thread.pool-8-thread-42) [446f3b0e] Command org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand throw exception: org.springframework.dao.DuplicateKeyException: CallableStatementCallback; SQL [{call insertimage(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}]; ERROR: duplicate key value violates unique constraint "pk_images" Where: SQL statement "INSERT INTO images( creation_date, image_guid, it_guid, size, ParentId, imageStatus, lastModified, vm_snapshot_id, volume_type, image_group_id, volume_format, active) VALUES( $1 , $2 , $3 , $4 , $5 , $6 , $7 , $8 , $9 , $10 , $11 , $12 )" PL/pgSQL function "insertimage" line 2 at SQL statement; nested exception is org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pk_images" Where: SQL statement "INSERT INTO images( creation_date, image_guid, it_guid, size, ParentId, imageStatus, lastModified, vm_snapshot_id, volume_type, image_group_id, volume_format, active) VALUES( $1 , $2 , $3 , $4 , $5 , $6 , $7 , $8 , $9 , $10 , $11 , $12 )" PL/pgSQL function "insertimage" line 2 at SQL statement at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:241) [spring-jdbc.jar:3.1.1.RELEASE] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72) [spring-jdbc.jar:3.1.1.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1030) [spring-jdbc.jar:3.1.1.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:1064) [spring-jdbc.jar:3.1.1.RELEASE] at org.springframework.jdbc.core.simple.AbstractJdbcCall.executeCallInternal(AbstractJdbcCall.java:388) [spring-jdbc.jar:3.1.1.RELEASE] at org.springframework.jdbc.core.simple.AbstractJdbcCall.doExecute(AbstractJdbcCall.java:351) [spring-jdbc.jar:3.1.1.RELEASE] at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:141) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:136) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeModification(SimpleJdbcCallsHandler.java:74) [dal.jar:] at org.ovirt.engine.core.dao.DefaultGenericDaoDbFacade.save(DefaultGenericDaoDbFacade.java:93) [dal.jar:] at org.ovirt.engine.core.bll.BaseImagesCommand.saveImage(BaseImagesCommand.java:405) [bll.jar:] at org.ovirt.engine.core.bll.ImportVmCommand.saveImage(ImportVmCommand.java:952) [bll.jar:] at org.ovirt.engine.core.bll.ImportVmCommand.addVmImagesAndSnapshots(ImportVmCommand.java:917) [bll.jar:] at org.ovirt.engine.core.bll.ImportVmCommand$4.runInTransaction(ImportVmCommand.java:737) [bll.jar:] at org.ovirt.engine.core.bll.ImportVmCommand$4.runInTransaction(ImportVmCommand.java:733) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:210) [utils.jar:] at org.ovirt.engine.core.bll.ImportVmCommand.processImages(ImportVmCommand.java:733) [bll.jar:] at org.ovirt.engine.core.bll.ImportVmCommand.executeCommand(ImportVmCommand.java:702) [bll.jar:] at org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand.executeCommand(ImportVmFromConfigurationCommand.java:129) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1199) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1338) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1967) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1362) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:359) [bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:193) [bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:160) [bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:169) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:90) [utils.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_111] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_111] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_111] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_111] Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pk_images" Where: SQL statement "INSERT INTO images( creation_date, image_guid, it_guid, size, ParentId, imageStatus, lastModified, vm_snapshot_id, volume_type, image_group_id, volume_format, active) VALUES( $1 , $2 , $3 , $4 , $5 , $6 , $7 , $8 , $9 , $10 , $11 , $12 )" PL/pgSQL function "insertimage" line 2 at SQL statement at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2103) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:512) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:381) at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.execute(CachedPreparedStatement.java:297) at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:404) at org.springframework.jdbc.core.JdbcTemplate$6.doInCallableStatement(JdbcTemplate.java:1066) [spring-jdbc.jar:3.1.1.RELEASE] at org.springframework.jdbc.core.JdbcTemplate$6.doInCallableStatement(JdbcTemplate.java:1) [spring-jdbc.jar:3.1.1.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1014) [spring-jdbc.jar:3.1.1.RELEASE] ... 33 more 2016-08-14 17:49:33,094 INFO [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (org.ovirt.thread.pool-8-thread-42) [446f3b0e] Command [id=aa68938d-49ec-4094-ac8d-9bc06376d222]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.VmStatistics; snapshot: 49950ad2-05cd-4c5a-bde7-b702199fa250. 2016-08-14 17:49:33,095 INFO [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (org.ovirt.thread.pool-8-thread-42) [446f3b0e] Command [id=aa68938d-49ec-4094-ac8d-9bc06376d222]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.network.VmNetworkStatistics; snapshot: ee0be571-554e-4a2f-9822-efc134f068fb. 2016-08-14 17:49:33,097 INFO [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (org.ovirt.thread.pool-8-thread-42) [446f3b0e] Command [id=aa68938d-49ec-4094-ac8d-9bc06376d222]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.network.VmNetworkInterface; snapshot: ee0be571-554e-4a2f-9822-efc134f068fb. 2016-08-14 17:49:33,098 INFO [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (org.ovirt.thread.pool-8-thread-42) [446f3b0e] Command [id=aa68938d-49ec-4094-ac8d-9bc06376d222]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.VmDynamic; snapshot: 49950ad2-05cd-4c5a-bde7-b702199fa250. 2016-08-14 17:49:33,099 INFO [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (org.ovirt.thread.pool-8-thread-42) [446f3b0e] Command [id=aa68938d-49ec-4094-ac8d-9bc06376d222]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.VmStatic; snapshot: 49950ad2-05cd-4c5a-bde7-b702199fa250. 2016-08-14 17:49:33,108 INFO [org.ovirt.engine.core.bll.ImportVmFromConfigurationCommand] (org.ovirt.thread.pool-8-thread-42) [446f3b0e] Lock freed to object EngineLock [exclusiveLocks= key: Hertz value: VM_NAME , sharedLocks= key: 49950ad2-05cd-4c5a-bde7-b702199fa250 value: REMOTE_VM ] 2016-08-14 17:50:06,441 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) starting processDomainRecovery for domain 2103720d-5821-482e-a265-2152ec97eb0f:Hertz-dstore1-L 2016-08-14 17:50:06,551 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) vds primitivo reported domain 2103720d-5821-482e-a265-2152ec97eb0f:Hertz-dstore1-L as in problem, attempting to move the vds to status NonOperational 2016-08-14 17:50:06,552 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) vds sovana reported domain 2103720d-5821-482e-a265-2152ec97eb0f:Hertz-dstore1-L as in problem, attempting to move the vds to status NonOperational 2016-08-14 17:50:06,552 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) vds Lambrusco reported domain 2103720d-5821-482e-a265-2152ec97eb0f:Hertz-dstore1-L as in problem, attempting to move the vds to status NonOperational 2016-08-14 17:50:06,553 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) Removing vds [6d37d71e-c8c5-47c8-a896-5112298bb83f, 89254f23-8748-402a-afc9-08438dca0975, a21e6b4f-7d06-49c5-8463-680d78517c73] from the domain in maintenance cache 2016-08-14 17:50:06,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-26) Removing host(s) [6d37d71e-c8c5-47c8-a896-5112298bb83f, 89254f23-8748-402a-afc9-08438dca0975, a21e6b4f-7d06-49c5-8463-680d78517c73] from hosts unseen domain report cache 2016-08-14 17:50:06,595 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-8-thread-38) [11443d31] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 89254f23-8748-402a-afc9-08438dca0975 Type: VDS 2016-08-14 17:50:06,607 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-8-thread-34) [170d64d2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: a21e6b4f-7d06-49c5-8463-680d78517c73 Type: VDS 2016-08-14 17:50:06,610 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-38) [11443d31] START, SetVdsStatusVDSCommand(HostName = sovana, HostId = 89254f23-8748-402a-afc9-08438dca0975, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6463356e 2016-08-14 17:50:06,621 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-8-thread-26) [4449c225] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 6d37d71e-c8c5-47c8-a896-5112298bb83f Type: VDS 2016-08-14 17:50:06,623 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-34) [170d64d2] START, SetVdsStatusVDSCommand(HostName = Lambrusco, HostId = a21e6b4f-7d06-49c5-8463-680d78517c73, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 704153b7 2016-08-14 17:50:06,623 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-38) [11443d31] FINISH, SetVdsStatusVDSCommand, log id: 6463356e 2016-08-14 17:50:06,635 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-34) [170d64d2] FINISH, SetVdsStatusVDSCommand, log id: 704153b7 2016-08-14 17:50:06,644 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-26) [4449c225] START, SetVdsStatusVDSCommand(HostName = primitivo, HostId = 6d37d71e-c8c5-47c8-a896-5112298bb83f, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 522d70ef 2016-08-14 17:50:06,657 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-38) [11443d31] Correlation ID: 11443d31, Job ID: 144d8e4d-3560-4a13-9307-2c08977b6c0c, Call Stack: null, Custom Event ID: -1, Message: Host sovana cannot access the Storage Domain(s) Hertz-dstore1-L attached to the Data Center IRDC-INTEL. Setting Host state to Non-Operational. 2016-08-14 17:50:06,666 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-34) [170d64d2] Correlation ID: 170d64d2, Job ID: 91f007cf-2d72-4f69-a7a0-023f150a1c59, Call Stack: null, Custom Event ID: -1, Message: Host Lambrusco cannot access the Storage Domain(s) Hertz-dstore1-L attached to the Data Center IRDC-INTEL. Setting Host state to Non-Operational.