On Mon, Oct 7, 2013 at 2:59 AM, Dan Kenigsberg wrote:
Would you please test if http://gerrit.ovirt.org/19906 solves the issue? (I haven't. Too late at night.)
Regards, Dan.
I can confirm that it resolves https://bugzilla.redhat.com/show_bug.cgi?id=1007980 so now I'm able to start VM without having to select run once and attaching a cd iso (note that is only valid for newly created VMs though) But migration still fails On engine: 2013-10-08 01:20:51,459 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-15) START, GlusterV olumesListVDSCommand(HostName = f18ovn03, HostId = b67bcfd4-f868-49d5-8704-4936ee922249), log id: 4425a532 2013-10-08 01:20:51,547 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, Gluster VolumesListVDSCommand, return: {ed71a4c2-6205-4aad-9aab-85da086d5ba3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@8defb13f , c8cbcac7-1d40-4cee-837d-bb97467fb2bd=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@91be2b8a}, log id: 4425a532 2013-10-08 01:20:51,697 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) RefreshVmList vm id d54660a2-45 ed-41ae-ab99-a6f93ebbdbb1 is migrating to vds f18ovn01 ignoring it in the refresh until migration is done 2013-10-08 01:20:54,765 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand] (pool-6-thread-38) START, GlusterServicesListVDSC ommand(HostName = f18ovn01, HostId = 80188ccc-83b2-4bc8-9385-8d07f7458a3c), log id: 41ed75a4 2013-10-08 01:20:54,766 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand] (pool-6-thread-36) START, GlusterServicesListVDSC ommand(HostName = f18ovn03, HostId = b67bcfd4-f868-49d5-8704-4936ee922249), log id: 7eea38bc 2013-10-08 01:20:56,618 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-2) START, GlusterVo lumesListVDSCommand(HostName = f18ovn01, HostId = 80188ccc-83b2-4bc8-9385-8d07f7458a3c), log id: 34d0827b 2013-10-08 01:20:56,722 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-84) Command ListVDS execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset 2013-10-08 01:20:56,725 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand] (pool-6-thread-36) Command GlusterServicesListVDS execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset 2013-10-08 01:20:56,725 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand] (pool-6-thread-36) FINISH, GlusterServicesListVDSCommand, log id: 7eea38bc 2013-10-08 01:20:56,727 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Failed to refresh VDS , vds = b67bcfd4-f868-49d5-8704-4936ee922249 : f18ovn03, VDS Network Error, continuing. java.net.SocketException: Connection reset 2013-10-08 01:20:58,583 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-48) Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: eb679feb-4da2-4fd0-a185-abbe459ffa70 Type: StoragePool 2013-10-08 01:20:58,603 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) Correlation ID: 16f779a3, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Gluster. Setting status to Non Responsive. 2013-10-08 01:20:58,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2013-10-08 01:20:58,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-48) HostName = f18ovn03 2013-10-08 01:20:58,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-48) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2013-10-08 01:20:59,356 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-41) Command ListVDS execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset On host where VM was running vdsm.log contains: Thread-4643::DEBUG::2013-10-08 01:20:53,700::BindingXMLRPC::979::vds::(wrapper) client [10.4.4.60]::call vmMigrate with ({'tunneled': 'false', 'dst': '10.4.4.58:54321', 'vmId': 'd54660a2-45ed-41ae-ab99-a6f93ebbdbb1', 'method': 'online', 'src': '10.4.4.59'},) {} flowID [7c55a4a9] Thread-4643::DEBUG::2013-10-08 01:20:53,700::API::462::vds::(migrate) {'tunneled': 'false', 'dst': '10.4.4.58:54321', 'vmId': 'd54660a2-45ed-41ae-ab99-a6f93ebbdbb1', 'method': 'online', 'src': '10.4.4.59'} Thread-4643::DEBUG::2013-10-08 01:20:53,701::BindingXMLRPC::986::vds::(wrapper) return vmMigrate with {'status': {'message': 'Migration in progress', 'code': 0}, 'progress': 0} Thread-4644::DEBUG::2013-10-08 01:20:53,702::vm::182::vm.Vm::(_setupVdsConnection) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Destination server is: 10.4.4.58:54321 Thread-4644::DEBUG::2013-10-08 01:20:53,703::vm::184::vm.Vm::(_setupVdsConnection) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Initiating connection with destination Thread-4644::DEBUG::2013-10-08 01:20:53,718::vm::238::vm.Vm::(_prepareGuest) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Migration started Thread-4644::DEBUG::2013-10-08 01:20:53,739::vm::302::vm.Vm::(run) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::migration semaphore acquired Thread-4644::DEBUG::2013-10-08 01:20:53,933::vm::360::vm.Vm::(_startUnderlyingMigration) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::starting migration to qemu+tls://10.4.4.58/system with miguri tcp://10.4.4.58 Thread-4645::DEBUG::2013-10-08 01:20:53,934::vm::718::vm.Vm::(run) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::migration downtime thread started Thread-4646::DEBUG::2013-10-08 01:20:53,935::vm::756::vm.Vm::(run) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::starting migration monitor thread Thread-4648::DEBUG::2013-10-08 01:20:54,321::BindingXMLRPC::979::vds::(wrapper) client [10.4.4.60]::call volumesList with () {} Thread-4648::DEBUG::2013-10-08 01:20:54,349::BindingXMLRPC::986::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'gviso': {'transportType': ['TCP'], 'uuid': 'c8cbcac7-1d40-4cee-837d-bb97467fb2bd', 'bricks': ['f18ovn01.mydomain:/gluster/ISO_GLUSTER/brick1', 'f18ovn03.mydomain:/gluster/ISO_GLUSTER/brick1'], 'volumeName': 'gviso', 'volumeType': 'REPLICATE', 'replicaCount': '2', 'brickCount': '2', 'distCount': '2', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'storage.owner-gid': '36', 'storage.owner-uid': '36', 'server.allow-insecure': 'on'}}, 'gvdata': {'transportType': ['TCP'], 'uuid': 'ed71a4c2-6205-4aad-9aab-85da086d5ba3', 'bricks': ['f18ovn01.mydomain:/gluster/DATA_GLUSTER/brick1', 'f18ovn03.mydomain:/gluster/DATA_GLUSTER/brick1'], 'volumeName': 'gvdata', 'volumeType': 'REPLICATE', 'replicaCount': '2', 'brickCount': '2', 'distCount': '2', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'server.allow-insecure': 'on', 'storage.owner-uid': '36', 'storage.owner-gid': '36'}}}} Thread-4644::ERROR::2013-10-08 01:20:54,873::libvirtconnection::94::libvirtconnection::(wrapper) connection to libvirt broken. ecode: 38 edom: 7 Thread-4644::ERROR::2013-10-08 01:20:54,873::libvirtconnection::96::libvirtconnection::(wrapper) taking calling process down. MainThread::DEBUG::2013-10-08 01:20:54,874::vdsm::45::vds::(sigtermHandler) Received signal 15 Thread-4644::DEBUG::2013-10-08 01:20:54,874::vm::733::vm.Vm::(cancel) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::canceling migration downtime thread Thread-4644::DEBUG::2013-10-08 01:20:54,875::vm::803::vm.Vm::(stop) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::stopping migration monitor thread Thread-4645::DEBUG::2013-10-08 01:20:54,875::vm::730::vm.Vm::(run) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::migration downtime thread exiting Thread-4644::ERROR::2013-10-08 01:20:54,875::vm::244::vm.Vm::(_recover) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Cannot recv data: Input/output error Thread-4644::ERROR::2013-10-08 01:20:55,008::vm::324::vm.Vm::(run) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 311, in run self._startUnderlyingMigration() File "/usr/share/vdsm/vm.py", line 388, in _startUnderlyingMigration None, maxBandwidth) File "/usr/share/vdsm/vm.py", line 826, in f ret = attr(*args, **kwargs) File "/usr/lib64/python2.7/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1253, in migrateToURI2 if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self) libvirtError: Cannot recv data: Input/output error MainThread::INFO::2013-10-08 01:20:55,321::vmChannels::183::vds::(stop) VM channels listener was stopped. MainThread::INFO::2013-10-08 01:20:55,322::momIF::81::MOM::(stop) Shutting down MOM MainThread::DEBUG::2013-10-08 01:20:55,322::task::579::TaskManager.Task::(_updateState) Task=`23df8b46-c916-4c52-98a9-3e6540171467`::moving from state init -> state preparing MainThread::INFO::2013-10-08 01:20:55,323::logUtils::44::dispatcher::(wrapper) Run and protect: prepareForShutdown(options=None) Thread-11::DEBUG::2013-10-08 01:20:55,323::storageServer::698::ConnectionMonitor::(_monitorConnections) Monitoring stopped VM Channels Listener::INFO::2013-10-08 01:20:55,672::vmChannels::178::vds::(run) VM channels listener thread has ended. Thread-23::DEBUG::2013-10-08 01:20:55,929::domainMonitor::161::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for cc790a86-72e2-4aa2-a0b6-700756c399c3 Thread-24::DEBUG::2013-10-08 01:20:55,930::domainMonitor::161::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291 Thread-24::INFO::2013-10-08 01:20:55,930::clusterlock::197::SANLock::(releaseHostId) Releasing host id for domain d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291 (id: 2) Thread-24::DEBUG::2013-10-08 01:20:55,930::domainMonitor::170::Storage.DomainMonitorThread::(_monitorLoop) Unable to release the host id 2 for domain d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291 Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 167, in _monitorLoop self.domain.releaseHostId(self.hostId, unused=True) File "/usr/share/vdsm/storage/sd.py", line 461, in releaseHostId self._clusterLock.releaseHostId(hostId, async, unused) File "/usr/share/vdsm/storage/clusterlock.py", line 204, in releaseHostId raise se.ReleaseHostIdFailure(self._sdUUID, e) ReleaseHostIdFailure: Cannot release host id: ('d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291', SanlockException(16, 'Sanlock lockspace remove failure', 'Device or resource busy')) MainThread::DEBUG::2013-10-08 01:20:55,932::taskManager::80::TaskManager::(prepareForShutdown) Request to stop all tasks MainThread::INFO::2013-10-08 01:20:55,932::logUtils::47::dispatcher::(wrapper) Run and protect: prepareForShutdown, Return response: None MainThread::DEBUG::2013-10-08 01:20:55,932::task::1168::TaskManager.Task::(prepare) Task=`23df8b46-c916-4c52-98a9-3e6540171467`::finished: None MainThread::DEBUG::2013-10-08 01:20:55,933::task::579::TaskManager.Task::(_updateState) Task=`23df8b46-c916-4c52-98a9-3e6540171467`::moving from state preparing -> state finished On target host vdsm.log: Thread-3393::DEBUG::2013-10-08 01:20:53,900::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291' (0 active users) Thread-3393::DEBUG::2013-10-08 01:20:53,900::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291' is free, finding out if anyone is waiting for it. Thread-3393::DEBUG::2013-10-08 01:20:53,900::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291', Clearing records. Thread-3393::DEBUG::2013-10-08 01:20:53,901::task::974::TaskManager.Task::(_decref) Task=`be76585d-3de3-4520-873e-9eb501e12311`::ref 0 aborting False Thread-3393::INFO::2013-10-08 01:20:53,901::clientIF::325::vds::(prepareVolumePath) prepared volume path: /rhev/data-center/eb679feb-4da2-4fd0-a185-abbe459ffa70/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49 Thread-3393::INFO::2013-10-08 01:20:53,901::clientIF::325::vds::(prepareVolumePath) prepared volume path: gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49 Thread-3393::DEBUG::2013-10-08 01:20:53,907::vm::2036::vm.Vm::(_startUnderlyingVm) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::_ongoingCreations released Thread-3393::DEBUG::2013-10-08 01:20:53,908::vm::3347::vm.Vm::(_waitForIncomingMigrationFinish) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Waiting 300 seconds for end of migration Thread-3392::ERROR::2013-10-08 01:20:53,907::vm::4789::vm.Vm::(_updateDevicesDomxmlCache) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Alias not found for device type disk during migration at destination host Thread-3392::DEBUG::2013-10-08 01:20:53,918::API::510::vds::(migrationCreate) Destination VM creation succeeded Thread-3392::DEBUG::2013-10-08 01:20:53,918::BindingXMLRPC::986::vds::(wrapper) return vmMigrationCreate with {'status': {'message': 'Done', 'code': 0}, 'migrationPort': 0, 'params': {'guestFQDN': '', 'acpiEnable': 'true', 'emulatedMachine': 'pc-1.0', 'afterMigrationStatus': 'Up', 'spiceSecureChannels': 'smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard', 'pid': '0', 'memGuaranteedSize': 1024, 'transparentHugePages': 'true', 'displaySecurePort': '-1', 'timeOffset': '-114', 'cpuType': 'Opteron_G2', 'smp': '1', 'pauseCode': 'NOERR', 'migrationDest': 'libvirt', 'custom': {'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19edevice_9c0a4779-62bc-414b-abb5-a2fab36104bedevice_1459cc81-2f88-45dd-ae3c-0c030a084d2e': 'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, deviceId=1459cc81-2f88-45dd-ae3c-0c030a084d2e, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}}', 'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19edevice_9c0a4779-62bc-414b-abb5-a2fab36104bedevice_1459cc81-2f88-45dd-ae3c-0c030a084d2edevice_5bea8f60-7de6-4c93-99ff-af67635e2af6device_8d086706-28b6-40fa-a86d-3733c9ebe5c1': 'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, deviceId=8d086706-28b6-40fa-a86d-3733c9ebe5c1, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}}', 'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19e': 'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, deviceId=db19ff20-8f6d-488c-bad8-19e33b01f19e, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}}', 'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19edevice_9c0a4779-62bc-414b-abb5-a2fab36104be': 'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, deviceId=9c0a4779-62bc-414b-abb5-a2fab36104be, device=virtio-serial, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x05, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-serial0, customProperties={}}', 'device_a5fa5a6d-f191-42d2-be55-d6227499c658': 'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, deviceId=a5fa5a6d-f191-42d2-be55-d6227499c658, device=scsi, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x04, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=scsi0, customProperties={}}', 'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19edevice_9c0a4779-62bc-414b-abb5-a2fab36104bedevice_1459cc81-2f88-45dd-ae3c-0c030a084d2edevice_5bea8f60-7de6-4c93-99ff-af67635e2af6': 'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, deviceId=5bea8f60-7de6-4c93-99ff-af67635e2af6, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}}', 'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9': 'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, deviceId=bd099ee2-faee-41c3-9e01-0a01f458abe9, device=disk, type=DISK, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x06, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-disk0, customProperties={}}'}, 'vmType': 'kvm', 'spiceSslCipherSuite': 'DEFAULT', 'memSize': 1024, 'vmName': 'c8again32', 'nice': '0', 'username': 'Unknown', 'vmId': 'd54660a2-45ed-41ae-ab99-a6f93ebbdbb1', 'displayIp': '0', 'keyboardLayout': 'en-us', 'displayPort': '-1', 'smartcardEnable': 'false', 'guestIPs': '', 'nicModel': 'rtl8139,pv', 'smpCoresPerSocket': '1', 'kvmEnable': 'true', 'pitReinjection': 'false', 'devices': [{'device': 'unix', 'alias': 'channel0', 'type': 'channel', 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '1'}}, {'device': 'unix', 'alias': 'channel1', 'type': 'channel', 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '2'}}, {'device': 'spicevmc', 'alias': 'channel2', 'type': 'channel', 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '3'}}, {'device': 'scsi', 'alias': 'scsi0', 'model': 'virtio-scsi', 'type': 'controller', 'address': {'slot': '0x04', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}}, {'device': 'usb', 'alias': 'usb0', 'type': 'controller', 'address': {'slot': '0x01', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x2'}}, {'device': 'ide', 'alias': 'ide0', 'type': 'controller', 'address': {'slot': '0x01', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x1'}}, {'device': 'virtio-serial', 'alias': 'virtio-serial0', 'type': 'controller', 'address': {'slot': '0x05', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}}, {'specParams': {'vram': '65536'}, 'alias': 'video0', 'deviceId': 'c4763c67-c69b-4b7c-b0cd-ac5beca82669', 'address': {'slot': '0x02', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'qxl', 'type': 'video'}, {'nicModel': 'pv', 'macAddr': '00:1a:4a:8f:04:f7', 'linkActive': True, 'network': 'ovirtmgmt', 'specParams': {}, 'custom': {}, 'filter': 'vdsm-no-mac-spoofing', 'alias': 'net0', 'deviceId': '4478616d-d9d4-4284-8e6b-2cce0d330504', 'address': {'slot': '0x03', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'bridge', 'type': 'interface', 'name': 'vnet0'}, {'index': '2', 'iface': 'ide', 'name': 'hdc', 'alias': 'ide0-1-0', 'shared': 'false', 'specParams': {'path': ''}, 'readonly': 'True', 'deviceId': 'a3d3a41d-7f47-4a64-b3fe-10f33312f929', 'address': {'bus': '1', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'device': 'cdrom', 'path': '', 'type': 'disk'}, {'index': 0, 'iface': 'virtio', 'volumeInfo': {'volPort': '0', 'volType': 'network', 'volfileServer': 'f18ovn01.mydomain', 'volTransport': 'tcp', 'protocol': 'gluster', 'path': 'gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49'}, 'format': 'raw', 'shared': 'false', 'optional': 'false', 'volumeID': 'a123801a-0a4d-4a47-a426-99d8480d2e49', 'propagateErrors': 'off', 'imageID': '15f9ca1c-c435-4892-9eb7-0c84583b2a7d', 'specParams': {}, 'readonly': 'false', 'domainID': 'd0b96d4a-62aa-4e9f-b50e-f7a0cb5be291', 'deviceId': '15f9ca1c-c435-4892-9eb7-0c84583b2a7d', 'truesize': '4337233920', 'poolID': 'eb679feb-4da2-4fd0-a185-abbe459ffa70', 'device': 'disk', 'reqsize': '0', 'path': '/rhev/data-center/eb679feb-4da2-4fd0-a185-abbe459ffa70/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49', 'apparentsize': '10737418240', 'type': 'disk', 'volumeChain': [{'path': '/rhev/data-center/eb679feb-4da2-4fd0-a185-abbe459ffa70/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49', 'domainID': 'd0b96d4a-62aa-4e9f-b50e-f7a0cb5be291', 'vmVolInfo': {'volPort': '0', 'volType': 'network', 'volfileServer': 'f18ovn01.mydomain', 'volTransport': 'tcp', 'protocol': 'gluster', 'path': 'gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49'}, 'volumeID': 'a123801a-0a4d-4a47-a426-99d8480d2e49', 'imageID': '15f9ca1c-c435-4892-9eb7-0c84583b2a7d'}]}, {'index': '0', 'iface': 'pci', 'name': 'vda', 'alias': 'virtio-disk0', 'readonly': 'False', 'address': {'slot': '0x06', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'disk', 'path': 'gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49', 'type': 'disk'}, {'target': 1048576, 'specParams': {'model': 'virtio'}, 'alias': 'balloon0', 'deviceId': '216acfca-9672-4390-91b0-feb7aa201484', 'address': {'slot': '0x07', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'memballoon', 'type': 'balloon'}], 'status': 'Migration Destination', 'clientIp': '', 'display': 'qxl'}} Thread-3396::DEBUG::2013-10-08 01:20:54,554::BindingXMLRPC::979::vds::(wrapper) client [10.4.4.60]::call vmGetStats with ('d54660a2-45ed-41ae-ab99-a6f93ebbdbb1',) {} Thread-3396::DEBUG::2013-10-08 01:20:54,555::BindingXMLRPC::986::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Migration Destination', 'hash': '0', 'acpiEnable': 'true', 'pid': '0', 'displayIp': '0', 'displayPort': '-1', 'displaySecurePort': '-1', 'timeOffset': '-114', 'pauseCode': 'NOERR', 'kvmEnable': 'true', 'network': {}, 'vmId': 'd54660a2-45ed-41ae-ab99-a6f93ebbdbb1', 'displayType': 'qxl', 'disks': {}, 'monitorResponse': '0', 'elapsedTime': '178', 'vmType': 'kvm', 'clientIp': ''}]} Thread-3397::DEBUG::2013-10-08 01:20:54,885::BindingXMLRPC::979::vds::(wrapper) client [10.4.4.59]::call vmDestroy with ('d54660a2-45ed-41ae-ab99-a6f93ebbdbb1',) {} Thread-3397::INFO::2013-10-08 01:20:54,886::API::317::vds::(destroy) vmContainerLock acquired by vm d54660a2-45ed-41ae-ab99-a6f93ebbdbb1 Thread-3397::DEBUG::2013-10-08 01:20:54,886::vm::4258::vm.Vm::(destroy) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::destroy Called Thread-3397::INFO::2013-10-08 01:20:54,887::vm::4204::vm.Vm::(releaseVm) vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Release VM resources Thread-3397::WARNING::2013-10-08 01:20:54,887::clientIF::337::vds::(teardownVolumePath) Drive is not a vdsm image: VOLWM_CHUNK_MB:1024 VOLWM_CHUNK_REPLICATE_MULT:2 VOLWM_FREE_PCT:50 _blockDev:False _checkIoTuneCategories:<bound method Drive._checkIoTuneCategories of <vm.Drive object at 0x7fae4415bc90>> _customize:<bound method Drive._customize of <vm.Drive object at 0x7fae4415bc90>> _deviceXML:<disk device="cdrom" type="file"> <driver name="qemu" type="raw"/> <source startupPolicy="optional"/> <target bus="ide" dev="hdc"/> <readonly/> <serial/> <alias name="ide0-1-0"/> <address bus="1" controller="0" target="0" type="drive" unit="0"/> </disk> _makeName:<bound method Drive._makeName of <vm.Drive object at 0x7fae4415bc90>> _validateIoTuneParams:<bound method Drive._validateIoTuneParams of <vm.Drive object at 0x7fae4415bc90>> address:{'bus': '1', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'} alias:ide0-1-0 apparentsize:0 blockDev:False cache:none conf:{'guestFQDN': '', 'acpiEnable': 'true', 'emulatedMachine': 'pc-1.0', 'afterMigrationStatus': 'Up', 'spiceSecureChannels': 'smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard', 'pid': '0', 'memGuaranteedSize': 1024, 'transparentHugePages': 'true', 'displaySecurePort': '-1', 'timeOffset': '-114', 'cpuType': 'Opteron_G2', 'smp': '1', 'pauseCode': 'NOERR', 'migrationDest': 'libvirt', 'custom': {'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19edevice_9c0a4779-62bc-414b-abb5-a2fab36104bedevice_1459cc81-2f88-45dd-ae3c-0c030a084d2e': 'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, deviceId=1459cc81-2f88-45dd-ae3c-0c030a084d2e, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}}', 'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19edevice_9c0a4779-62bc-414b-abb5-a2fab36104bedevice_1459cc81-2f88-45dd-ae3c-0c030a084d2edevice_5bea8f60-7de6-4c93-99ff-af67635e2af6device_8d086706-28b6-40fa-a86d-3733c9ebe5c1': 'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, deviceId=8d086706-28b6-40fa-a86d-3733c9ebe5c1, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}}', 'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19e': 'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, deviceId=db19ff20-8f6d-488c-bad8-19e33b01f19e, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}}', 'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19edevice_9c0a4779-62bc-414b-abb5-a2fab36104be': 'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, deviceId=9c0a4779-62bc-414b-abb5-a2fab36104be, device=virtio-serial, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x05, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-serial0, customProperties={}}', 'device_a5fa5a6d-f191-42d2-be55-d6227499c658': 'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, deviceId=a5fa5a6d-f191-42d2-be55-d6227499c658, device=scsi, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x04, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=scsi0, customProperties={}}', 'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9device_db19ff20-8f6d-488c-bad8-19e33b01f19edevice_9c0a4779-62bc-414b-abb5-a2fab36104bedevice_1459cc81-2f88-45dd-ae3c-0c030a084d2edevice_5bea8f60-7de6-4c93-99ff-af67635e2af6': 'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, deviceId=5bea8f60-7de6-4c93-99ff-af67635e2af6, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}}', 'device_a5fa5a6d-f191-42d2-be55-d6227499c658device_bd099ee2-faee-41c3-9e01-0a01f458abe9': 'VmDevice {vmId=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, deviceId=bd099ee2-faee-41c3-9e01-0a01f458abe9, device=disk, type=DISK, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x06, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-disk0, customProperties={}}'}, 'vmType': 'kvm', 'spiceSslCipherSuite': 'DEFAULT', 'memSize': 1024, 'vmName': 'c8again32', 'nice': '0', 'username': 'Unknown', 'vmId': 'd54660a2-45ed-41ae-ab99-a6f93ebbdbb1', 'displayIp': '0', 'keyboardLayout': 'en-us', 'displayPort': '-1', 'smartcardEnable': 'false', 'guestIPs': '', 'nicModel': 'rtl8139,pv', 'smpCoresPerSocket': '1', 'kvmEnable': 'true', 'pitReinjection': 'false', 'devices': [{'device': 'unix', 'alias': 'channel0', 'type': 'channel', 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '1'}}, {'device': 'unix', 'alias': 'channel1', 'type': 'channel', 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '2'}}, {'device': 'spicevmc', 'alias': 'channel2', 'type': 'channel', 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '3'}}, {'device': 'scsi', 'alias': 'scsi0', 'model': 'virtio-scsi', 'type': 'controller', 'address': {'slot': '0x04', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}}, {'device': 'usb', 'alias': 'usb0', 'type': 'controller', 'address': {'slot': '0x01', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x2'}}, {'device': 'ide', 'alias': 'ide0', 'type': 'controller', 'address': {'slot': '0x01', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x1'}}, {'device': 'virtio-serial', 'alias': 'virtio-serial0', 'type': 'controller', 'address': {'slot': '0x05', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}}, {'specParams': {'vram': '65536'}, 'alias': 'video0', 'deviceId': 'c4763c67-c69b-4b7c-b0cd-ac5beca82669', 'address': {'slot': '0x02', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'qxl', 'type': 'video'}, {'nicModel': 'pv', 'macAddr': '00:1a:4a:8f:04:f7', 'linkActive': True, 'network': 'ovirtmgmt', 'specParams': {}, 'custom': {}, 'filter': 'vdsm-no-mac-spoofing', 'alias': 'net0', 'deviceId': '4478616d-d9d4-4284-8e6b-2cce0d330504', 'address': {'slot': '0x03', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'bridge', 'type': 'interface', 'name': 'vnet0'}, {'index': '2', 'iface': 'ide', 'name': 'hdc', 'alias': 'ide0-1-0', 'shared': 'false', 'specParams': {'path': ''}, 'readonly': 'True', 'deviceId': 'a3d3a41d-7f47-4a64-b3fe-10f33312f929', 'address': {'bus': '1', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'device': 'cdrom', 'path': '', 'type': 'disk'}, {'index': 0, 'iface': 'virtio', 'volumeInfo': {'volPort': '0', 'volType': 'network', 'volfileServer': 'f18ovn01.mydomain', 'volTransport': 'tcp', 'protocol': 'gluster', 'path': 'gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49'}, 'format': 'raw', 'shared': 'false', 'optional': 'false', 'volumeID': 'a123801a-0a4d-4a47-a426-99d8480d2e49', 'propagateErrors': 'off', 'imageID': '15f9ca1c-c435-4892-9eb7-0c84583b2a7d', 'specParams': {}, 'readonly': 'false', 'domainID': 'd0b96d4a-62aa-4e9f-b50e-f7a0cb5be291', 'deviceId': '15f9ca1c-c435-4892-9eb7-0c84583b2a7d', 'truesize': '4337233920', 'poolID': 'eb679feb-4da2-4fd0-a185-abbe459ffa70', 'device': 'disk', 'reqsize': '0', 'path': '/rhev/data-center/eb679feb-4da2-4fd0-a185-abbe459ffa70/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49', 'apparentsize': '10737418240', 'type': 'disk', 'volumeChain': [{'path': '/rhev/data-center/eb679feb-4da2-4fd0-a185-abbe459ffa70/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49', 'domainID': 'd0b96d4a-62aa-4e9f-b50e-f7a0cb5be291', 'vmVolInfo': {'volPort': '0', 'volType': 'network', 'volfileServer': 'f18ovn01.mydomain', 'volTransport': 'tcp', 'protocol': 'gluster', 'path': 'gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49'}, 'volumeID': 'a123801a-0a4d-4a47-a426-99d8480d2e49', 'imageID': '15f9ca1c-c435-4892-9eb7-0c84583b2a7d'}]}, {'index': '0', 'iface': 'pci', 'name': 'vda', 'alias': 'virtio-disk0', 'readonly': 'False', 'address': {'slot': '0x06', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'disk', 'path': 'gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49', 'type': 'disk'}, {'target': 1048576, 'specParams': {'model': 'virtio'}, 'alias': 'balloon0', 'deviceId': '216acfca-9672-4390-91b0-feb7aa201484', 'address': {'slot': '0x07', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'memballoon', 'type': 'balloon'}], 'status': 'Migration Destination', 'clientIp': '', 'display': 'qxl'} createXmlElem:<bound method Drive.createXmlElem of <vm.Drive object at 0x7fae4415bc90>> device:cdrom deviceId:a3d3a41d-7f47-4a64-b3fe-10f33312f929 getNextVolumeSize:<bound method Drive.getNextVolumeSize of <vm.Drive object at 0x7fae4415bc90>> getXML:<bound method Drive.getXML of <vm.Drive object at 0x7fae4415bc90>> iface:ide index:2 isDiskReplicationInProgress:<bound method Drive.isDiskReplicationInProgress of <vm.Drive object at 0x7fae4415bc90>> isVdsmImage:<bound method Drive.isVdsmImage of <vm.Drive object at 0x7fae4415bc90>> log:<logUtils.SimpleLogAdapter object at 0x7fae2058f1d0> name:hdc networkDev:False path: readonly:True reqsize:0 serial: shared:false specParams:{'path': ''} truesize:0 type:disk volExtensionChunk:1024 watermarkLimit:536870912 Traceback (most recent call last): File "/usr/share/vdsm/clientIF.py", line 331, in teardownVolumePath res = self.irs.teardownImage(drive['domainID'], File "/usr/share/vdsm/vm.py", line 1344, in __getitem__ raise KeyError(key) KeyError: 'domainID' Thread-3393::ERROR::2013-10-08 01:20:54,888::libvirtconnection::94::libvirtconnection::(wrapper) connection to libvirt broken. ecode: 1 edom: 7 Thread-3393::ERROR::2013-10-08 01:20:54,892::libvirtconnection::96::libvirtconnection::(wrapper) taking calling process down. firewall on destination seems ok: Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- 192.168.3.3 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:54321 ...