--_000_687766E0007010429FBFF98CA70824888A0C30mcexch01mcongroup_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, we tried this (Roys Mail below) , and yes, shutdown always works. But now this problem comes up with regular machines now as well. The environment is setup like this Engine: Ovirt 3.5.1.1-1.el6 - on CentOS6 The Storage Backend is gluster 3.6.2-1.el7 on CentOS 7 Compute hosts: Libvirt Version: libvirt-1.2.9.1-2.fc20, kvm 2.1.2 - 7.fc20,= vdsm vdsm-4.16.10-8.gitc937927.fc20 All compute Servers are 100% identical. The storage cluster was tested manually and works just fine. The network interfaces are not fully utilized, more like 15%. Log output ist as below. The thing in the log output I do not understand is= this "2015-02-26T14:50:26.650595Z qemu-system-x86_64: load of migration failed: = Input/output error"
From the qemu log.
Also if I shut down machines, put their host into maintenance and start the= m somewhere else, everything works just fine. Can someone help with this ? Any idea where to look ? Regards Soeren
From VDSM Log
I just tried to migrate a machine, this here happens on the source <-- snip --> vdsm.log:Thread-49548::DEBUG::2015-02-26 15:42:26,692::__init__::469::jsonr= pc.JsonRpcServer::(_serveRequest) Calling 'VM.migrate' in bridge with {u'pa= rams': {u'tunneled': u'false', u'dstqemu': u'172.19.2.31', u'src': u'comput= e04', u'dst': u'compute01:54321', u'vmId': u'b75823d1-00f0-457e-a692-8b95f7= 3907db', u'abortOnError': u'true', u'method': u'online'}, u'vmID': u'b75823= d1-00f0-457e-a692-8b95f73907db'} vdsm.log:Thread-49548::DEBUG::2015-02-26 15:42:26,694::API::510::vds::(migr= ate) {u'tunneled': u'false', u'dstqemu': u'IPADDR', u'src': u'compute04', u= 'dst': u'compute01:54321', u'vmId': u'b75823d1-00f0-457e-a692-8b95f73907db'= , u'abortOnError': u'true', u'method': u'online'} vdsm.log:Thread-49549::DEBUG::2015-02-26 15:42:26,699::migration::103::vm.V= m::(_setupVdsConnection) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Des= tination server is: compute01:54321 vdsm.log:Thread-49549::DEBUG::2015-02-26 15:42:26,702::migration::105::vm.V= m::(_setupVdsConnection) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Ini= tiating connection with destination vdsm.log:Thread-49549::DEBUG::2015-02-26 15:42:26,733::migration::155::vm.V= m::(_prepareGuest) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration= started vdsm.log:Thread-49549::DEBUG::2015-02-26 15:42:26,755::migration::238::vm.V= m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::migration semaphore= acquired after 0 seconds vdsm.log:Thread-49549::DEBUG::2015-02-26 15:42:27,211::migration::298::vm.V= m::(_startUnderlyingMigration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db= `::starting migration to qemu+tls://compute01/system with miguri tcp://IPAD= DR vdsm.log:Thread-49550::DEBUG::2015-02-26 15:42:27,213::migration::361::vm.V= m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::migration downtime = thread started vdsm.log:Thread-49551::DEBUG::2015-02-26 15:42:27,216::migration::410::vm.V= m::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::start= ing migration monitor thread vdsm.log:Thread-49550::DEBUG::2015-02-26 15:43:42,218::migration::370::vm.V= m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration d= owntime to 50 vdsm.log:Thread-49550::DEBUG::2015-02-26 15:44:57,222::migration::370::vm.V= m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration d= owntime to 100 vdsm.log:Thread-49550::DEBUG::2015-02-26 15:46:12,227::migration::370::vm.V= m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration d= owntime to 150 vdsm.log:Thread-49551::WARNING::2015-02-26 15:47:07,279::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (1791MiB) > lowmark (203MiB). Refer to RHBZ#9192= 01. vdsm.log:Thread-49551::WARNING::2015-02-26 15:47:17,281::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (1398MiB) > lowmark (203MiB). Refer to RHBZ#9192= 01. vdsm.log:Thread-49550::DEBUG::2015-02-26 15:47:27,233::migration::370::vm.V= m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration d= owntime to 200 vdsm.log:Thread-49551::WARNING::2015-02-26 15:47:27,283::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (1066MiB) > lowmark (203MiB). Refer to RHBZ#9192= 01. vdsm.log:Thread-49551::WARNING::2015-02-26 15:47:37,285::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (701MiB) > lowmark (203MiB). Refer to RHBZ#91920= 1. vdsm.log:Thread-49551::WARNING::2015-02-26 15:47:47,287::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (361MiB) > lowmark (203MiB). Refer to RHBZ#91920= 1. vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:07,291::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (683MiB) > lowmark (13MiB). Refer to RHBZ#919201= . vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:17,292::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (350MiB) > lowmark (13MiB). Refer to RHBZ#919201= . vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:27,294::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (18MiB) > lowmark (13MiB). Refer to RHBZ#919201. vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:37,296::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (646MiB) > lowmark (13MiB). Refer to RHBZ#919201= . vdsm.log:Thread-49550::DEBUG::2015-02-26 15:48:42,238::migration::370::vm.V= m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration d= owntime to 250 vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:47,334::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (317MiB) > lowmark (13MiB). Refer to RHBZ#919201= . vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:57,342::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (1018MiB) > lowmark (13MiB). Refer to RHBZ#91920= 1. vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:07,344::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (679MiB) > lowmark (13MiB). Refer to RHBZ#919201= . vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:17,346::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (357MiB) > lowmark (13MiB). Refer to RHBZ#919201= . vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:27,348::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (31MiB) > lowmark (13MiB). Refer to RHBZ#919201. vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:37,349::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (854MiB) > lowmark (13MiB). Refer to RHBZ#919201= . vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:47,351::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (525MiB) > lowmark (13MiB). Refer to RHBZ#919201= . vdsm.log:Thread-49550::DEBUG::2015-02-26 15:49:57,242::migration::370::vm.V= m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration d= owntime to 300 vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:57,353::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (183MiB) > lowmark (13MiB). Refer to RHBZ#919201= . vdsm.log:Thread-49551::WARNING::2015-02-26 15:50:07,355::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (785MiB) > lowmark (13MiB). Refer to RHBZ#919201= . vdsm.log:Thread-49551::WARNING::2015-02-26 15:50:17,357::migration::458::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration stalling: remaining (457MiB) > lowmark (13MiB). Refer to RHBZ#919201= . vdsm.log:Thread-49551::WARNING::2015-02-26 15:50:27,359::migration::445::vm= .Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig= ration is stuck: Hasn't progressed in 150.069458961 seconds. Aborting. vdsm.log:Thread-49551::DEBUG::2015-02-26 15:50:27,362::migration::470::vm.V= m::(stop) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::stopping migration= monitor thread vdsm.log:Thread-49549::DEBUG::2015-02-26 15:50:27,852::migration::376::vm.V= m::(cancel) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::canceling migrat= ion downtime thread vdsm.log:Thread-49549::DEBUG::2015-02-26 15:50:27,852::migration::470::vm.V= m::(stop) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::stopping migration= monitor thread vdsm.log:Thread-49550::DEBUG::2015-02-26 15:50:27,853::migration::373::vm.V= m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::migration downtime = thread exiting vdsm.log:Thread-49549::ERROR::2015-02-26 15:50:27,855::migration::161::vm.V= m::(_recover) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::operation abor= ted: migration job: canceled by client vdsm.log:Thread-49549::ERROR::2015-02-26 15:50:28,049::migration::260::vm.V= m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Failed to migrate vdsm.log:Thread-49690::DEBUG::2015-02-26 15:50:33,284::__init__::469::jsonr= pc.JsonRpcServer::(_serveRequest) Calling 'VM.getMigrationStatus' in bridge= with {u'vmID': u'b75823d1-00f0-457e-a692-8b95f73907db'} <--snip --> At the same time on the destination server <-- snip --> vdsm.log:Thread-350501::DEBUG::2015-02-26 15:42:25,923::BindingXMLRPC::1133= ::vds::(wrapper) client [172.19.2.34]::call vmGetStats with ('b75823d1-00f0= -457e-a692-8b95f73907db',) {} vdsm.log:Thread-350502::DEBUG::2015-02-26 15:42:26,018::BindingXMLRPC::1133= ::vds::(wrapper) client [172.19.2.34]::call vmMigrationCreate with ALL THE = MACHINE START PARAMETERS here vdsm.log:Thread-350502::INFO::2015-02-26 15:42:26,019::clientIF::394::vds::= (createVm) vmContainerLock acquired by vm b75823d1-00f0-457e-a692-8b95f7390= 7db vdsm.log:Thread-350502::DEBUG::2015-02-26 15:42:26,033::clientIF::407::vds:= :(createVm) Total desktops after creation of b75823d1-00f0-457e-a692-8b95f7= 3907db is 3 vdsm.log:Thread-350503::DEBUG::2015-02-26 15:42:26,033::vm::2264::vm.Vm::(_= startUnderlyingVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Start vdsm.log:Thread-350502::DEBUG::2015-02-26 15:42:26,036::vm::5658::vm.Vm::(w= aitForMigrationDestinationPrepare) vmId=3D`b75823d1-00f0-457e-a692-8b95f739= 07db`::migration destination: waiting for VM creation vdsm.log:Thread-350503::DEBUG::2015-02-26 15:42:26,036::vm::2268::vm.Vm::(_= startUnderlyingVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::_ongoingC= reations acquired vdsm.log:Thread-350502::DEBUG::2015-02-26 15:42:26,038::vm::5663::vm.Vm::(w= aitForMigrationDestinationPrepare) vmId=3D`b75823d1-00f0-457e-a692-8b95f739= 07db`::migration destination: waiting 48s for path preparation vdsm.log:Thread-350503::INFO::2015-02-26 15:42:26,038::vm::3261::vm.Vm::(_r= un) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::VM wrapper has started vdsm.log:Thread-350503::WARNING::2015-02-26 15:42:26,041::vm::2056::vm.Vm::= (buildConfDevices) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Unknown t= ype found, device: '{'device': 'unix', 'alias': 'channel0', 'type': 'channe= l', 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'po= rt': '1'}}' found vdsm.log:Thread-350503::WARNING::2015-02-26 15:42:26,041::vm::2056::vm.Vm::= (buildConfDevices) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Unknown t= ype found, device: '{'device': 'unix', 'alias': 'channel1', 'type': 'channe= l', 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'po= rt': '2'}}' found vdsm.log:Thread-350503::DEBUG::2015-02-26 15:42:26,349::vm::1058::vm.Vm::(_= _init__) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Ignoring param (tar= get, 10485760) in BalloonDevice vdsm.log:Thread-350503::DEBUG::2015-02-26 15:42:26,350::vm::2294::vm.Vm::(_= startUnderlyingVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::_ongoingC= reations released vdsm.log:Thread-350502::ERROR::2015-02-26 15:42:26,351::vm::5638::vm.Vm::(_= updateDevicesDomxmlCache) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Al= ias not found for device type graphics during migration at destination host vdsm.log:Thread-350503::DEBUG::2015-02-26 15:42:26,353::vm::4128::vm.Vm::(_= waitForUnderlyingMigration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::= Waiting 21600 seconds for end of migration vdsm.log:Thread-350502::DEBUG::2015-02-26 15:42:26,377::BindingXMLRPC::1140= ::vds::(wrapper) return vmMigrationCreate with {ALL THE MACHINE PARAMETERS = HERE vdsm.log:libvirtEventLoop::DEBUG::2015-02-26 15:42:27,195::vm::5571::vm.Vm:= :(_onLibvirtLifecycleEvent) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::= event Started detail 1 opaque None vdsm.log:libvirtEventLoop::DEBUG::2015-02-26 15:50:27,037::vm::5571::vm.Vm:= :(_onLibvirtLifecycleEvent) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::= event Stopped detail 5 opaque None vdsm.log:libvirtEventLoop::INFO::2015-02-26 15:50:27,038::vm::2366::vm.Vm::= (_onQemuDeath) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::underlying pr= ocess disconnected vdsm.log:libvirtEventLoop::INFO::2015-02-26 15:50:27,038::vm::4952::vm.Vm::= (releaseVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Release VM resou= rces vdsm.log:Thread-350503::DEBUG::2015-02-26 15:50:27,044::libvirtconnection::= 143::root::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 mess= age: Domain not found: no domain with matching uuid 'b75823d1-00f0-457e-a69= 2-8b95f73907db' vdsm.log:Thread-350503::ERROR::2015-02-26 15:50:27,047::vm::2325::vm.Vm::(_= startUnderlyingVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Failed to= start a migration destination vm vdsm.log:MigrationError: Domain not found: no domain with matching uuid 'b7= 5823d1-00f0-457e-a692-8b95f73907db' vdsm.log:Thread-350503::DEBUG::2015-02-26 15:50:27,058::vm::2786::vm.Vm::(s= etDownStatus) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Changed state = to Down: VM failed to migrate (code=3D8) vdsm.log:Thread-351517::DEBUG::2015-02-26 15:50:27,089::BindingXMLRPC::1133= ::vds::(wrapper) client [172.19.2.34]::call vmDestroy with ('b75823d1-00f0-= 457e-a692-8b95f73907db',) {} vdsm.log:Thread-351517::INFO::2015-02-26 15:50:27,092::API::332::vds::(dest= roy) vmContainerLock acquired by vm b75823d1-00f0-457e-a692-8b95f73907db vdsm.log:Thread-351517::DEBUG::2015-02-26 15:50:27,154::vm::5026::vm.Vm::(d= estroy) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::destroy Called vdsm.log:libvirtEventLoop::WARNING::2015-02-26 15:50:27,162::utils::129::ro= ot::(rmFile) File: /var/lib/libvirt/qemu/channels/b75823d1-00f0-457e-a692-8= b95f73907db.com.redhat.rhevm.vdsm already removed vdsm.log:libvirtEventLoop::WARNING::2015-02-26 15:50:27,163::utils::129::ro= ot::(rmFile) File: /var/lib/libvirt/qemu/channels/b75823d1-00f0-457e-a692-8= b95f73907db.org.qemu.guest_agent.0 already removed vdsm.log:libvirtEventLoop::INFO::2015-02-26 15:50:27,164::logUtils::44::dis= patcher::(wrapper) Run and protect: inappropriateDevices(thiefId=3D'b75823d= 1-00f0-457e-a692-8b95f73907db') vdsm.log:Thread-351517::DEBUG::2015-02-26 15:50:27,170::vm::5020::vm.Vm::(d= eleteVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Total desktops afte= r destroy of b75823d1-00f0-457e-a692-8b95f73907db is 2 vdsm.log:libvirtEventLoop::WARNING::2015-02-26 15:50:27,170::vm::1953::vm.V= m::(_set_lastStatus) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::trying = to set state to Down when already Down <-- snip --> And I can see the error "Domain not found: no domain with matching uuid 'b7= 5823d1-00f0-457e-a692-8b95f73907db'" in the middle, but I am not entirely s= ure what this means. In the qemu log I can see this <-- snip -->
From QEMU LOG
2015-02-26 14:42:26.859+0000: starting up Domain id=3D8 is tainted: hook-script 2015-02-26T14:50:26.650595Z qemu-system-x86_64: load of migration failed: I= nput/output error 2015-02-26 14:50:26.684+0000: shutting down <-- snip --> the time here seems to be logged in GMT and in the snippets above it is CET= - might this be the problem ??? Here is the qemu commandline (that was removed above in cleaned <-- snip --> /usr/bin/qemu-kvm -name SERVERNAME -S -machine pc-1.0,accel=3Dkvm,usb=3Doff -cpu SandyBridge -m 10240 -realtime mlock=3Doff -smp 4,maxcpus=3D64,sockets=3D16,cores=3D4,threads=3D1 -uuid b75823d1-00f0-457e-a692-8b95f73907db -smbios type=3D1,manufacturer=3DoVirt,product=3DoVirt Node,version=3D20-3,s= erial=3D4C4C4544-0038-3210-8048-B7C04F303232,uuid=3Db75823d1-00f0-457e-a692= -8b95f73907db -no-user-config -nodefaults -chardev socket,id=3Dcharmonitor,path=3D/var/lib/libvirt/qemu/al-exchange-0= 1.monitor,server,nowait -mon chardev=3Dcharmonitor,id=3Dmonitor,mode=3Dcontrol -rtc base=3D2015-02-26T15:42:26,driftfix=3Dslew -global kvm-pit.lost_tick_policy=3Ddiscard -no-hpet -no-shutdown -boot strict=3Don -device piix3-usb-uhci,id=3Dusb,bus=3Dpci.0,addr=3D0x1.0x2 -device virtio-scsi-pci,id=3Dscsi0,bus=3Dpci.0,addr=3D0x4 -device virtio-serial-pci,id=3Dvirtio-serial0,max_ports=3D16,bus=3Dpci.0,ad= dr=3D0x5 -drive file=3D/rhev/data-center/mnt/XXXX:_data_export_iso/85821683-bc7c-429= b-bff2-ce25f56fc16f/images/11111111-1111-1111-1111-111111111111/XXXX.iso,if= =3Dnone,id=3Ddrive-ide0-1-0,readonly=3Don,format=3Draw,serial=3D -device id= e-cd,bus=3Dide.1,unit=3D0,drive=3Ddrive-ide0-1-0,id=3Dide0-1-0,bootindex=3D= 2 -drive file=3D/rhev/data-center/0f954891-b1cd-4f09-99ae-75d404d95f9d/276e9b= a7-e19a-49c5-8ad7-26711934d5e4/images/c37bfa94-718c-4125-9202-bc299535eca5/= 299bd6d1-f4a0-4296-9944-176b2252a886,if=3Dnone,id=3Ddrive-virtio-disk0,form= at=3Draw,serial=3Dc37bfa94-718c-4125-9202-bc299535eca5,cache=3Dnone,werror= =3Dstop,rerror=3Dstop,aio=3Dthreads -device virtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x6,drive=3Ddrive-virt= io-disk0,id=3Dvirtio-disk0,bootindex=3D1 -drive file=3D/rhev/data-center/0f954891-b1cd-4f09-99ae-75d404d95f9d/276e9b= a7-e19a-49c5-8ad7-26711934d5e4/images/75134ccc-b74e-4955-90a5-95d4ceff403b/= 1e5b5b86-c31d-476c-acb2-a5bd6a65490b,if=3Dnone,id=3Ddrive-virtio-disk1,form= at=3Draw,serial=3D75134ccc-b74e-4955-90a5-95d4ceff403b,cache=3Dnone,werror= =3Dstop,rerror=3Dstop,aio=3Dthreads -device virtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x8,drive=3Ddrive-virt= io-disk1,id=3Dvirtio-disk1 -netdev tap,fd=3D30,id=3Dhostnet0,vhost=3Don,vhostfd=3D31 -device virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D00:1a:4a:c7:72:0a,= bus=3Dpci.0,addr=3D0x3 -chardev socket,id=3Dcharchannel0,path=3D/var/lib/libvirt/qemu/channels/b75= 823d1-00f0-457e-a692-8b95f73907db.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=3Dvirtio-serial0.0,nr=3D1,chardev=3Dcharchannel0= ,id=3Dchannel0,name=3Dcom.redhat.rhevm.vdsm -chardev socket,id=3Dcharchannel1,path=3D/var/lib/libvirt/qemu/channels/b75= 823d1-00f0-457e-a692-8b95f73907db.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=3Dvirtio-serial0.0,nr=3D2,chardev=3Dcharchannel1= ,id=3Dchannel1,name=3Dorg.qemu.guest_agent.0 -device usb-tablet,id=3Dinput0 -vnc IP-ADDRESS:2,password -k de -device cirrus-vga,id=3Dvideo0,bus=3Dpci.0,addr=3D0x2 -incoming tcp:[::]:49152 -device virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x7 -msg timestamp=3Don <-- snip --> Here is the content of the .meta file in the directories of the 2 images ab= ove <--snip--> DOMAIN=3D276e9ba7-e19a-49c5-8ad7-26711934d5e4 VOLTYPE=3DLEAF CTIME=3D1424445378 FORMAT=3DRAW IMAGE=3Dc37bfa94-718c-4125-9202-bc299535eca5 DISKTYPE=3D2 PUUID=3D00000000-0000-0000-0000-000000000000 LEGALITY=3DLEGAL MTIME=3D0 POOL_UUID=3D DESCRIPTION=3D{"DiskAlias":"al-exchange-01_Disk1","DiskDescription":"System= "} TYPE=3DSPARSE SIZE=3D146800640 EOF <--snip--> <--snip--> DOMAIN=3D276e9ba7-e19a-49c5-8ad7-26711934d5e4 CTIME=3D1424785898 FORMAT=3DRAW DISKTYPE=3D2 LEGALITY=3DLEGAL SIZE=3D419430400 VOLTYPE=3DLEAF DESCRIPTION=3D{"DiskAlias":"al-exchange-01_Disk2","DiskDescription":"Data"} IMAGE=3D75134ccc-b74e-4955-90a5-95d4ceff403b PUUID=3D00000000-0000-0000-0000-000000000000 MTIME=3D0 POOL_UUID=3D TYPE=3DSPARSE EOF <--snip--> From: Roy Golan [mailto:rgolan@redhat.com] Sent: Wednesday, February 18, 2015 12:12 PM To: Soeren Malchow; users@ovirt.org Subject: Re: [ovirt-users] Hosted Engine Migration fails On 02/16/2015 04:55 AM, Soeren Malchow wrote: Dear all, we ahve a setup with several hosts running fedora 10 with the virt-preview = packages installed (for snapshot live merge) and a hosted engine running ce= ntos 6.6- We are experiencing a problem with the Live Migration of the Hosted Engine,= in the case of setting the host for the Engine into maintenance as well as= a manual migration. I tried this on the "ovirtmgmt" network and when that failed I did some res= earch and tried to use another network interface (separate from ovirtmgmt o= n layer 2), this also fails. It looks as if the migration is still going through the ovirtmgmt interface= , at least judging from the network traffic, and I think the error that I f= ound (RHBZ#919201) is actually the right one. -- vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:25,743::migration::458::vm.= Vm::(monitor_migration) vmId=3D`XXX`::Migration stalling: remaining (35MiB)=
lowmark (15MiB). Refer to RHBZ#919201. vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:35,745::migration::458::vm.= Vm::(monitor_migration) vmId=3D`XXX`::Migration stalling: remaining (129MiB= ) > lowmark (15MiB). Refer to RHBZ#919201. vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:45,747::migration::458::vm.= Vm::(monitor_migration) vmId=3D`XXX`::Migration stalling: remaining (42MiB)= lowmark (15MiB). Refer to RHBZ#919201. vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:55,749::migration::458::vm.= Vm::(monitor_migration) vmId=3D`XXX`::Migration stalling: remaining (88MiB)= lowmark (15MiB). Refer to RHBZ#919201. -- The ovirtmgmt interface is 2 x 1Gbit (LACP connected to Dell Switches with = MLAG) and by far not fully utilized. Can anyone help me where to go form here ?
still need the whole log. probably the guest (engine) is doing lots of memo= ry i/o if you got a fair amount of running VMs and Hosts. that will stalls= the migration because the guest pages are getting dirty faster than qemu can copy. you have 2 options: 1. try several more times. 2. shutdown the engine vm, it should start on another host Regards Soeren _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users --_000_687766E0007010429FBFF98CA70824888A0C30mcexch01mcongroup_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
</p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">But now=
</p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:Wingdings; panose-1:5 0 0 0 0 0 0 0 0 0;} @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Consolas; panose-1:2 11 6 9 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif; color:black; mso-fareast-language:EN-US;} a:link, span.MsoHyperlink {mso-style-priority:99; color:#0563C1; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} pre {mso-style-priority:99; mso-style-link:"HTML Preformatted Char"; margin:0in; margin-bottom:.0001pt; font-size:10.0pt; font-family:"Courier New"; color:black;} span.EmailStyle17 {mso-style-type:personal; font-family:"Calibri",sans-serif; color:windowtext;} span.HTMLPreformattedChar {mso-style-name:"HTML Preformatted Char"; mso-style-priority:99; mso-style-link:"HTML Preformatted"; font-family:Consolas; color:black; mso-fareast-language:EN-US;} span.EmailStyle20 {mso-style-type:personal-reply; font-family:"Calibri",sans-serif; color:#1F497D;} span.gwt-inlinelabel {mso-style-name:gwt-inlinelabel;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;} @page WordSection1 {size:8.5in 11.0in; margin:70.85pt 70.85pt 56.7pt 70.85pt;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> </head> <body bgcolor=3D"white" lang=3D"DE" link=3D"#0563C1" vlink=3D"#954F72"> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Hi,<o:p></o:p></span><= /p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p> </o:p></spa= n></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">we trie= d this (Roys Mail below) , and yes, shutdown always works.<o:p></o:p></span= this problem comes up with regular machines now as well.<o:p></o:p></span>= </p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">The env= ironment is setup like this<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F4E79;mso-styl= e-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><span s= tyle=3D"color:windowtext">Engine: Ovirt </span></span><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US" style=3D= "color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fil= l-alpha:100.0%"><span style=3D"color:windowtext">3.5.1.1-1.el6 – on C= entOS6 <o:p></o:p></span></span></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">The Storage Backe= nd is gluster 3.6.2-1.el7 on CentOS 7<o:p></o:p></span></span></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">Compute hosts: Li= bvirt Version: libvirt-1.2.9.1-2.fc20, kvm 2.1.2 - 7.fc20, vdsm vdsm-4.16.10-8.gitc937927.fc20<o:p></o:p></span></spa= n></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">All compute Serve= rs are 100% identical.<o:p></o:p></span></span></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">The storage clust= er was tested manually and works just fine.<o:p></o:p></span></span></span>= </p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">The network inter= faces are not fully utilized, more like 15%.<o:p></o:p></span></span></span= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">Log output ist as= below. The thing in the log output I do not understand is this<o:p></o:p></span></span></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">“</span></s= pan></span><span lang=3D"EN-US" style=3D"color:#1F497D">2015-02-26T14:50:26= .650595Z qemu-system-x86_64: load of migration failed: Input/output error” <o= :p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">From th= e qemu log.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">Also if= I shut down machines, put their host into maintenance and start them somew= here else, everything works just fine.</span><span class=3D"gwt-inlinelabel= "><span lang=3D"EN-US" style=3D"color:#1F4E79;mso-style-textfill-fill-color= :#1F4E79;mso-style-textfill-fill-alpha:100.0%"><o:p></o:p></span></span></p=
</span></span></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
<o:p> </o:p></span></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><b><span lang=3D"EN-US" style=3D"font-size:12.0pt;co= lor:#1F497D">From VDSM Log<o:p></o:p></span></b></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">I just =
</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350502::DEBUG::2015-02-26 15:42:26,033::clientIF::407::vds::(creat= eVm) Total desktops after creation of b75823d1-00f0-457e-a692-8b95f73907db = is 3<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350503::DEBUG::2015-02-26 15:42:26,033::vm::2264::vm.Vm::(_startUn= derlyingVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Start<o:p></o:p>= </span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350502::DEBUG::2015-02-26 15:42:26,036::vm::5658::vm.Vm::(waitForM= igrationDestinationPrepare) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::= migration destination: waiting for VM creation<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350503::DEBUG::2015-02-26 15:42:26,036::vm::2268::vm.Vm::(_startUn= derlyingVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::_ongoingCreation= s acquired<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350502::DEBUG::2015-02-26 15:42:26,038::vm::5663::vm.Vm::(waitForM= igrationDestinationPrepare) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::= migration destination: waiting 48s for path
<p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">Can someone help = with this ? Any idea where to look ?<o:p></o:p></span></span></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">Regards<o:p></o:p= style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex= tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">Soeren <o:p></o:p></span></span></span></p> <p class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span lang=3D"EN-US"= tried to migrate a machine, this here happens on the source <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:Wingdings;= color:#1F497D">ß</span><span lang=3D"EN-US" style=3D"color:#1F497D"> = snip </span><span lang=3D"EN-US" style=3D"font-family:Wingdings;color:#1F497D">&= agrave;</span><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p></o:p></spa= n></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49548::DEBUG::2015-02-26 15:42:26,692::__init__::469::jsonrpc.Json= RpcServer::(_serveRequest) Calling 'VM.migrate' in bridge with {u'params': = {u'tunneled': u'false', u'dstqemu': u'172.19.2.31', u'src': u'compute04', u'dst': u'compute01:54321', u'vmId': u'b75823d1-00f0= -457e-a692-8b95f73907db', u'abortOnError': u'true', u'method': u'online'}, = u'vmID': u'b75823d1-00f0-457e-a692-8b95f73907db'}<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49548::DEBUG::2015-02-26 15:42:26,694::API::510::vds::(migrate) {u= 'tunneled': u'false', u'dstqemu': u'IPADDR', u'src': u'compute04', u'dst': = u'compute01:54321', u'vmId': u'b75823d1-00f0-457e-a692-8b95f73907db', u'abortOnError': u'true', u'method': u'online'}<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49549::DEBUG::2015-02-26 15:42:26,699::migration::103::vm.Vm::(_se= tupVdsConnection) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Destinatio= n server is: compute01:54321<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49549::DEBUG::2015-02-26 15:42:26,702::migration::105::vm.Vm::(_se= tupVdsConnection) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Initiating= connection with destination<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49549::DEBUG::2015-02-26 15:42:26,733::migration::155::vm.Vm::(_pr= epareGuest) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration starte= d<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49549::DEBUG::2015-02-26 15:42:26,755::migration::238::vm.Vm::(run= ) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::migration semaphore acquir= ed after 0 seconds<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49549::DEBUG::2015-02-26 15:42:27,211::migration::298::vm.Vm::(_st= artUnderlyingMigration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::star= ting migration to qemu+tls://compute01/system with miguri tcp://IPADDR<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49550::DEBUG::2015-02-26 15:42:27,213::migration::361::vm.Vm::(run= ) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::migration downtime thread = started<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::DEBUG::2015-02-26 15:42:27,216::migration::410::vm.Vm::(mon= itor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::starting mig= ration monitor thread<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49550::DEBUG::2015-02-26 15:43:42,218::migration::370::vm.Vm::(run= ) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime= to 50<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49550::DEBUG::2015-02-26 15:44:57,222::migration::370::vm.Vm::(run= ) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime= to 100<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49550::DEBUG::2015-02-26 15:46:12,227::migration::370::vm.Vm::(run= ) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime= to 150<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:47:07,279::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (1791MiB) > lowmark (203MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:47:17,281::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (1398MiB) > lowmark (203MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49550::DEBUG::2015-02-26 15:47:27,233::migration::370::vm.Vm::(run= ) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime= to 200<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:47:27,283::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (1066MiB) > lowmark (203MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:47:37,285::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (701MiB) > lowmark (203MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:47:47,287::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (361MiB) > lowmark (203MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:48:07,291::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (683MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:48:17,292::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (350MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:48:27,294::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (18MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:48:37,296::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (646MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49550::DEBUG::2015-02-26 15:48:42,238::migration::370::vm.Vm::(run= ) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime= to 250<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:48:47,334::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (317MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:48:57,342::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (1018MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:49:07,344::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (679MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:49:17,346::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (357MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:49:27,348::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (31MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:49:37,349::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (854MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:49:47,351::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (525MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49550::DEBUG::2015-02-26 15:49:57,242::migration::370::vm.Vm::(run= ) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime= to 300<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:49:57,353::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (183MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:50:07,355::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (785MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:50:17,357::migration::458::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = stalling: remaining (457MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::WARNING::2015-02-26 15:50:27,359::migration::445::vm.Vm::(m= onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration = is stuck: Hasn't progressed in 150.069458961 seconds. Aborting.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49551::DEBUG::2015-02-26 15:50:27,362::migration::470::vm.Vm::(sto= p) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::stopping migration monito= r thread<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49549::DEBUG::2015-02-26 15:50:27,852::migration::376::vm.Vm::(can= cel) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::canceling migration dow= ntime thread<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49549::DEBUG::2015-02-26 15:50:27,852::migration::470::vm.Vm::(sto= p) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::stopping migration monito= r thread<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49550::DEBUG::2015-02-26 15:50:27,853::migration::373::vm.Vm::(run= ) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::migration downtime thread = exiting<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49549::ERROR::2015-02-26 15:50:27,855::migration::161::vm.Vm::(_re= cover) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::operation aborted: mi= gration job: canceled by client<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49549::ERROR::2015-02-26 15:50:28,049::migration::260::vm.Vm::(run= ) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Failed to migrate<o:p></o:= p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-49690::DEBUG::2015-02-26 15:50:33,284::__init__::469::jsonrpc.Json= RpcServer::(_serveRequest) Calling 'VM.getMigrationStatus' in bridge with {= u'vmID': u'b75823d1-00f0-457e-a692-8b95f73907db'}<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:Wingdings;= color:#1F497D">ß</span><span lang=3D"EN-US" style=3D"color:#1F497D">s= nip </span><span lang=3D"EN-US" style=3D"font-family:Wingdings;color:#1F497D">&= agrave;</span><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p></o:p></spa= n></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">At the = same time on the destination server<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:Wingdings;= color:#1F497D">ß</span><span lang=3D"EN-US" style=3D"color:#1F497D"> = snip </span><span lang=3D"EN-US" style=3D"font-family:Wingdings;color:#1F497D">&= agrave;</span><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p></o:p></spa= n></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350501::DEBUG::2015-02-26 15:42:25,923::BindingXMLRPC::1133::vds::= (wrapper) client [172.19.2.34]::call vmGetStats with ('b75823d1-00f0-457e-a= 692-8b95f73907db',) {}<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350502::DEBUG::2015-02-26 15:42:26,018::BindingXMLRPC::1133::vds::= (wrapper) client [172.19.2.34]::call vmMigrationCreate with ALL THE MACHINE= START PARAMETERS here<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350502::INFO::2015-02-26 15:42:26,019::clientIF::394::vds::(create= Vm) vmContainerLock acquired by vm b75823d1-00f0-457e-a692-8b95f73907db<o:p= preparation<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350503::INFO::2015-02-26 15:42:26,038::vm::3261::vm.Vm::(_run) vmI= d=3D`b75823d1-00f0-457e-a692-8b95f73907db`::VM wrapper has started<o:p></o:= p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350503::WARNING::2015-02-26 15:42:26,041::vm::2056::vm.Vm::(buildC= onfDevices) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Unknown type fou= nd, device: '{'device': 'unix', 'alias': 'channel0', 'type': 'channel', 'address': {'bus': '0', 'controller': '0', = 'type': 'virtio-serial', 'port': '1'}}' found<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350503::WARNING::2015-02-26 15:42:26,041::vm::2056::vm.Vm::(buildC= onfDevices) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Unknown type fou= nd, device: '{'device': 'unix', 'alias': 'channel1', 'type': 'channel', 'address': {'bus': '0', 'controller': '0', = 'type': 'virtio-serial', 'port': '2'}}' found<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350503::DEBUG::2015-02-26 15:42:26,349::vm::1058::vm.Vm::(__init__= ) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Ignoring param (target, 10= 485760) in BalloonDevice<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350503::DEBUG::2015-02-26 15:42:26,350::vm::2294::vm.Vm::(_startUn= derlyingVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::_ongoingCreation= s released<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350502::ERROR::2015-02-26 15:42:26,351::vm::5638::vm.Vm::(_updateD= evicesDomxmlCache) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Alias not= found for device type graphics during migration at destination host<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350503::DEBUG::2015-02-26 15:42:26,353::vm::4128::vm.Vm::(_waitFor= UnderlyingMigration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Waiting= 21600 seconds for end of migration<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350502::DEBUG::2015-02-26 15:42:26,377::BindingXMLRPC::1140::vds::= (wrapper) return vmMigrationCreate with {ALL THE MACHINE PARAMETERS HERE<o:= p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:libvirtEventLoop::DEBUG::2015-02-26 15:42:27,195::vm::5571::vm.Vm::(_onLi= bvirtLifecycleEvent) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::event S= tarted detail 1 opaque None<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:libvirtEventLoop::DEBUG::2015-02-26 15:50:27,037::vm::5571::vm.Vm::(_onLi= bvirtLifecycleEvent) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::event S= topped detail 5 opaque None<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:libvirtEventLoop::INFO::2015-02-26 15:50:27,038::vm::2366::vm.Vm::(_onQem= uDeath) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::underlying process d= isconnected<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:libvirtEventLoop::INFO::2015-02-26 15:50:27,038::vm::4952::vm.Vm::(releas= eVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Release VM resources<o:= p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350503::DEBUG::2015-02-26 15:50:27,044::libvirtconnection::143::ro= ot::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Do= main not found: no domain with matching uuid 'b75823d1-00f0-457e-a692-8b95f73907db'<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350503::ERROR::2015-02-26 15:50:27,047::vm::2325::vm.Vm::(_startUn= derlyingVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Failed to start = a migration destination vm<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:MigrationError: Domain not found: no domain with matching uuid 'b75823d1-= 00f0-457e-a692-8b95f73907db'<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-350503::DEBUG::2015-02-26 15:50:27,058::vm::2786::vm.Vm::(setDownS= tatus) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Changed state to Down= : VM failed to migrate (code=3D8)<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-351517::DEBUG::2015-02-26 15:50:27,089::BindingXMLRPC::1133::vds::= (wrapper) client [172.19.2.34]::call vmDestroy with ('b75823d1-00f0-457e-a6= 92-8b95f73907db',) {}<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-351517::INFO::2015-02-26 15:50:27,092::API::332::vds::(destroy) vm= ContainerLock acquired by vm b75823d1-00f0-457e-a692-8b95f73907db<o:p></o:p=
</span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-351517::DEBUG::2015-02-26 15:50:27,154::vm::5026::vm.Vm::(destroy)= vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::destroy Called<o:p></o:p></= span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:libvirtEventLoop::WARNING::2015-02-26 15:50:27,162::utils::129::root::(rm= File) File: /var/lib/libvirt/qemu/channels/b75823d1-00f0-457e-a692-8b95f739= 07db.com.redhat.rhevm.vdsm already removed<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:libvirtEventLoop::WARNING::2015-02-26 15:50:27,163::utils::129::root::(rm= File) File: /var/lib/libvirt/qemu/channels/b75823d1-00f0-457e-a692-8b95f739= 07db.org.qemu.guest_agent.0 already removed<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:libvirtEventLoop::INFO::2015-02-26 15:50:27,164::logUtils::44::dispatcher= ::(wrapper) Run and protect: inappropriateDevices(thiefId=3D'b75823d1-00f0-= 457e-a692-8b95f73907db')<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:Thread-351517::DEBUG::2015-02-26 15:50:27,170::vm::5020::vm.Vm::(deleteVm= ) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Total desktops after destr= oy of b75823d1-00f0-457e-a692-8b95f73907db is 2<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo= g:libvirtEventLoop::WARNING::2015-02-26 15:50:27,170::vm::1953::vm.Vm::(_se= t_lastStatus) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::trying to set = state to Down when already Down<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:Wingdings;=
color:#1F497D">ß</span><span lang=3D"EN-US" style=3D"color:#1F497D"> = snip </span><span lang=3D"EN-US" style=3D"font-family:Wingdings;color:#1F497D">&= agrave;</span><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p></o:p></spa= n></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">And I c= an see the error “Domain not found: no domain with matching uuid 'b75= 823d1-00f0-457e-a692-8b95f73907db'” in the middle, but I am not entir= ely sure what this means. <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">In the = qemu log I can see this<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:Wingdings;= color:#1F497D">ß</span><span lang=3D"EN-US" style=3D"color:#1F497D"> = snip </span><span lang=3D"EN-US" style=3D"font-family:Wingdings;color:#1F497D">&= agrave;</span><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p></o:p></spa= n></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><b><span lang=3D"EN-US" style=3D"font-size:12.0pt;co= lor:#1F497D">From QEMU LOG<o:p></o:p></span></b></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">2015-02= -26 14:42:26.859+0000: starting up<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">Domain = id=3D8 is tainted: hook-script<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">2015-02= -26T14:50:26.650595Z qemu-system-x86_64: load of migration failed: Input/ou= tput error<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">2015-02= -26 14:50:26.684+0000: shutting down<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:Wingdings;= color:#1F497D">ß</span><span lang=3D"EN-US" style=3D"color:#1F497D"> = snip </span><span lang=3D"EN-US" style=3D"font-family:Wingdings;color:#1F497D">&= agrave;</span><span lang=3D"EN-US" style=3D"color:#1F497D"> <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">the tim= e here seems to be logged in GMT and in the snippets above it is CET –= ; might this be the problem ???<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">Here is= the qemu commandline (that was removed above in cleaned <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:Wingdings;= color:#1F497D">ß</span><span style=3D"color:#1F497D"> snip </span><span lang=3D"EN-US" style=3D"font-family:Wingdings;color:#1F497D">&= agrave;</span><span style=3D"color:#1F497D"><o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">/usr/bin/qemu-kvm <o:p= ></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">-name SERVERNAME <o:p>= </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-S <o:p= ></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-machin= e pc-1.0,accel=3Dkvm,usb=3Doff <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-cpu Sa= ndyBridge <o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-m 1024= 0 <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-realti= me mlock=3Doff <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-smp 4,= maxcpus=3D64,sockets=3D16,cores=3D4,threads=3D1 <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-uuid b= 75823d1-00f0-457e-a692-8b95f73907db <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-smbios= type=3D1,manufacturer=3DoVirt,product=3DoVirt Node,version=3D20-3,serial= =3D4C4C4544-0038-3210-8048-B7C04F303232,uuid=3Db75823d1-00f0-457e-a692-8b95= f73907db <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-no-use= r-config <o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-nodefa= ults <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-charde= v socket,id=3Dcharmonitor,path=3D/var/lib/libvirt/qemu/al-exchange-01.monit= or,server,nowait <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-mon ch= ardev=3Dcharmonitor,id=3Dmonitor,mode=3Dcontrol <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-rtc ba= se=3D2015-02-26T15:42:26,driftfix=3Dslew <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-global= kvm-pit.lost_tick_policy=3Ddiscard <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-no-hpe= t <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-no-shu= tdown <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-boot s= trict=3Don <o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-device= piix3-usb-uhci,id=3Dusb,bus=3Dpci.0,addr=3D0x1.0x2 <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-device= virtio-scsi-pci,id=3Dscsi0,bus=3Dpci.0,addr=3D0x4 <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-device= virtio-serial-pci,id=3Dvirtio-serial0,max_ports=3D16,bus=3Dpci.0,addr=3D0x= 5 <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-drive = file=3D/rhev/data-center/mnt/XXXX:_data_export_iso/85821683-bc7c-429b-bff2-= ce25f56fc16f/images/11111111-1111-1111-1111-111111111111/XXXX.iso,if=3Dnone= ,id=3Ddrive-ide0-1-0,readonly=3Don,format=3Draw,serial=3D -device ide-cd,bus=3Dide.1,unit=3D0,drive=3Ddrive-ide0-1-0,id=3Dide0-1-0,b= ootindex=3D2 <o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-drive = file=3D/rhev/data-center/0f954891-b1cd-4f09-99ae-75d404d95f9d/276e9ba7-e19a= -49c5-8ad7-26711934d5e4/images/c37bfa94-718c-4125-9202-bc299535eca5/299bd6d= 1-f4a0-4296-9944-176b2252a886,if=3Dnone,id=3Ddrive-virtio-disk0,format=3Dra= w,serial=3Dc37bfa94-718c-4125-9202-bc299535eca5,cache=3Dnone,werror=3Dstop,= rerror=3Dstop,aio=3Dthreads <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-device= virtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x6,drive=3Ddrive-virtio-disk= 0,id=3Dvirtio-disk0,bootindex=3D1 <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-drive = file=3D/rhev/data-center/0f954891-b1cd-4f09-99ae-75d404d95f9d/276e9ba7-e19a= -49c5-8ad7-26711934d5e4/images/75134ccc-b74e-4955-90a5-95d4ceff403b/1e5b5b8= 6-c31d-476c-acb2-a5bd6a65490b,if=3Dnone,id=3Ddrive-virtio-disk1,format=3Dra= w,serial=3D75134ccc-b74e-4955-90a5-95d4ceff403b,cache=3Dnone,werror=3Dstop,= rerror=3Dstop,aio=3Dthreads <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-device= virtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x8,drive=3Ddrive-virtio-disk= 1,id=3Dvirtio-disk1 <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-netdev= tap,fd=3D30,id=3Dhostnet0,vhost=3Don,vhostfd=3D31 <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-device= virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D00:1a:4a:c7:72:0a,bus=3Dp= ci.0,addr=3D0x3 <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-charde= v socket,id=3Dcharchannel0,path=3D/var/lib/libvirt/qemu/channels/b75823d1-0= 0f0-457e-a692-8b95f73907db.com.redhat.rhevm.vdsm,server,nowait <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-device= virtserialport,bus=3Dvirtio-serial0.0,nr=3D1,chardev=3Dcharchannel0,id=3Dc= hannel0,name=3Dcom.redhat.rhevm.vdsm <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-charde= v socket,id=3Dcharchannel1,path=3D/var/lib/libvirt/qemu/channels/b75823d1-0= 0f0-457e-a692-8b95f73907db.org.qemu.guest_agent.0,server,nowait <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-device= virtserialport,bus=3Dvirtio-serial0.0,nr=3D2,chardev=3Dcharchannel1,id=3Dc= hannel1,name=3Dorg.qemu.guest_agent.0 <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-device= usb-tablet,id=3Dinput0 <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-vnc IP= -ADDRESS:2,password <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-k de <= o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-device= cirrus-vga,id=3Dvideo0,bus=3Dpci.0,addr=3D0x2 <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-incomi= ng tcp:[::]:49152 <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-device= virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x7 <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">-msg ti= mestamp=3Don<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:Wingdings;= color:#1F497D">ß</span><span lang=3D"EN-US" style=3D"color:#1F497D"> = snip </span><span lang=3D"EN-US" style=3D"font-family:Wingdings;color:#1F497D">&= agrave;</span><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p></o:p></spa= n></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">Here is= the content of the .meta file in the directories of the 2 images above<o:p= ></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><--s= nip--> <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">DOMAIN= =3D276e9ba7-e19a-49c5-8ad7-26711934d5e4<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">VOLTYPE= =3DLEAF<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">CTIME= =3D1424445378<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">FORMAT= =3DRAW<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">IMAGE= =3Dc37bfa94-718c-4125-9202-bc299535eca5<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">DISKTYP= E=3D2<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">PUUID= =3D00000000-0000-0000-0000-000000000000<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">LEGALIT= Y=3DLEGAL<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">MTIME= =3D0<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">POOL_UU= ID=3D<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">DESCRIP= TION=3D{"DiskAlias":"al-exchange-01_Disk1","DiskDe= scription":"System"}<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">TYPE=3D= SPARSE<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">SIZE=3D= 146800640<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">EOF<o:p= ></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><--s= nip--><o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><--s= nip--> <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">DOMAIN= =3D276e9ba7-e19a-49c5-8ad7-26711934d5e4<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">CTIME= =3D1424785898<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">FORMAT= =3DRAW<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">DISKTYP= E=3D2<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">LEGALIT= Y=3DLEGAL<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">SIZE=3D= 419430400<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">VOLTYPE= =3DLEAF<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">DESCRIP= TION=3D{"DiskAlias":"al-exchange-01_Disk2","DiskDe= scription":"Data"}<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">IMAGE= =3D75134ccc-b74e-4955-90a5-95d4ceff403b<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">PUUID= =3D00000000-0000-0000-0000-000000000000<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">MTIME= =3D0<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">POOL_UU= ID=3D<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">TYPE=3D= SPARSE<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D">EOF<o:p= ></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><--s= nip--><o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n= bsp;</o:p></span></p> <p class=3D"MsoNormal"><a name=3D"_MailEndCompose"><span lang=3D"EN-US" sty= le=3D"color:#1F497D"><o:p> </o:p></span></a></p> <div> <div style=3D"border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in = 0in 0in"> <p class=3D"MsoNormal"><b><span lang=3D"EN-US" style=3D"color:windowtext;ms= o-fareast-language:DE">From:</span></b><span lang=3D"EN-US" style=3D"color:= windowtext;mso-fareast-language:DE"> Roy Golan [mailto:rgolan@redhat.com] <br> <b>Sent:</b> Wednesday, February 18, 2015 12:12 PM<br> <b>To:</b> Soeren Malchow; users@ovirt.org<br> <b>Subject:</b> Re: [ovirt-users] Hosted Engine Migration fails<o:p></o:p><= /span></p> </div> </div> <p class=3D"MsoNormal"><o:p> </o:p></p> <div> <p class=3D"MsoNormal">On 02/16/2015 04:55 AM, Soeren Malchow wrote:<span s= tyle=3D"font-size:12.0pt;mso-fareast-language:DE"><o:p></o:p></span></p> </div> <blockquote style=3D"margin-top:5.0pt;margin-bottom:5.0pt"> <p class=3D"MsoNormal">Dear all,<o:p></o:p></p> <p class=3D"MsoNormal"> <o:p></o:p></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">we ahve a setup with several ho= sts running fedora 10 with the virt-preview packages installed (for snapsho= t live merge) and a hosted engine running centos 6.6-</span><o:p></o:p></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"> </span><o:p></o:p></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">We are experiencing a problem w= ith the Live Migration of the Hosted Engine, in the case of setting the hos= t for the Engine into maintenance as well as a manual migration.</span><o:p= ></o:p></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">I tried this on the “ovir= tmgmt” network and when that failed I did some research and tried to = use another network interface (separate from ovirtmgmt on layer 2), this al= so fails.</span><o:p></o:p></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"> </span><o:p></o:p></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">It looks as if the migration is= still going through the ovirtmgmt interface, at least judging from the net= work traffic, and I think the error that I found (RHBZ#919201) is actually = the right one. </span><o:p></o:p></p> <p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a= lt:auto"><span lang=3D"EN-US" style=3D"font-size:12.0pt"> --</span><o:= p></o:p></p> <p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a= lt:auto"><span lang=3D"EN-US" style=3D"font-size:12.0pt">vdsm.log:Thread-67= 45::WARNING::2015-02-16 03:22:25,743::migration::458::vm.Vm::(monitor_migra= tion) vmId=3D`XXX`::Migration stalling: remaining (35MiB) > lowmark (15MiB). Refer to RHBZ#919201.</span><o:p></o:p></p> <p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a= lt:auto"><span lang=3D"EN-US" style=3D"font-size:12.0pt">vdsm.log:Thread-67= 45::WARNING::2015-02-16 03:22:35,745::migration::458::vm.Vm::(monitor_migra= tion) vmId=3D`XXX`::Migration stalling: remaining (129MiB) > lowmark (15MiB). Refer to RHBZ#919201.</span><o:p></o:p></p> <p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a= lt:auto"><span lang=3D"EN-US" style=3D"font-size:12.0pt">vdsm.log:Thread-67= 45::WARNING::2015-02-16 03:22:45,747::migration::458::vm.Vm::(monitor_migra= tion) vmId=3D`XXX`::Migration stalling: remaining (42MiB) > lowmark (15MiB). Refer to RHBZ#919201.</span><o:p></o:p></p> <p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a= lt:auto"><span lang=3D"EN-US" style=3D"font-size:12.0pt">vdsm.log:Thread-67= 45::WARNING::2015-02-16 03:22:55,749::migration::458::vm.Vm::(monitor_migra= tion) vmId=3D`XXX`::Migration stalling: remaining (88MiB) > lowmark (15MiB). Refer to RHBZ#919201.</span><o:p></o:p></p> <p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a= lt:auto"><span lang=3D"EN-US" style=3D"font-size:12.0pt">--</span><o:p></o:= p></p> <p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a= lt:auto"><span lang=3D"EN-US" style=3D"font-size:12.0pt">The ovirtmgmt inte= rface is 2 x 1Gbit (LACP connected to Dell Switches with MLAG) and by far n= ot fully utilized.</span><o:p></o:p></p> <p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a= lt:auto"><span lang=3D"EN-US" style=3D"font-size:12.0pt">Can anyone help me= where to go form here ?</span><o:p></o:p></p> </blockquote> <p class=3D"MsoNormal"><span style=3D"font-size:12.0pt;font-family:"Ti= mes New Roman",serif;mso-fareast-language:DE"><br> still need the whole log. probably the guest (engine) is doing lots of memo= ry i/o if you got a fair amount of running VMs and Hosts. that will s= talls the migration because <br> the guest pages are getting dirty faster than qemu can copy.<br> <br> <br> you have 2 options:<br> <br> 1. try several more times.<br> <br> 2. shutdown the engine vm, it should start on another host<br> <br> <br> <o:p></o:p></span></p> <blockquote style=3D"margin-top:5.0pt;margin-bottom:5.0pt"> <p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a= lt:auto"><span lang=3D"EN-US" style=3D"font-size:12.0pt">Regards</span><o:p= ></o:p></p> <p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a= lt:auto"><span lang=3D"EN-US" style=3D"font-size:12.0pt">Soeren </span><o:p></o:p></p> <p class=3D"MsoNormal"><span style=3D"font-size:12.0pt;font-family:"Ti= mes New Roman",serif;mso-fareast-language:DE"><br> <br> <br> <o:p></o:p></span></p> <pre>_______________________________________________<o:p></o:p></pre> <pre>Users mailing list<o:p></o:p></pre> <pre><a href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><o:p></o:p></pre= > <pre><a href=3D"http://lists.ovirt.org/mailman/listinfo/users">http://lists= .ovirt.org/mailman/listinfo/users</a><o:p></o:p></pre> </blockquote> <p class=3D"MsoNormal"><span style=3D"font-size:12.0pt;font-family:"Ti= mes New Roman",serif;mso-fareast-language:DE"><o:p> </o:p></span>= </p> </div> </body> </html>
--_000_687766E0007010429FBFF98CA70824888A0C30mcexch01mcongroup_--