Re: [ovirt-users] Hosted Engine Migration fails

Thursday, 26 February 2015

--_000_687766E0007010429FBFF98CA70824888A0C30mcexch01mcongroup_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,

we tried this (Roys Mail below) , and yes, shutdown always works.

But now this problem comes up with regular machines now as well.

The environment is setup like this

Engine: Ovirt 3.5.1.1-1.el6 - on CentOS6

The Storage Backend is gluster 3.6.2-1.el7 on CentOS 7

Compute hosts: Libvirt Version: libvirt-1.2.9.1-2.fc20, kvm 2.1.2 - 7.fc20,=
 vdsm vdsm-4.16.10-8.gitc937927.fc20

All compute Servers are 100% identical.

The storage cluster was tested manually and works just fine.

The network interfaces are not fully utilized, more like 15%.

Log output ist as below. The thing in the log output I do not understand is=
 this

"2015-02-26T14:50:26.650595Z qemu-system-x86_64: load of migration failed: =
Input/output error"

...
From the qemu log. 
Also if I shut down machines, put their host into maintenance and start the=
m somewhere else, everything works just fine.

Can someone help with this ? Any idea where to look ?

Regards
Soeren

...
From VDSM Log 
I just tried to migrate a machine, this here happens on the source

<-- snip --...

vdsm.log:Thread-49548::DEBUG::2015-02-26 15:42:26,692::__init__::469::jsonr=
pc.JsonRpcServer::(_serveRequest) Calling 'VM.migrate' in bridge with {u'pa=
rams': {u'tunneled': u'false', u'dstqemu':
u'172.19.2.31', u'src': u'comput=
e04', u'dst': u'compute01:54321', u'vmId':
u'b75823d1-00f0-457e-a692-8b95f7=
3907db', u'abortOnError': u'true', u'method':
u'online'}, u'vmID': u'b75823=
d1-00f0-457e-a692-8b95f73907db'}
vdsm.log:Thread-49548::DEBUG::2015-02-26 15:42:26,694::API::510::vds::(migr=
ate) {u'tunneled': u'false', u'dstqemu': u'IPADDR',
u'src': u'compute04', u=
'dst': u'compute01:54321', u'vmId':
u'b75823d1-00f0-457e-a692-8b95f73907db'=
, u'abortOnError': u'true', u'method': u'online'}
vdsm.log:Thread-49549::DEBUG::2015-02-26 15:42:26,699::migration::103::vm.V=
m::(_setupVdsConnection) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Des=
tination server is: compute01:54321
vdsm.log:Thread-49549::DEBUG::2015-02-26 15:42:26,702::migration::105::vm.V=
m::(_setupVdsConnection) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Ini=
tiating connection with destination
vdsm.log:Thread-49549::DEBUG::2015-02-26 15:42:26,733::migration::155::vm.V=
m::(_prepareGuest) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration=
 started
vdsm.log:Thread-49549::DEBUG::2015-02-26 15:42:26,755::migration::238::vm.V=
m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::migration semaphore=
 acquired after 0 seconds
vdsm.log:Thread-49549::DEBUG::2015-02-26 15:42:27,211::migration::298::vm.V=
m::(_startUnderlyingMigration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db=
`::starting migration to qemu+tls://compute01/system with miguri tcp://IPAD=
DR
vdsm.log:Thread-49550::DEBUG::2015-02-26 15:42:27,213::migration::361::vm.V=
m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::migration downtime =
thread started
vdsm.log:Thread-49551::DEBUG::2015-02-26 15:42:27,216::migration::410::vm.V=
m::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::start=
ing migration monitor thread
vdsm.log:Thread-49550::DEBUG::2015-02-26 15:43:42,218::migration::370::vm.V=
m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration d=
owntime to 50
vdsm.log:Thread-49550::DEBUG::2015-02-26 15:44:57,222::migration::370::vm.V=
m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration d=
owntime to 100
vdsm.log:Thread-49550::DEBUG::2015-02-26 15:46:12,227::migration::370::vm.V=
m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration d=
owntime to 150
vdsm.log:Thread-49551::WARNING::2015-02-26 15:47:07,279::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (1791MiB) > lowmark (203MiB). Refer to RHBZ#9192=
01.
vdsm.log:Thread-49551::WARNING::2015-02-26 15:47:17,281::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (1398MiB) > lowmark (203MiB). Refer to RHBZ#9192=
01.
vdsm.log:Thread-49550::DEBUG::2015-02-26 15:47:27,233::migration::370::vm.V=
m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration d=
owntime to 200
vdsm.log:Thread-49551::WARNING::2015-02-26 15:47:27,283::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (1066MiB) > lowmark (203MiB). Refer to RHBZ#9192=
01.
vdsm.log:Thread-49551::WARNING::2015-02-26 15:47:37,285::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (701MiB) > lowmark (203MiB). Refer to RHBZ#91920=
1.
vdsm.log:Thread-49551::WARNING::2015-02-26 15:47:47,287::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (361MiB) > lowmark (203MiB). Refer to RHBZ#91920=
1.
vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:07,291::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (683MiB) > lowmark (13MiB). Refer to RHBZ#919201=
.
vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:17,292::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (350MiB) > lowmark (13MiB). Refer to RHBZ#919201=
.
vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:27,294::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (18MiB) > lowmark (13MiB). Refer to RHBZ#919201.
vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:37,296::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (646MiB) > lowmark (13MiB). Refer to RHBZ#919201=
.
vdsm.log:Thread-49550::DEBUG::2015-02-26 15:48:42,238::migration::370::vm.V=
m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration d=
owntime to 250
vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:47,334::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (317MiB) > lowmark (13MiB). Refer to RHBZ#919201=
.
vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:57,342::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (1018MiB) > lowmark (13MiB). Refer to RHBZ#91920=
1.
vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:07,344::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (679MiB) > lowmark (13MiB). Refer to RHBZ#919201=
.
vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:17,346::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (357MiB) > lowmark (13MiB). Refer to RHBZ#919201=
.
vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:27,348::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (31MiB) > lowmark (13MiB). Refer to RHBZ#919201.
vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:37,349::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (854MiB) > lowmark (13MiB). Refer to RHBZ#919201=
.
vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:47,351::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (525MiB) > lowmark (13MiB). Refer to RHBZ#919201=
.
vdsm.log:Thread-49550::DEBUG::2015-02-26 15:49:57,242::migration::370::vm.V=
m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration d=
owntime to 300
vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:57,353::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (183MiB) > lowmark (13MiB). Refer to RHBZ#919201=
.
vdsm.log:Thread-49551::WARNING::2015-02-26 15:50:07,355::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (785MiB) > lowmark (13MiB). Refer to RHBZ#919201=
.
vdsm.log:Thread-49551::WARNING::2015-02-26 15:50:17,357::migration::458::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration stalling: remaining (457MiB) > lowmark (13MiB). Refer to RHBZ#919201=
.
vdsm.log:Thread-49551::WARNING::2015-02-26 15:50:27,359::migration::445::vm=
.Vm::(monitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Mig=
ration is stuck: Hasn't progressed in 150.069458961 seconds. Aborting.
vdsm.log:Thread-49551::DEBUG::2015-02-26 15:50:27,362::migration::470::vm.V=
m::(stop) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::stopping migration=
 monitor thread
vdsm.log:Thread-49549::DEBUG::2015-02-26 15:50:27,852::migration::376::vm.V=
m::(cancel) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::canceling migrat=
ion downtime thread
vdsm.log:Thread-49549::DEBUG::2015-02-26 15:50:27,852::migration::470::vm.V=
m::(stop) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::stopping migration=
 monitor thread
vdsm.log:Thread-49550::DEBUG::2015-02-26 15:50:27,853::migration::373::vm.V=
m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::migration downtime =
thread exiting
vdsm.log:Thread-49549::ERROR::2015-02-26 15:50:27,855::migration::161::vm.V=
m::(_recover) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::operation abor=
ted: migration job: canceled by client
vdsm.log:Thread-49549::ERROR::2015-02-26 15:50:28,049::migration::260::vm.V=
m::(run) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Failed to migrate
vdsm.log:Thread-49690::DEBUG::2015-02-26 15:50:33,284::__init__::469::jsonr=
pc.JsonRpcServer::(_serveRequest) Calling 'VM.getMigrationStatus' in bridge=
 with {u'vmID': u'b75823d1-00f0-457e-a692-8b95f73907db'}

<--snip --...

At the same time on the destination server

<-- snip --...

vdsm.log:Thread-350501::DEBUG::2015-02-26 15:42:25,923::BindingXMLRPC::1133=
::vds::(wrapper) client [172.19.2.34]::call vmGetStats with ('b75823d1-00f0=
-457e-a692-8b95f73907db',) {}
vdsm.log:Thread-350502::DEBUG::2015-02-26 15:42:26,018::BindingXMLRPC::1133=
::vds::(wrapper) client [172.19.2.34]::call vmMigrationCreate with ALL THE =
MACHINE START PARAMETERS here
vdsm.log:Thread-350502::INFO::2015-02-26 15:42:26,019::clientIF::394::vds::=
(createVm) vmContainerLock acquired by vm b75823d1-00f0-457e-a692-8b95f7390=
7db
vdsm.log:Thread-350502::DEBUG::2015-02-26 15:42:26,033::clientIF::407::vds:=
:(createVm) Total desktops after creation of b75823d1-00f0-457e-a692-8b95f7=
3907db is 3
vdsm.log:Thread-350503::DEBUG::2015-02-26 15:42:26,033::vm::2264::vm.Vm::(_=
startUnderlyingVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Start
vdsm.log:Thread-350502::DEBUG::2015-02-26 15:42:26,036::vm::5658::vm.Vm::(w=
aitForMigrationDestinationPrepare) vmId=3D`b75823d1-00f0-457e-a692-8b95f739=
07db`::migration destination: waiting for VM creation
vdsm.log:Thread-350503::DEBUG::2015-02-26 15:42:26,036::vm::2268::vm.Vm::(_=
startUnderlyingVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::_ongoingC=
reations acquired
vdsm.log:Thread-350502::DEBUG::2015-02-26 15:42:26,038::vm::5663::vm.Vm::(w=
aitForMigrationDestinationPrepare) vmId=3D`b75823d1-00f0-457e-a692-8b95f739=
07db`::migration destination: waiting 48s for path preparation
vdsm.log:Thread-350503::INFO::2015-02-26 15:42:26,038::vm::3261::vm.Vm::(_r=
un) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::VM wrapper has started
vdsm.log:Thread-350503::WARNING::2015-02-26 15:42:26,041::vm::2056::vm.Vm::=
(buildConfDevices) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Unknown t=
ype found, device: '{'device': 'unix', 'alias':
'channel0', 'type': 'channe=
l', 'address': {'bus': '0', 'controller': '0',
'type': 'virtio-serial', 'po=
rt': '1'}}' found
vdsm.log:Thread-350503::WARNING::2015-02-26 15:42:26,041::vm::2056::vm.Vm::=
(buildConfDevices) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Unknown t=
ype found, device: '{'device': 'unix', 'alias':
'channel1', 'type': 'channe=
l', 'address': {'bus': '0', 'controller': '0',
'type': 'virtio-serial', 'po=
rt': '2'}}' found
vdsm.log:Thread-350503::DEBUG::2015-02-26 15:42:26,349::vm::1058::vm.Vm::(_=
_init__) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Ignoring param (tar=
get, 10485760) in BalloonDevice
vdsm.log:Thread-350503::DEBUG::2015-02-26 15:42:26,350::vm::2294::vm.Vm::(_=
startUnderlyingVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::_ongoingC=
reations released
vdsm.log:Thread-350502::ERROR::2015-02-26 15:42:26,351::vm::5638::vm.Vm::(_=
updateDevicesDomxmlCache) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Al=
ias not found for device type graphics during migration at destination host
vdsm.log:Thread-350503::DEBUG::2015-02-26 15:42:26,353::vm::4128::vm.Vm::(_=
waitForUnderlyingMigration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::=
Waiting 21600 seconds for end of migration
vdsm.log:Thread-350502::DEBUG::2015-02-26 15:42:26,377::BindingXMLRPC::1140=
::vds::(wrapper) return vmMigrationCreate with {ALL THE MACHINE PARAMETERS =
HERE
vdsm.log:libvirtEventLoop::DEBUG::2015-02-26 15:42:27,195::vm::5571::vm.Vm:=
:(_onLibvirtLifecycleEvent) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::=
event Started detail 1 opaque None
vdsm.log:libvirtEventLoop::DEBUG::2015-02-26 15:50:27,037::vm::5571::vm.Vm:=
:(_onLibvirtLifecycleEvent) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::=
event Stopped detail 5 opaque None
vdsm.log:libvirtEventLoop::INFO::2015-02-26 15:50:27,038::vm::2366::vm.Vm::=
(_onQemuDeath) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::underlying pr=
ocess disconnected
vdsm.log:libvirtEventLoop::INFO::2015-02-26 15:50:27,038::vm::4952::vm.Vm::=
(releaseVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Release VM resou=
rces
vdsm.log:Thread-350503::DEBUG::2015-02-26 15:50:27,044::libvirtconnection::=
143::root::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 mess=
age: Domain not found: no domain with matching uuid 'b75823d1-00f0-457e-a69=
2-8b95f73907db'
vdsm.log:Thread-350503::ERROR::2015-02-26 15:50:27,047::vm::2325::vm.Vm::(_=
startUnderlyingVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Failed to=
 start a migration destination vm
vdsm.log:MigrationError: Domain not found: no domain with matching uuid 'b7=
5823d1-00f0-457e-a692-8b95f73907db'
vdsm.log:Thread-350503::DEBUG::2015-02-26 15:50:27,058::vm::2786::vm.Vm::(s=
etDownStatus) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Changed state =
to Down: VM failed to migrate (code=3D8)
vdsm.log:Thread-351517::DEBUG::2015-02-26 15:50:27,089::BindingXMLRPC::1133=
::vds::(wrapper) client [172.19.2.34]::call vmDestroy with ('b75823d1-00f0-=
457e-a692-8b95f73907db',) {}
vdsm.log:Thread-351517::INFO::2015-02-26 15:50:27,092::API::332::vds::(dest=
roy) vmContainerLock acquired by vm b75823d1-00f0-457e-a692-8b95f73907db
vdsm.log:Thread-351517::DEBUG::2015-02-26 15:50:27,154::vm::5026::vm.Vm::(d=
estroy) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::destroy Called
vdsm.log:libvirtEventLoop::WARNING::2015-02-26 15:50:27,162::utils::129::ro=
ot::(rmFile) File: /var/lib/libvirt/qemu/channels/b75823d1-00f0-457e-a692-8=
b95f73907db.com.redhat.rhevm.vdsm already removed
vdsm.log:libvirtEventLoop::WARNING::2015-02-26 15:50:27,163::utils::129::ro=
ot::(rmFile) File: /var/lib/libvirt/qemu/channels/b75823d1-00f0-457e-a692-8=
b95f73907db.org.qemu.guest_agent.0 already removed
vdsm.log:libvirtEventLoop::INFO::2015-02-26 15:50:27,164::logUtils::44::dis=
patcher::(wrapper) Run and protect: inappropriateDevices(thiefId=3D'b75823d=
1-00f0-457e-a692-8b95f73907db')
vdsm.log:Thread-351517::DEBUG::2015-02-26 15:50:27,170::vm::5020::vm.Vm::(d=
eleteVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Total desktops afte=
r destroy of b75823d1-00f0-457e-a692-8b95f73907db is 2
vdsm.log:libvirtEventLoop::WARNING::2015-02-26 15:50:27,170::vm::1953::vm.V=
m::(_set_lastStatus) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::trying =
to set state to Down when already Down

<-- snip --...

And I can see the error "Domain not found: no domain with matching uuid 'b7=
5823d1-00f0-457e-a692-8b95f73907db'" in the middle, but I am not entirely s=
ure what this means.

In the qemu log I can see this

<-- snip --...

...
From QEMU LOG 
2015-02-26 14:42:26.859+0000: starting up
Domain id=3D8 is tainted: hook-script
2015-02-26T14:50:26.650595Z qemu-system-x86_64: load of migration failed: I=
nput/output error
2015-02-26 14:50:26.684+0000: shutting down

<-- snip --...

the time here seems to be logged in GMT and in the snippets above it is CET=
 - might this be the problem ???

Here is the qemu commandline (that was removed above in cleaned

<-- snip --...
 /usr/bin/qemu-kvm
-name SERVERNAME
-S
-machine pc-1.0,accel=3Dkvm,usb=3Doff
-cpu SandyBridge
-m 10240
-realtime mlock=3Doff
-smp 4,maxcpus=3D64,sockets=3D16,cores=3D4,threads=3D1
-uuid b75823d1-00f0-457e-a692-8b95f73907db
-smbios type=3D1,manufacturer=3DoVirt,product=3DoVirt Node,version=3D20-3,s=
erial=3D4C4C4544-0038-3210-8048-B7C04F303232,uuid=3Db75823d1-00f0-457e-a692=
-8b95f73907db
-no-user-config
-nodefaults
-chardev socket,id=3Dcharmonitor,path=3D/var/lib/libvirt/qemu/al-exchange-0=
1.monitor,server,nowait
-mon chardev=3Dcharmonitor,id=3Dmonitor,mode=3Dcontrol
-rtc base=3D2015-02-26T15:42:26,driftfix=3Dslew
-global kvm-pit.lost_tick_policy=3Ddiscard
-no-hpet
-no-shutdown
-boot strict=3Don
-device piix3-usb-uhci,id=3Dusb,bus=3Dpci.0,addr=3D0x1.0x2
-device virtio-scsi-pci,id=3Dscsi0,bus=3Dpci.0,addr=3D0x4
-device virtio-serial-pci,id=3Dvirtio-serial0,max_ports=3D16,bus=3Dpci.0,ad=
dr=3D0x5
-drive file=3D/rhev/data-center/mnt/XXXX:_data_export_iso/85821683-bc7c-429=
b-bff2-ce25f56fc16f/images/11111111-1111-1111-1111-111111111111/XXXX.iso,if=
=3Dnone,id=3Ddrive-ide0-1-0,readonly=3Don,format=3Draw,serial=3D -device id=
e-cd,bus=3Dide.1,unit=3D0,drive=3Ddrive-ide0-1-0,id=3Dide0-1-0,bootindex=3D=
2
-drive file=3D/rhev/data-center/0f954891-b1cd-4f09-99ae-75d404d95f9d/276e9b=
a7-e19a-49c5-8ad7-26711934d5e4/images/c37bfa94-718c-4125-9202-bc299535eca5/=
299bd6d1-f4a0-4296-9944-176b2252a886,if=3Dnone,id=3Ddrive-virtio-disk0,form=
at=3Draw,serial=3Dc37bfa94-718c-4125-9202-bc299535eca5,cache=3Dnone,werror=
=3Dstop,rerror=3Dstop,aio=3Dthreads
-device virtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x6,drive=3Ddrive-virt=
io-disk0,id=3Dvirtio-disk0,bootindex=3D1
-drive file=3D/rhev/data-center/0f954891-b1cd-4f09-99ae-75d404d95f9d/276e9b=
a7-e19a-49c5-8ad7-26711934d5e4/images/75134ccc-b74e-4955-90a5-95d4ceff403b/=
1e5b5b86-c31d-476c-acb2-a5bd6a65490b,if=3Dnone,id=3Ddrive-virtio-disk1,form=
at=3Draw,serial=3D75134ccc-b74e-4955-90a5-95d4ceff403b,cache=3Dnone,werror=
=3Dstop,rerror=3Dstop,aio=3Dthreads
-device virtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x8,drive=3Ddrive-virt=
io-disk1,id=3Dvirtio-disk1
-netdev tap,fd=3D30,id=3Dhostnet0,vhost=3Don,vhostfd=3D31
-device virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D00:1a:4a:c7:72:0a,=
bus=3Dpci.0,addr=3D0x3
-chardev socket,id=3Dcharchannel0,path=3D/var/lib/libvirt/qemu/channels/b75=
823d1-00f0-457e-a692-8b95f73907db.com.redhat.rhevm.vdsm,server,nowait
-device virtserialport,bus=3Dvirtio-serial0.0,nr=3D1,chardev=3Dcharchannel0=
,id=3Dchannel0,name=3Dcom.redhat.rhevm.vdsm
-chardev socket,id=3Dcharchannel1,path=3D/var/lib/libvirt/qemu/channels/b75=
823d1-00f0-457e-a692-8b95f73907db.org.qemu.guest_agent.0,server,nowait
-device virtserialport,bus=3Dvirtio-serial0.0,nr=3D2,chardev=3Dcharchannel1=
,id=3Dchannel1,name=3Dorg.qemu.guest_agent.0
-device usb-tablet,id=3Dinput0
-vnc IP-ADDRESS:2,password
-k de
-device cirrus-vga,id=3Dvideo0,bus=3Dpci.0,addr=3D0x2
-incoming tcp:[::]:49152
-device virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x7
-msg timestamp=3Don
<-- snip --...

Here is the content of the .meta file in the directories of the 2 images ab=
ove

<--snip--...

DOMAIN=3D276e9ba7-e19a-49c5-8ad7-26711934d5e4
VOLTYPE=3DLEAF
CTIME=3D1424445378
FORMAT=3DRAW
IMAGE=3Dc37bfa94-718c-4125-9202-bc299535eca5
DISKTYPE=3D2
PUUID=3D00000000-0000-0000-0000-000000000000
LEGALITY=3DLEGAL
MTIME=3D0
POOL_UUID=3D
DESCRIPTION=3D{"DiskAlias":"al-exchange-01_Disk1","DiskDescription":"System=
"}
TYPE=3DSPARSE
SIZE=3D146800640
EOF
<--snip--...

<--snip--...

DOMAIN=3D276e9ba7-e19a-49c5-8ad7-26711934d5e4
CTIME=3D1424785898
FORMAT=3DRAW
DISKTYPE=3D2
LEGALITY=3DLEGAL
SIZE=3D419430400
VOLTYPE=3DLEAF
DESCRIPTION=3D{"DiskAlias":"al-exchange-01_Disk2","DiskDescription":"Data"}
IMAGE=3D75134ccc-b74e-4955-90a5-95d4ceff403b
PUUID=3D00000000-0000-0000-0000-000000000000
MTIME=3D0
POOL_UUID=3D
TYPE=3DSPARSE
EOF
<--snip--...

From: Roy Golan [mailto:rgolan@redhat.com]
Sent: Wednesday, February 18, 2015 12:12 PM
To: Soeren Malchow; users(a)ovirt.org
Subject: Re: [ovirt-users] Hosted Engine Migration fails

On 02/16/2015 04:55 AM, Soeren Malchow wrote:
Dear all,

we ahve a setup with several hosts running fedora 10 with the virt-preview =
packages installed (for snapshot live merge) and a hosted engine running ce=
ntos 6.6-

We are experiencing a problem with the Live Migration of the Hosted Engine,=
 in the case of setting the host for the Engine into maintenance as well as=
 a manual migration.
I tried this on the "ovirtmgmt" network and when that failed I did some res=
earch and tried to use another network interface (separate from ovirtmgmt o=
n layer 2), this also fails.

It looks as if the migration is still going through the ovirtmgmt interface=
, at least judging from the network traffic, and I think the error that I f=
ound (RHBZ#919201) is actually the right one.
 --
vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:25,743::migration::458::vm.=
Vm::(monitor_migration) vmId=3D`XXX`::Migration stalling: remaining (35MiB)=
...
 lowmark (15MiB). Refer to RHBZ#919201.
vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:35,745::migration::458::vm.=
Vm::(monitor_migration) vmId=3D`XXX`::Migration stalling: remaining (129MiB=
)...
 lowmark (15MiB). Refer to RHBZ#919201.
vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:45,747::migration::458::vm.=
Vm::(monitor_migration) vmId=3D`XXX`::Migration stalling: remaining (42MiB)=
...
 lowmark (15MiB). Refer to RHBZ#919201.
vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:55,749::migration::458::vm.=
Vm::(monitor_migration) vmId=3D`XXX`::Migration stalling: remaining (88MiB)=
...
 lowmark (15MiB). Refer to RHBZ#919201. --
The ovirtmgmt interface is 2 x 1Gbit (LACP connected to Dell Switches with =
MLAG) and by far not fully utilized.
Can anyone help me where to go form here ?

still need the whole log. probably the guest (engine) is doing lots of memo=
ry i/o if you got a fair amount of running VMs and Hosts.  that will stalls=
 the migration because
the guest pages are getting dirty faster than qemu can copy.

you have 2 options:

1. try several more times.

2. shutdown the engine vm, it should start on another host

Regards
Soeren

_______________________________________________

Users mailing list

Users@ovirt.org<mailto:Users@ovirt.org...

http://lists.ovirt.org/mailman/listinfo/users

--_000_687766E0007010429FBFF98CA70824888A0C30mcexch01mcongroup_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml"
xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word"
=
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml"
xmlns=3D"http:=
//www.w3.org/TR/REC-html40"...

<head...
 <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Dus-ascii"=
...
 <meta name=3D"Generator" content=3D"Microsoft
Word 15 (filtered medium)"...
 <style><!--
/* Font Definitions */
@font-face
	{font-family:Wingdings;
	panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:Consolas;
	panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;
	color:black;
	mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:#0563C1;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:#954F72;
	text-decoration:underline;}
pre
	{mso-style-priority:99;
	mso-style-link:"HTML Preformatted Char";
	margin:0in;
	margin-bottom:.0001pt;
	font-size:10.0pt;
	font-family:"Courier New";
	color:black;}
span.EmailStyle17
	{mso-style-type:personal;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
span.HTMLPreformattedChar
	{mso-style-name:"HTML Preformatted Char";
	mso-style-priority:99;
	mso-style-link:"HTML Preformatted";
	font-family:Consolas;
	color:black;
	mso-fareast-language:EN-US;}
span.EmailStyle20
	{mso-style-type:personal-reply;
	font-family:"Calibri",sans-serif;
	color:#1F497D;}
span.gwt-inlinelabel
	{mso-style-name:gwt-inlinelabel;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-size:10.0pt;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:70.85pt 70.85pt 56.7pt 70.85pt;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml...

<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /...
 </xml><![endif]--><!--[if gte mso
9]><xml...
 <o:shapelayout
v:ext=3D"edit"...
 <o:idmap v:ext=3D"edit"
data=3D"1" /...

</o:shapelayout></xml><![endif]--...

</head...
 <body bgcolor=3D"white"
lang=3D"DE" link=3D"#0563C1" vlink=3D"#954F72"...

<div class=3D"WordSection1"...
 <p class=3D"MsoNormal"><span
style=3D"color:#1F497D">Hi,<o:p></o:p></span><=
/p...
 <p class=3D"MsoNormal"><span
style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
n></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">we trie=
d this (Roys Mail below) , and yes, shutdown always
works.<o:p></o:p></span=
></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">But now=
 this problem comes up with regular machines now as
well.<o:p></o:p></span>=
</p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">The env=
ironment is setup like this<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F4E79;mso-styl=
e-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><span s=
tyle=3D"color:windowtext">Engine: Ovirt
</span></span><span class=3D"gwt-inlinelabel"><span
lang=3D"EN-US" style=3D=
"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fil=
l-alpha:100.0%"><span style=3D"color:windowtext">3.5.1.1-1.el6
&#8211; on C=
entOS6
<o:p></o:p></span></span></span></p...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><o:p>&nbsp;</o:p></span></span></p...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">The
Storage Backe=
nd is gluster 3.6.2-1.el7 on CentOS
7<o:p></o:p></span></span></span></p...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><o:p>&nbsp;</o:p></span></span></p...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">Compute
hosts: Li=
bvirt Version: libvirt-1.2.9.1-2.fc20, kvm
 2.1.2 - 7.fc20, vdsm
vdsm-4.16.10-8.gitc937927.fc20<o:p></o:p></span></spa=
n></span></p...
 <p
class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span
lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><o:p>&nbsp;</o:p></span></span></p...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">All
compute Serve=
rs are 100%
identical.<o:p></o:p></span></span></span></p...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><o:p>&nbsp;</o:p></span></span></p...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">The
storage clust=
er was tested manually and works just
fine.<o:p></o:p></span></span></span>=
</p...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><o:p>&nbsp;</o:p></span></span></p...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">The
network inter=
faces are not fully utilized, more like
15%.<o:p></o:p></span></span></span=
></p...
 <p
class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span
lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><o:p>&nbsp;</o:p></span></span></p...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">Log
output ist as=
 below. The thing in the log output I do not
 understand is
this<o:p></o:p></span></span></span></p...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><o:p>&nbsp;</o:p></span></span></p...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><span
style=3D"color:windowtext">&#8220;</span></s=
pan></span><span lang=3D"EN-US"
style=3D"color:#1F497D">2015-02-26T14:50:26=
.650595Z
 qemu-system-x86_64: load of migration failed: Input/output error&#8221; <o=
:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">From th=
e qemu log.<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">Also if=
 I shut down machines, put their host into maintenance and start them somew=
here else, everything works just fine.</span><span
class=3D"gwt-inlinelabel=
"><span lang=3D"EN-US"
style=3D"color:#1F4E79;mso-style-textfill-fill-color=
:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><o:p></o:p></span></span></p=
...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><o:p>&nbsp;</o:p></span></span></p...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">Can
someone help =
with this ? Any idea where to look
?<o:p></o:p></span></span></span></p...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><o:p>&nbsp;</o:p></span></span></p...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><span
style=3D"color:windowtext">Regards<o:p></o:p=
></span></span></span></p...
 <p
class=3D"MsoNormal"><span class=3D"gwt-inlinelabel"><span
lang=3D"EN-US"=
 style=3D"color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-tex=
tfill-fill-alpha:100.0%"><span style=3D"color:windowtext">Soeren
<o:p></o:p></span></span></span></p...
 <p class=3D"MsoNormal"><span
class=3D"gwt-inlinelabel"><span lang=3D"EN-US"=
><o:p>&nbsp;</o:p></span></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><b><span lang=3D"EN-US"
style=3D"font-size:12.0pt;co=
lor:#1F497D">From VDSM Log<o:p></o:p></span></b></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">I just =
tried to migrate a machine, this here happens on the source
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"font-family:Wingdings;=
color:#1F497D">&szlig;</span><span lang=3D"EN-US"
style=3D"color:#1F497D"> =
snip
</span><span lang=3D"EN-US"
style=3D"font-family:Wingdings;color:#1F497D">&=
agrave;</span><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p></o:p></spa=
n></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49548::DEBUG::2015-02-26 15:42:26,692::__init__::469::jsonrpc.Json=
RpcServer::(_serveRequest) Calling 'VM.migrate' in bridge with {u'params':
=
{u'tunneled': u'false', u'dstqemu': u'172.19.2.31',
 u'src': u'compute04', u'dst': u'compute01:54321',
u'vmId': u'b75823d1-00f0=
-457e-a692-8b95f73907db', u'abortOnError': u'true', u'method':
u'online'}, =
u'vmID':
u'b75823d1-00f0-457e-a692-8b95f73907db'}<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49548::DEBUG::2015-02-26 15:42:26,694::API::510::vds::(migrate) {u=
'tunneled': u'false', u'dstqemu': u'IPADDR',
u'src': u'compute04', u'dst': =
u'compute01:54321', u'vmId':
u'b75823d1-00f0-457e-a692-8b95f73907db',
 u'abortOnError': u'true', u'method':
u'online'}<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49549::DEBUG::2015-02-26 15:42:26,699::migration::103::vm.Vm::(_se=
tupVdsConnection) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Destinatio=
n server is: compute01:54321<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49549::DEBUG::2015-02-26 15:42:26,702::migration::105::vm.Vm::(_se=
tupVdsConnection) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Initiating=
 connection with destination<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49549::DEBUG::2015-02-26 15:42:26,733::migration::155::vm.Vm::(_pr=
epareGuest) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration starte=
d<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49549::DEBUG::2015-02-26 15:42:26,755::migration::238::vm.Vm::(run=
) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::migration semaphore acquir=
ed after 0 seconds<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49549::DEBUG::2015-02-26 15:42:27,211::migration::298::vm.Vm::(_st=
artUnderlyingMigration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::star=
ting migration to qemu&#43;tls://compute01/system
 with miguri tcp://IPADDR<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49550::DEBUG::2015-02-26 15:42:27,213::migration::361::vm.Vm::(run=
) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::migration downtime thread =
started<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::DEBUG::2015-02-26 15:42:27,216::migration::410::vm.Vm::(mon=
itor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::starting mig=
ration monitor thread<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49550::DEBUG::2015-02-26 15:43:42,218::migration::370::vm.Vm::(run=
) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime=
 to 50<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49550::DEBUG::2015-02-26 15:44:57,222::migration::370::vm.Vm::(run=
) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime=
 to 100<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49550::DEBUG::2015-02-26 15:46:12,227::migration::370::vm.Vm::(run=
) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime=
 to 150<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:47:07,279::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (1791MiB) &gt; lowmark
 (203MiB). Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:47:17,281::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (1398MiB) &gt; lowmark
 (203MiB). Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49550::DEBUG::2015-02-26 15:47:27,233::migration::370::vm.Vm::(run=
) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime=
 to 200<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:47:27,283::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (1066MiB) &gt; lowmark
 (203MiB). Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:47:37,285::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (701MiB) &gt; lowmark (203MiB).
 Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:47:47,287::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (361MiB) &gt; lowmark (203MiB).
 Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:48:07,291::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (683MiB) &gt; lowmark (13MiB).
 Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:48:17,292::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (350MiB) &gt; lowmark (13MiB).
 Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:48:27,294::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (18MiB) &gt; lowmark (13MiB).
 Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:48:37,296::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (646MiB) &gt; lowmark (13MiB).
 Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49550::DEBUG::2015-02-26 15:48:42,238::migration::370::vm.Vm::(run=
) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime=
 to 250<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:48:47,334::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (317MiB) &gt; lowmark (13MiB).
 Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:48:57,342::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (1018MiB) &gt; lowmark
 (13MiB). Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:49:07,344::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (679MiB) &gt; lowmark (13MiB).
 Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:49:17,346::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (357MiB) &gt; lowmark (13MiB).
 Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:49:27,348::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (31MiB) &gt; lowmark (13MiB).
 Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:49:37,349::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (854MiB) &gt; lowmark (13MiB).
 Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:49:47,351::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (525MiB) &gt; lowmark (13MiB).
 Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49550::DEBUG::2015-02-26 15:49:57,242::migration::370::vm.Vm::(run=
) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime=
 to 300<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:49:57,353::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (183MiB) &gt; lowmark (13MiB).
 Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:50:07,355::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (785MiB) &gt; lowmark (13MiB).
 Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:50:17,357::migration::458::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
stalling: remaining (457MiB) &gt; lowmark (13MiB).
 Refer to RHBZ#919201.<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::WARNING::2015-02-26 15:50:27,359::migration::445::vm.Vm::(m=
onitor_migration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Migration =
is stuck: Hasn't progressed in 150.069458961
 seconds. Aborting.<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49551::DEBUG::2015-02-26 15:50:27,362::migration::470::vm.Vm::(sto=
p) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::stopping migration monito=
r thread<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49549::DEBUG::2015-02-26 15:50:27,852::migration::376::vm.Vm::(can=
cel) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::canceling migration dow=
ntime thread<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49549::DEBUG::2015-02-26 15:50:27,852::migration::470::vm.Vm::(sto=
p) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::stopping migration monito=
r thread<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49550::DEBUG::2015-02-26 15:50:27,853::migration::373::vm.Vm::(run=
) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::migration downtime thread =
exiting<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49549::ERROR::2015-02-26 15:50:27,855::migration::161::vm.Vm::(_re=
cover) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::operation aborted: mi=
gration job: canceled by client<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-49549::ERROR::2015-02-26 15:50:28,049::migration::260::vm.Vm::(run=
) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Failed to migrate<o:p></o:=
p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-49690::DEBUG::2015-02-26 15:50:33,284::__init__::469::jsonrpc.Json=
RpcServer::(_serveRequest) Calling 'VM.getMigrationStatus' in bridge with {=
u'vmID':
u'b75823d1-00f0-457e-a692-8b95f73907db'}<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"font-family:Wingdings;=
color:#1F497D">&szlig;</span><span lang=3D"EN-US"
style=3D"color:#1F497D">s=
nip
</span><span lang=3D"EN-US"
style=3D"font-family:Wingdings;color:#1F497D">&=
agrave;</span><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p></o:p></spa=
n></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">At the =
same time on the destination server<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"font-family:Wingdings;=
color:#1F497D">&szlig;</span><span lang=3D"EN-US"
style=3D"color:#1F497D"> =
snip
</span><span lang=3D"EN-US"
style=3D"font-family:Wingdings;color:#1F497D">&=
agrave;</span><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p></o:p></spa=
n></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-350501::DEBUG::2015-02-26 15:42:25,923::BindingXMLRPC::1133::vds::=
(wrapper) client [172.19.2.34]::call vmGetStats with ('b75823d1-00f0-457e-a=
692-8b95f73907db',) {}<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-350502::DEBUG::2015-02-26 15:42:26,018::BindingXMLRPC::1133::vds::=
(wrapper) client [172.19.2.34]::call vmMigrationCreate with ALL THE MACHINE=
 START PARAMETERS here<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-350502::INFO::2015-02-26 15:42:26,019::clientIF::394::vds::(create=
Vm) vmContainerLock acquired by vm b75823d1-00f0-457e-a692-8b95f73907db<o:p=
></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-350502::DEBUG::2015-02-26 15:42:26,033::clientIF::407::vds::(creat=
eVm) Total desktops after creation of b75823d1-00f0-457e-a692-8b95f73907db =
is 3<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-350503::DEBUG::2015-02-26 15:42:26,033::vm::2264::vm.Vm::(_startUn=
derlyingVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Start<o:p></o:p>=
</span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-350502::DEBUG::2015-02-26 15:42:26,036::vm::5658::vm.Vm::(waitForM=
igrationDestinationPrepare) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::=
migration destination: waiting for VM
creation<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-350503::DEBUG::2015-02-26 15:42:26,036::vm::2268::vm.Vm::(_startUn=
derlyingVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::_ongoingCreation=
s acquired<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-350502::DEBUG::2015-02-26 15:42:26,038::vm::5663::vm.Vm::(waitForM=
igrationDestinationPrepare) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::=
migration destination: waiting 48s for path
 preparation<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-350503::INFO::2015-02-26 15:42:26,038::vm::3261::vm.Vm::(_run) vmI=
d=3D`b75823d1-00f0-457e-a692-8b95f73907db`::VM wrapper has started<o:p></o:=
p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-350503::WARNING::2015-02-26 15:42:26,041::vm::2056::vm.Vm::(buildC=
onfDevices) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Unknown type fou=
nd, device: '{'device': 'unix', 'alias':
 'channel0', 'type': 'channel', 'address': {'bus':
'0', 'controller': '0', =
'type': 'virtio-serial', 'port': '1'}}'
found<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-350503::WARNING::2015-02-26 15:42:26,041::vm::2056::vm.Vm::(buildC=
onfDevices) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Unknown type fou=
nd, device: '{'device': 'unix', 'alias':
 'channel1', 'type': 'channel', 'address': {'bus':
'0', 'controller': '0', =
'type': 'virtio-serial', 'port': '2'}}'
found<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-350503::DEBUG::2015-02-26 15:42:26,349::vm::1058::vm.Vm::(__init__=
) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Ignoring param (target, 10=
485760) in BalloonDevice<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-350503::DEBUG::2015-02-26 15:42:26,350::vm::2294::vm.Vm::(_startUn=
derlyingVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::_ongoingCreation=
s released<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-350502::ERROR::2015-02-26 15:42:26,351::vm::5638::vm.Vm::(_updateD=
evicesDomxmlCache) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Alias not=
 found for device type graphics during migration
 at destination host<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-350503::DEBUG::2015-02-26 15:42:26,353::vm::4128::vm.Vm::(_waitFor=
UnderlyingMigration) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Waiting=
 21600 seconds for end of migration<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-350502::DEBUG::2015-02-26 15:42:26,377::BindingXMLRPC::1140::vds::=
(wrapper) return vmMigrationCreate with {ALL THE MACHINE PARAMETERS HERE<o:=
p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:libvirtEventLoop::DEBUG::2015-02-26 15:42:27,195::vm::5571::vm.Vm::(_onLi=
bvirtLifecycleEvent) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::event S=
tarted detail 1 opaque None<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:libvirtEventLoop::DEBUG::2015-02-26 15:50:27,037::vm::5571::vm.Vm::(_onLi=
bvirtLifecycleEvent) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::event S=
topped detail 5 opaque None<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:libvirtEventLoop::INFO::2015-02-26 15:50:27,038::vm::2366::vm.Vm::(_onQem=
uDeath) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::underlying process d=
isconnected<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:libvirtEventLoop::INFO::2015-02-26 15:50:27,038::vm::4952::vm.Vm::(releas=
eVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Release VM resources<o:=
p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-350503::DEBUG::2015-02-26 15:50:27,044::libvirtconnection::143::ro=
ot::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Do=
main not found: no domain with matching
 uuid
'b75823d1-00f0-457e-a692-8b95f73907db'<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-350503::ERROR::2015-02-26 15:50:27,047::vm::2325::vm.Vm::(_startUn=
derlyingVm) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Failed to start =
a migration destination vm<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:MigrationError: Domain not found: no domain with matching uuid 'b75823d1-=
00f0-457e-a692-8b95f73907db'<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-350503::DEBUG::2015-02-26 15:50:27,058::vm::2786::vm.Vm::(setDownS=
tatus) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Changed state to Down=
: VM failed to migrate (code=3D8)<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-351517::DEBUG::2015-02-26 15:50:27,089::BindingXMLRPC::1133::vds::=
(wrapper) client [172.19.2.34]::call vmDestroy with ('b75823d1-00f0-457e-a6=
92-8b95f73907db',) {}<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-351517::INFO::2015-02-26 15:50:27,092::API::332::vds::(destroy) vm=
ContainerLock acquired by vm b75823d1-00f0-457e-a692-8b95f73907db<o:p></o:p=
></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:Thread-351517::DEBUG::2015-02-26 15:50:27,154::vm::5026::vm.Vm::(destroy)=
 vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::destroy
Called<o:p></o:p></=
span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:libvirtEventLoop::WARNING::2015-02-26 15:50:27,162::utils::129::root::(rm=
File) File: /var/lib/libvirt/qemu/channels/b75823d1-00f0-457e-a692-8b95f739=
07db.com.redhat.rhevm.vdsm already removed<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:libvirtEventLoop::WARNING::2015-02-26 15:50:27,163::utils::129::root::(rm=
File) File: /var/lib/libvirt/qemu/channels/b75823d1-00f0-457e-a692-8b95f739=
07db.org.qemu.guest_agent.0 already removed<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:libvirtEventLoop::INFO::2015-02-26 15:50:27,164::logUtils::44::dispatcher=
::(wrapper) Run and protect: inappropriateDevices(thiefId=3D'b75823d1-00f0-=
457e-a692-8b95f73907db')<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">vdsm.lo=
g:Thread-351517::DEBUG::2015-02-26 15:50:27,170::vm::5020::vm.Vm::(deleteVm=
) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::Total desktops after destr=
oy of b75823d1-00f0-457e-a692-8b95f73907db
 is 2<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">vdsm.lo=
g:libvirtEventLoop::WARNING::2015-02-26 15:50:27,170::vm::1953::vm.Vm::(_se=
t_lastStatus) vmId=3D`b75823d1-00f0-457e-a692-8b95f73907db`::trying to set =
state to Down when already Down<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"font-family:Wingdings;=
color:#1F497D">&szlig;</span><span lang=3D"EN-US"
style=3D"color:#1F497D"> =
snip
</span><span lang=3D"EN-US"
style=3D"font-family:Wingdings;color:#1F497D">&=
agrave;</span><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p></o:p></spa=
n></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">And I c=
an see the error &#8220;Domain not found: no domain with matching uuid 'b75=
823d1-00f0-457e-a692-8b95f73907db'&#8221; in the middle, but I am not entir=
ely sure what this means.
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">In the =
qemu log I can see this<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"font-family:Wingdings;=
color:#1F497D">&szlig;</span><span lang=3D"EN-US"
style=3D"color:#1F497D"> =
snip
</span><span lang=3D"EN-US"
style=3D"font-family:Wingdings;color:#1F497D">&=
agrave;</span><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p></o:p></spa=
n></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><b><span lang=3D"EN-US"
style=3D"font-size:12.0pt;co=
lor:#1F497D">From QEMU LOG<o:p></o:p></span></b></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">2015-02=
-26 14:42:26.859&#43;0000: starting up<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">Domain =
id=3D8 is tainted: hook-script<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">2015-02=
-26T14:50:26.650595Z qemu-system-x86_64: load of migration failed: Input/ou=
tput error<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">2015-02=
-26 14:50:26.684&#43;0000: shutting down<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"font-family:Wingdings;=
color:#1F497D">&szlig;</span><span lang=3D"EN-US"
style=3D"color:#1F497D"> =
snip
</span><span lang=3D"EN-US"
style=3D"font-family:Wingdings;color:#1F497D">&=
agrave;</span><span lang=3D"EN-US"
style=3D"color:#1F497D"...

<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">the tim=
e here seems to be logged in GMT and in the snippets above it is CET &#8211=
; might this be the problem ???<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">Here is=
 the qemu commandline (that was removed above in cleaned
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"font-family:Wingdings;=
color:#1F497D">&szlig;</span><span
style=3D"color:#1F497D"> snip
</span><span lang=3D"EN-US"
style=3D"font-family:Wingdings;color:#1F497D">&=
agrave;</span><span
style=3D"color:#1F497D"><o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
style=3D"color:#1F497D">/usr/bin/qemu-kvm <o:p=
></o:p></span></p...
 <p
class=3D"MsoNormal"><span style=3D"color:#1F497D">-name
SERVERNAME <o:p>=
</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-S <o:p=
></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-machin=
e pc-1.0,accel=3Dkvm,usb=3Doff
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-cpu Sa=
ndyBridge <o:p...

</o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">-m 1024=
0 <o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-realti=
me mlock=3Doff
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-smp 4,=
maxcpus=3D64,sockets=3D16,cores=3D4,threads=3D1
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-uuid b=
75823d1-00f0-457e-a692-8b95f73907db
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-smbios=
 type=3D1,manufacturer=3DoVirt,product=3DoVirt Node,version=3D20-3,serial=
=3D4C4C4544-0038-3210-8048-B7C04F303232,uuid=3Db75823d1-00f0-457e-a692-8b95=
f73907db
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-no-use=
r-config <o:p...

</o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">-nodefa=
ults <o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-charde=
v socket,id=3Dcharmonitor,path=3D/var/lib/libvirt/qemu/al-exchange-01.monit=
or,server,nowait
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-mon ch=
ardev=3Dcharmonitor,id=3Dmonitor,mode=3Dcontrol
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-rtc ba=
se=3D2015-02-26T15:42:26,driftfix=3Dslew
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-global=
 kvm-pit.lost_tick_policy=3Ddiscard
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-no-hpe=
t <o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-no-shu=
tdown <o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-boot s=
trict=3Don <o:p...

</o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">-device=
 piix3-usb-uhci,id=3Dusb,bus=3Dpci.0,addr=3D0x1.0x2
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-device=
 virtio-scsi-pci,id=3Dscsi0,bus=3Dpci.0,addr=3D0x4
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-device=
 virtio-serial-pci,id=3Dvirtio-serial0,max_ports=3D16,bus=3Dpci.0,addr=3D0x=
5
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-drive =
file=3D/rhev/data-center/mnt/XXXX:_data_export_iso/85821683-bc7c-429b-bff2-=
ce25f56fc16f/images/11111111-1111-1111-1111-111111111111/XXXX.iso,if=3Dnone=
,id=3Ddrive-ide0-1-0,readonly=3Don,format=3Draw,serial=3D
 -device ide-cd,bus=3Dide.1,unit=3D0,drive=3Ddrive-ide0-1-0,id=3Dide0-1-0,b=
ootindex=3D2 <o:p...

</o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">-drive =
file=3D/rhev/data-center/0f954891-b1cd-4f09-99ae-75d404d95f9d/276e9ba7-e19a=
-49c5-8ad7-26711934d5e4/images/c37bfa94-718c-4125-9202-bc299535eca5/299bd6d=
1-f4a0-4296-9944-176b2252a886,if=3Dnone,id=3Ddrive-virtio-disk0,format=3Dra=
w,serial=3Dc37bfa94-718c-4125-9202-bc299535eca5,cache=3Dnone,werror=3Dstop,=
rerror=3Dstop,aio=3Dthreads
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-device=
 virtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x6,drive=3Ddrive-virtio-disk=
0,id=3Dvirtio-disk0,bootindex=3D1
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-drive =
file=3D/rhev/data-center/0f954891-b1cd-4f09-99ae-75d404d95f9d/276e9ba7-e19a=
-49c5-8ad7-26711934d5e4/images/75134ccc-b74e-4955-90a5-95d4ceff403b/1e5b5b8=
6-c31d-476c-acb2-a5bd6a65490b,if=3Dnone,id=3Ddrive-virtio-disk1,format=3Dra=
w,serial=3D75134ccc-b74e-4955-90a5-95d4ceff403b,cache=3Dnone,werror=3Dstop,=
rerror=3Dstop,aio=3Dthreads
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-device=
 virtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x8,drive=3Ddrive-virtio-disk=
1,id=3Dvirtio-disk1
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-netdev=
 tap,fd=3D30,id=3Dhostnet0,vhost=3Don,vhostfd=3D31
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-device=
 virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D00:1a:4a:c7:72:0a,bus=3Dp=
ci.0,addr=3D0x3
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-charde=
v socket,id=3Dcharchannel0,path=3D/var/lib/libvirt/qemu/channels/b75823d1-0=
0f0-457e-a692-8b95f73907db.com.redhat.rhevm.vdsm,server,nowait
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-device=
 virtserialport,bus=3Dvirtio-serial0.0,nr=3D1,chardev=3Dcharchannel0,id=3Dc=
hannel0,name=3Dcom.redhat.rhevm.vdsm
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-charde=
v socket,id=3Dcharchannel1,path=3D/var/lib/libvirt/qemu/channels/b75823d1-0=
0f0-457e-a692-8b95f73907db.org.qemu.guest_agent.0,server,nowait
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-device=
 virtserialport,bus=3Dvirtio-serial0.0,nr=3D2,chardev=3Dcharchannel1,id=3Dc=
hannel1,name=3Dorg.qemu.guest_agent.0
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-device=
 usb-tablet,id=3Dinput0
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-vnc IP=
-ADDRESS:2,password
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-k de <=
o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-device=
 cirrus-vga,id=3Dvideo0,bus=3Dpci.0,addr=3D0x2
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-incomi=
ng tcp:[::]:49152
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-device=
 virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x7
<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">-msg ti=
mestamp=3Don<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"font-family:Wingdings;=
color:#1F497D">&szlig;</span><span lang=3D"EN-US"
style=3D"color:#1F497D"> =
snip
</span><span lang=3D"EN-US"
style=3D"font-family:Wingdings;color:#1F497D">&=
agrave;</span><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p></o:p></spa=
n></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">Here is=
 the content of the .meta file in the directories of the 2 images above<o:p=
></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">&lt;--s=
nip--&gt; <o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">DOMAIN=
=3D276e9ba7-e19a-49c5-8ad7-26711934d5e4<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">VOLTYPE=
=3DLEAF<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">CTIME=
=3D1424445378<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">FORMAT=
=3DRAW<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">IMAGE=
=3Dc37bfa94-718c-4125-9202-bc299535eca5<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">DISKTYP=
E=3D2<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">PUUID=
=3D00000000-0000-0000-0000-000000000000<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">LEGALIT=
Y=3DLEGAL<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">MTIME=
=3D0<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">POOL_UU=
ID=3D<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">DESCRIP=
TION=3D{&quot;DiskAlias&quot;:&quot;al-exchange-01_Disk1&quot;,&quot;DiskDe=
scription&quot;:&quot;System&quot;}<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">TYPE=3D=
SPARSE<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">SIZE=3D=
146800640<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">EOF<o:p=
></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">&lt;--s=
nip--&gt;<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">&lt;--s=
nip--&gt; <o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">DOMAIN=
=3D276e9ba7-e19a-49c5-8ad7-26711934d5e4<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">CTIME=
=3D1424785898<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">FORMAT=
=3DRAW<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">DISKTYP=
E=3D2<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">LEGALIT=
Y=3DLEGAL<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">SIZE=3D=
419430400<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">VOLTYPE=
=3DLEAF<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">DESCRIP=
TION=3D{&quot;DiskAlias&quot;:&quot;al-exchange-01_Disk2&quot;,&quot;DiskDe=
scription&quot;:&quot;Data&quot;}<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">IMAGE=
=3D75134ccc-b74e-4955-90a5-95d4ceff403b<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">PUUID=
=3D00000000-0000-0000-0000-000000000000<o:p></o:p></span></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">MTIME=
=3D0<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">POOL_UU=
ID=3D<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">TYPE=3D=
SPARSE<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">EOF<o:p=
></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">&lt;--s=
nip--&gt;<o:p></o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p...
 <p
class=3D"MsoNormal"><a name=3D"_MailEndCompose"><span
lang=3D"EN-US" sty=
le=3D"color:#1F497D"><o:p>&nbsp;</o:p></span></a></p...

<div...
 <div
style=3D"border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in =
0in 0in"...
 <p
class=3D"MsoNormal"><b><span lang=3D"EN-US"
style=3D"color:windowtext;ms=
o-fareast-language:DE">From:</span></b><span
lang=3D"EN-US" style=3D"color:=
windowtext;mso-fareast-language:DE"> Roy Golan [mailto:rgolan@redhat.com]
<br...
 <b>Sent:</b> Wednesday, February 18, 2015 12:12
PM<br...
 <b>To:</b> Soeren
Malchow; users(a)ovirt.org&lt;br...
 <b>Subject:</b> Re:
[ovirt-users] Hosted Engine Migration fails<o:p></o:p><=
/span></p...

</div...

</div...
 <p
class=3D"MsoNormal"><o:p>&nbsp;</o:p></p...

<div...
 <p
class=3D"MsoNormal">On 02/16/2015 04:55 AM, Soeren Malchow wrote:<span s=
tyle=3D"font-size:12.0pt;mso-fareast-language:DE"><o:p></o:p></span></p...

</div...

<blockquote style=3D"margin-top:5.0pt;margin-bottom:5.0pt"...
 <p class=3D"MsoNormal">Dear
all,<o:p></o:p></p...
 <p
class=3D"MsoNormal">&nbsp;<o:p></o:p></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US">we ahve a setup with several ho=
sts running fedora 10 with the virt-preview packages installed (for snapsho=
t live merge) and a hosted engine running centos
6.6-</span><o:p></o:p></p...
 <p
class=3D"MsoNormal"><span
lang=3D"EN-US">&nbsp;</span><o:p></o:p></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US">We are experiencing a problem w=
ith the Live Migration of the Hosted Engine, in the case of setting the hos=
t for the Engine into maintenance as well as a manual migration.</span><o:p=
></o:p></p...
 <p
class=3D"MsoNormal"><span lang=3D"EN-US">I tried this on the
&#8220;ovir=
tmgmt&#8221; network and when that failed I did some research and tried to =
use another network interface (separate from ovirtmgmt on layer 2), this al=
so fails.</span><o:p></o:p></p...
 <p
class=3D"MsoNormal"><span
lang=3D"EN-US">&nbsp;</span><o:p></o:p></p...
 <p class=3D"MsoNormal"><span
lang=3D"EN-US">It looks as if the migration is=
 still going through the ovirtmgmt interface, at least judging from the net=
work traffic, and I think the error that I found (RHBZ#919201) is actually =
the right one.
</span><o:p></o:p></p...
 <p
class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto"><span lang=3D"EN-US"
style=3D"font-size:12.0pt">&nbsp;--</span><o:=
p></o:p></p...
 <p class=3D"MsoNormal"
style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto"><span lang=3D"EN-US"
style=3D"font-size:12.0pt">vdsm.log:Thread-67=
45::WARNING::2015-02-16 03:22:25,743::migration::458::vm.Vm::(monitor_migra=
tion) vmId=3D`XXX`::Migration stalling: remaining
 (35MiB) &gt; lowmark (15MiB). Refer to
RHBZ#919201.</span><o:p></o:p></p...
 <p
class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto"><span lang=3D"EN-US"
style=3D"font-size:12.0pt">vdsm.log:Thread-67=
45::WARNING::2015-02-16 03:22:35,745::migration::458::vm.Vm::(monitor_migra=
tion) vmId=3D`XXX`::Migration stalling: remaining
 (129MiB) &gt; lowmark (15MiB). Refer to
RHBZ#919201.</span><o:p></o:p></p...
 <p
class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto"><span lang=3D"EN-US"
style=3D"font-size:12.0pt">vdsm.log:Thread-67=
45::WARNING::2015-02-16 03:22:45,747::migration::458::vm.Vm::(monitor_migra=
tion) vmId=3D`XXX`::Migration stalling: remaining
 (42MiB) &gt; lowmark (15MiB). Refer to
RHBZ#919201.</span><o:p></o:p></p...
 <p
class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto"><span lang=3D"EN-US"
style=3D"font-size:12.0pt">vdsm.log:Thread-67=
45::WARNING::2015-02-16 03:22:55,749::migration::458::vm.Vm::(monitor_migra=
tion) vmId=3D`XXX`::Migration stalling: remaining
 (88MiB) &gt; lowmark (15MiB). Refer to
RHBZ#919201.</span><o:p></o:p></p...
 <p
class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto"><span lang=3D"EN-US"
style=3D"font-size:12.0pt">--</span><o:p></o:=
p></p...
 <p class=3D"MsoNormal"
style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto"><span lang=3D"EN-US"
style=3D"font-size:12.0pt">The ovirtmgmt inte=
rface is 2 x 1Gbit (LACP connected to Dell Switches with MLAG) and by far n=
ot fully utilized.</span><o:p></o:p></p...
 <p
class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto"><span lang=3D"EN-US"
style=3D"font-size:12.0pt">Can anyone help me=
 where to go form here ?</span><o:p></o:p></p...

</blockquote...
 <br...
 still need the whole log. probably the guest (engine) is doing
lots of memo=
ry i/o if you got a fair amount of running VMs and Hosts.&nbsp; that will s=
talls the migration because
<br...
 the guest pages are getting dirty faster than qemu can
copy.<br...

<br...

<br...

you have 2 options:<br...

<br...

1. try several more times.<br...

<br...
 2. shutdown the engine vm, it should
start on another host<br...

<br...

<br...

<o:p></o:p></span></p...

<blockquote style=3D"margin-top:5.0pt;margin-bottom:5.0pt"...
 <p class=3D"MsoNormal"
style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto"><span lang=3D"EN-US"
style=3D"font-size:12.0pt">Regards</span><o:p=
></o:p></p...
 <p class=3D"MsoNormal"
style=3D"mso-margin-top-alt:auto;mso-margin-bottom-a=
lt:auto"><span lang=3D"EN-US"
style=3D"font-size:12.0pt">Soeren
</span><o:p></o:p></p...
 <p
class=3D"MsoNormal"><span
style=3D"font-size:12.0pt;font-family:&quot;Ti=
mes New Roman&quot;,serif;mso-fareast-language:DE"><br...

<br...

<br...

<o:p></o:p></span></p...

<pre>_______________________________________________<o:p></o:p></pre...
 <pre>Users mailing
list<o:p></o:p></pre...

<pre><a
href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><o:p></o:p></pre=
...
 <pre><a
href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
.ovirt.org/mailman/listinfo/users</a><o:p></o:p></pre...

</blockquote...
 <o:p>&nbsp;</o:p>=
</p...

</div...

</body...

</html...

--_000_687766E0007010429FBFF98CA70824888A0C30mcexch01mcongroup_--

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [ovirt-users] Hosted Engine Migration fails