<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:8.5in 11.0in;
margin:70.85pt 70.85pt 56.7pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoPlainText">Dear Doron,<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><span lang="EN-US">thanks for the answer, i rechecked again:<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Network on Source host BEFORE migration<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="mso-fareast-language:DE"><img width="1106" height="655" id="Picture_x0020_1" src="cid:image002.jpg@01D05212.E14A0D40"></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Network on target host before migration<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="mso-fareast-language:DE"><img width="1101" height="641" id="Picture_x0020_2" src="cid:image004.jpg@01D05212.E14A0D40"></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">The same hosts during migration look like this<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Source<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="mso-fareast-language:DE"><img width="1120" height="660" id="Picture_x0020_3" src="cid:image008.jpg@01D05212.E14A0D40"></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Target<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="mso-fareast-language:DE"><img width="1108" height="641" id="Picture_x0020_4" src="cid:image010.jpg@01D05212.E14A0D40"></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoPlainText"><a name="_MailEndCompose"><span lang="EN-US"><o:p> </o:p></span></a></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">The migration is not even using full network speed. It is using only about 25 – 30% of the available bandwidth.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">I tried with one of our test machines, and I had one that was working (a CentOS machine doing nothing) however, the other machine that did not work was a Windows machine but it was also doing nothing.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Here is a screenshot of an SCP copy between the 2 machines – source -> target<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="mso-fareast-language:DE"><img width="1537" height="166" id="Picture_x0020_6" src="cid:image015.png@01D05212.E14A0D40"></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">One thing I can see however is, that at somepoint the network traffic drops back to a few KB/s even though the migration is not finished and shortly after that the migration stops unsuccessfully.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">The ovirtmgmt network hosts only the engine and is used for migration, the storage and VM networks are completely separated. The environment is not in production yet.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Regards<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Soeren <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US" style="mso-fareast-language:DE">-----Original Message-----<br>
From: Doron Fediuck [mailto:dfediuck@redhat.com] <br>
Sent: Thursday, February 26, 2015 5:46 PM<br>
To: Soeren Malchow; Roy Golan; users@ovirt.org<br>
Cc: Dan Kenigsberg; Omer Frenkel<br>
Subject: Re: [ovirt-users] Hosted Engine Migration fails</span></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">On 26/02/15 17:31, Soeren Malchow wrote:<o:p></o:p></p>
<p class="MsoPlainText">> Hi,<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> we tried this (Roys Mail below) , and yes, shutdown always works.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> But now this problem comes up with regular machines now as well.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> The environment is setup like this<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Engine: Ovirt 3.5.1.1-1.el6 – on CentOS6<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> The Storage Backend is gluster 3.6.2-1.el7 on CentOS 7<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Compute hosts: Libvirt Version: libvirt-1.2.9.1-2.fc20, kvm 2.1.2 -
<o:p></o:p></p>
<p class="MsoPlainText">> 7.fc20, vdsm vdsm-4.16.10-8.gitc937927.fc20<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> All compute Servers are 100% identical.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> The storage cluster was tested manually and works just fine.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> The network interfaces are not fully utilized, more like 15%.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Log output ist as below. The thing in the log output I do not
<o:p></o:p></p>
<p class="MsoPlainText">> understand is this<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> “2015-02-26T14:50:26.650595Z qemu-system-x86_64: load of migration<o:p></o:p></p>
<p class="MsoPlainText">> failed: Input/output error”<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> From the qemu log.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Also if I shut down machines, put their host into maintenance and
<o:p></o:p></p>
<p class="MsoPlainText">> start them somewhere else, everything works just fine.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Can someone help with this ? Any idea where to look ?<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Regards<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Soeren<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> *From VDSM Log*<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> I just tried to migrate a machine, this here happens on the source<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> ßsnip à<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49548::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,692::__init__::469::jsonrpc.JsonRpcServer::(_serveRequest)<o:p></o:p></p>
<p class="MsoPlainText">> Calling 'VM.migrate' in bridge with {u'params': {u'tunneled':
<o:p></o:p></p>
<p class="MsoPlainText">> u'false',<o:p></o:p></p>
<p class="MsoPlainText">> u'dstqemu': u'172.19.2.31', u'src': u'compute04', u'dst':<o:p></o:p></p>
<p class="MsoPlainText">> u'compute01:54321', u'vmId': u'b75823d1-00f0-457e-a692-8b95f73907db',<o:p></o:p></p>
<p class="MsoPlainText">> u'abortOnError': u'true', u'method': u'online'}, u'vmID':<o:p></o:p></p>
<p class="MsoPlainText">> u'b75823d1-00f0-457e-a692-8b95f73907db'}<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49548::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,694::API::510::vds::(migrate) {u'tunneled': u'false',<o:p></o:p></p>
<p class="MsoPlainText">> u'dstqemu': u'IPADDR', u'src': u'compute04', u'dst': <o:p>
</o:p></p>
<p class="MsoPlainText">> u'compute01:54321',<o:p></o:p></p>
<p class="MsoPlainText">> u'vmId': u'b75823d1-00f0-457e-a692-8b95f73907db', u'abortOnError':<o:p></o:p></p>
<p class="MsoPlainText">> u'true', u'method': u'online'}<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49549::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,699::migration::103::vm.Vm::(_setupVdsConnection)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Destination server is:<o:p></o:p></p>
<p class="MsoPlainText">> compute01:54321<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49549::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,702::migration::105::vm.Vm::(_setupVdsConnection)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Initiating connection
<o:p></o:p></p>
<p class="MsoPlainText">> with destination<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49549::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,733::migration::155::vm.Vm::(_prepareGuest)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration started<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49549::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,755::migration::238::vm.Vm::(run)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::migration semaphore
<o:p></o:p></p>
<p class="MsoPlainText">> acquired after 0 seconds<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49549::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:27,211::migration::298::vm.Vm::(_startUnderlyingMigration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::starting migration to<o:p></o:p></p>
<p class="MsoPlainText">> qemu+tls://compute01/system with miguri tcp://IPADDR<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49550::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:27,213::migration::361::vm.Vm::(run)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::migration downtime thread
<o:p></o:p></p>
<p class="MsoPlainText">> started<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:27,216::migration::410::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::starting migration
<o:p></o:p></p>
<p class="MsoPlainText">> monitor thread<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49550::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:43:42,218::migration::370::vm.Vm::(run)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration
<o:p></o:p></p>
<p class="MsoPlainText">> downtime to 50<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49550::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:44:57,222::migration::370::vm.Vm::(run)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration
<o:p></o:p></p>
<p class="MsoPlainText">> downtime to 100<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49550::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:46:12,227::migration::370::vm.Vm::(run)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration
<o:p></o:p></p>
<p class="MsoPlainText">> downtime to 150<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:47:07,279::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (1791MiB) > lowmark (203MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:47:17,281::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (1398MiB) > lowmark (203MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49550::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:47:27,233::migration::370::vm.Vm::(run)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration
<o:p></o:p></p>
<p class="MsoPlainText">> downtime to 200<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:47:27,283::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (1066MiB) > lowmark (203MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:47:37,285::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (701MiB) > lowmark (203MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:47:47,287::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (361MiB) > lowmark (203MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:48:07,291::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (683MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:48:17,292::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (350MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:48:27,294::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (18MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:48:37,296::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (646MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49550::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:48:42,238::migration::370::vm.Vm::(run)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration
<o:p></o:p></p>
<p class="MsoPlainText">> downtime to 250<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:48:47,334::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (317MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:48:57,342::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (1018MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:49:07,344::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (679MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:49:17,346::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (357MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:49:27,348::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (31MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:49:37,349::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (854MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:49:47,351::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (525MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49550::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:49:57,242::migration::370::vm.Vm::(run)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration
<o:p></o:p></p>
<p class="MsoPlainText">> downtime to 300<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:49:57,353::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (183MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:07,355::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (785MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:17,357::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling:<o:p></o:p></p>
<p class="MsoPlainText">> remaining (457MiB) > lowmark (13MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,359::migration::445::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration is stuck:
<o:p></o:p></p>
<p class="MsoPlainText">> Hasn't progressed in 150.069458961 seconds. Aborting.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49551::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,362::migration::470::vm.Vm::(stop)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::stopping migration
<o:p></o:p></p>
<p class="MsoPlainText">> monitor thread<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49549::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,852::migration::376::vm.Vm::(cancel)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::canceling migration
<o:p></o:p></p>
<p class="MsoPlainText">> downtime thread<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49549::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,852::migration::470::vm.Vm::(stop)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::stopping migration
<o:p></o:p></p>
<p class="MsoPlainText">> monitor thread<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49550::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,853::migration::373::vm.Vm::(run)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::migration downtime thread
<o:p></o:p></p>
<p class="MsoPlainText">> exiting<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49549::ERROR::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,855::migration::161::vm.Vm::(_recover)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::operation aborted:<o:p></o:p></p>
<p class="MsoPlainText">> migration job: canceled by client<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49549::ERROR::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:28,049::migration::260::vm.Vm::(run)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Failed to migrate<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-49690::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:33,284::__init__::469::jsonrpc.JsonRpcServer::(_serveRequest)<o:p></o:p></p>
<p class="MsoPlainText">> Calling 'VM.getMigrationStatus' in bridge with {u'vmID':<o:p></o:p></p>
<p class="MsoPlainText">> u'b75823d1-00f0-457e-a692-8b95f73907db'}<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> ßsnip à<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> At the same time on the destination server<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> ßsnip à<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350501::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:25,923::BindingXMLRPC::1133::vds::(wrapper) client
<o:p></o:p></p>
<p class="MsoPlainText">> [172.19.2.34]::call vmGetStats with<o:p></o:p></p>
<p class="MsoPlainText">> ('b75823d1-00f0-457e-a692-8b95f73907db',) {}<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350502::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,018::BindingXMLRPC::1133::vds::(wrapper) client
<o:p></o:p></p>
<p class="MsoPlainText">> [172.19.2.34]::call vmMigrationCreate with ALL THE MACHINE START
<o:p></o:p></p>
<p class="MsoPlainText">> PARAMETERS here<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350502::INFO::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,019::clientIF::394::vds::(createVm) vmContainerLock acquired
<o:p></o:p></p>
<p class="MsoPlainText">> by vm b75823d1-00f0-457e-a692-8b95f73907db<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350502::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,033::clientIF::407::vds::(createVm) Total desktops after
<o:p></o:p></p>
<p class="MsoPlainText">> creation of b75823d1-00f0-457e-a692-8b95f73907db is 3<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350503::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,033::vm::2264::vm.Vm::(_startUnderlyingVm)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Start<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350502::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,036::vm::5658::vm.Vm::(waitForMigrationDestinationPrepare)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::migration destination:<o:p></o:p></p>
<p class="MsoPlainText">> waiting for VM creation<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350503::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,036::vm::2268::vm.Vm::(_startUnderlyingVm)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::_ongoingCreations
<o:p></o:p></p>
<p class="MsoPlainText">> acquired<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350502::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,038::vm::5663::vm.Vm::(waitForMigrationDestinationPrepare)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::migration destination:<o:p></o:p></p>
<p class="MsoPlainText">> waiting 48s for path preparation<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350503::INFO::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,038::vm::3261::vm.Vm::(_run)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::VM wrapper has started<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350503::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,041::vm::2056::vm.Vm::(buildConfDevices)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Unknown type found, device:<o:p></o:p></p>
<p class="MsoPlainText">> '{'device': 'unix', 'alias': 'channel0', 'type': 'channel', 'address':<o:p></o:p></p>
<p class="MsoPlainText">> {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '1'}}'<o:p></o:p></p>
<p class="MsoPlainText">> found<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350503::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,041::vm::2056::vm.Vm::(buildConfDevices)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Unknown type found, device:<o:p></o:p></p>
<p class="MsoPlainText">> '{'device': 'unix', 'alias': 'channel1', 'type': 'channel', 'address':<o:p></o:p></p>
<p class="MsoPlainText">> {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '2'}}'<o:p></o:p></p>
<p class="MsoPlainText">> found<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350503::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,349::vm::1058::vm.Vm::(__init__)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Ignoring param (target,<o:p></o:p></p>
<p class="MsoPlainText">> 10485760) in BalloonDevice<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350503::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,350::vm::2294::vm.Vm::(_startUnderlyingVm)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::_ongoingCreations
<o:p></o:p></p>
<p class="MsoPlainText">> released<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350502::ERROR::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,351::vm::5638::vm.Vm::(_updateDevicesDomxmlCache)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Alias not found for
<o:p></o:p></p>
<p class="MsoPlainText">> device type graphics during migration at destination host<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350503::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,353::vm::4128::vm.Vm::(_waitForUnderlyingMigration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Waiting 21600 seconds for
<o:p></o:p></p>
<p class="MsoPlainText">> end of migration<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350502::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:26,377::BindingXMLRPC::1140::vds::(wrapper) return
<o:p></o:p></p>
<p class="MsoPlainText">> vmMigrationCreate with {ALL THE MACHINE PARAMETERS HERE<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:libvirtEventLoop::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:42:27,195::vm::5571::vm.Vm::(_onLibvirtLifecycleEvent)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::event Started detail 1
<o:p></o:p></p>
<p class="MsoPlainText">> opaque None<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:libvirtEventLoop::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,037::vm::5571::vm.Vm::(_onLibvirtLifecycleEvent)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::event Stopped detail 5
<o:p></o:p></p>
<p class="MsoPlainText">> opaque None<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:libvirtEventLoop::INFO::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,038::vm::2366::vm.Vm::(_onQemuDeath)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::underlying process
<o:p></o:p></p>
<p class="MsoPlainText">> disconnected<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:libvirtEventLoop::INFO::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,038::vm::4952::vm.Vm::(releaseVm)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Release VM resources<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350503::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,044::libvirtconnection::143::root::(wrapper) Unknown<o:p></o:p></p>
<p class="MsoPlainText">> libvirterror: ecode: 42 edom: 10 level: 2 message: Domain not found:
<o:p></o:p></p>
<p class="MsoPlainText">> no domain with matching uuid 'b75823d1-00f0-457e-a692-8b95f73907db'<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350503::ERROR::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,047::vm::2325::vm.Vm::(_startUnderlyingVm)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Failed to start a
<o:p></o:p></p>
<p class="MsoPlainText">> migration destination vm<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:MigrationError: Domain not found: no domain with matching
<o:p></o:p></p>
<p class="MsoPlainText">> uuid 'b75823d1-00f0-457e-a692-8b95f73907db'<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-350503::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,058::vm::2786::vm.Vm::(setDownStatus)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Changed state to Down: VM
<o:p></o:p></p>
<p class="MsoPlainText">> failed to migrate (code=8)<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-351517::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,089::BindingXMLRPC::1133::vds::(wrapper) client
<o:p></o:p></p>
<p class="MsoPlainText">> [172.19.2.34]::call vmDestroy with<o:p></o:p></p>
<p class="MsoPlainText">> ('b75823d1-00f0-457e-a692-8b95f73907db',) {}<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-351517::INFO::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,092::API::332::vds::(destroy) vmContainerLock acquired by vm
<o:p></o:p></p>
<p class="MsoPlainText">> b75823d1-00f0-457e-a692-8b95f73907db<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-351517::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,154::vm::5026::vm.Vm::(destroy)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::destroy Called<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:libvirtEventLoop::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,162::utils::129::root::(rmFile) File:<o:p></o:p></p>
<p class="MsoPlainText">> /var/lib/libvirt/qemu/channels/b75823d1-00f0-457e-a692-8b95f73907db.co<o:p></o:p></p>
<p class="MsoPlainText">> m.redhat.rhevm.vdsm<o:p></o:p></p>
<p class="MsoPlainText">> already removed<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:libvirtEventLoop::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,163::utils::129::root::(rmFile) File:<o:p></o:p></p>
<p class="MsoPlainText">> /var/lib/libvirt/qemu/channels/b75823d1-00f0-457e-a692-8b95f73907db.or<o:p></o:p></p>
<p class="MsoPlainText">> g.qemu.guest_agent.0<o:p></o:p></p>
<p class="MsoPlainText">> already removed<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:libvirtEventLoop::INFO::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,164::logUtils::44::dispatcher::(wrapper) Run and protect:<o:p></o:p></p>
<p class="MsoPlainText">> inappropriateDevices(thiefId='b75823d1-00f0-457e-a692-8b95f73907db')<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-351517::DEBUG::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,170::vm::5020::vm.Vm::(deleteVm)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Total desktops after
<o:p></o:p></p>
<p class="MsoPlainText">> destroy of b75823d1-00f0-457e-a692-8b95f73907db is 2<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:libvirtEventLoop::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">> 15:50:27,170::vm::1953::vm.Vm::(_set_lastStatus)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::trying to set state to
<o:p></o:p></p>
<p class="MsoPlainText">> Down when already Down<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> ßsnip à<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> And I can see the error “Domain not found: no domain with matching
<o:p></o:p></p>
<p class="MsoPlainText">> uuid 'b75823d1-00f0-457e-a692-8b95f73907db'” in the middle, but I am
<o:p></o:p></p>
<p class="MsoPlainText">> not entirely sure what this means.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> In the qemu log I can see this<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> ßsnip à<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> *From QEMU LOG*<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> 2015-02-26 14:42:26.859+0000: starting up<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Domain id=8 is tainted: hook-script<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> 2015-02-26T14:50:26.650595Z qemu-system-x86_64: load of migration<o:p></o:p></p>
<p class="MsoPlainText">> failed: Input/output error<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> 2015-02-26 14:50:26.684+0000: shutting down<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> ßsnip à<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> the time here seems to be logged in GMT and in the snippets above it
<o:p></o:p></p>
<p class="MsoPlainText">> is CET – might this be the problem ???<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Here is the qemu commandline (that was removed above in cleaned<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> ßsnip à<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> /usr/bin/qemu-kvm<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -name SERVERNAME<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -S<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -machine pc-1.0,accel=kvm,usb=off<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -cpu SandyBridge<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -m 10240<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -realtime mlock=off<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -smp 4,maxcpus=64,sockets=16,cores=4,threads=1<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -uuid b75823d1-00f0-457e-a692-8b95f73907db<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -smbios type=1,manufacturer=oVirt,product=oVirt<o:p></o:p></p>
<p class="MsoPlainText">> Node,version=20-3,serial=4C4C4544-0038-3210-8048-B7C04F303232,uuid=b75<o:p></o:p></p>
<p class="MsoPlainText">> 823d1-00f0-457e-a692-8b95f73907db<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -no-user-config<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -nodefaults<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -chardev<o:p></o:p></p>
<p class="MsoPlainText">> socket,id=charmonitor,path=/var/lib/libvirt/qemu/al-exchange-01.monito<o:p></o:p></p>
<p class="MsoPlainText">> r,server,nowait<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -mon chardev=charmonitor,id=monitor,mode=control<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -rtc base=2015-02-26T15:42:26,driftfix=slew<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -global kvm-pit.lost_tick_policy=discard<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -no-hpet<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -no-shutdown<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -boot strict=on<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -device <o:p></o:p></p>
<p class="MsoPlainText">> virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -drive<o:p></o:p></p>
<p class="MsoPlainText">> file=/rhev/data-center/mnt/XXXX:_data_export_iso/85821683-bc7c-429b-bf<o:p></o:p></p>
<p class="MsoPlainText">> f2-ce25f56fc16f/images/11111111-1111-1111-1111-111111111111/XXXX.iso,i<o:p></o:p></p>
<p class="MsoPlainText">> f=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=<o:p></o:p></p>
<p class="MsoPlainText">> -device<o:p></o:p></p>
<p class="MsoPlainText">> ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=2<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -drive<o:p></o:p></p>
<p class="MsoPlainText">> file=/rhev/data-center/0f954891-b1cd-4f09-99ae-75d404d95f9d/276e9ba7-e<o:p></o:p></p>
<p class="MsoPlainText">> 19a-49c5-8ad7-26711934d5e4/images/c37bfa94-718c-4125-9202-bc299535eca5<o:p></o:p></p>
<p class="MsoPlainText">> /299bd6d1-f4a0-4296-9944-176b2252a886,if=none,id=drive-virtio-disk0,fo<o:p></o:p></p>
<p class="MsoPlainText">> rmat=raw,serial=c37bfa94-718c-4125-9202-bc299535eca5,cache=none,werror<o:p></o:p></p>
<p class="MsoPlainText">> =stop,rerror=stop,aio=threads<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -device<o:p></o:p></p>
<p class="MsoPlainText">> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id<o:p></o:p></p>
<p class="MsoPlainText">> =virtio-disk0,bootindex=1<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -drive<o:p></o:p></p>
<p class="MsoPlainText">> file=/rhev/data-center/0f954891-b1cd-4f09-99ae-75d404d95f9d/276e9ba7-e<o:p></o:p></p>
<p class="MsoPlainText">> 19a-49c5-8ad7-26711934d5e4/images/75134ccc-b74e-4955-90a5-95d4ceff403b<o:p></o:p></p>
<p class="MsoPlainText">> /1e5b5b86-c31d-476c-acb2-a5bd6a65490b,if=none,id=drive-virtio-disk1,fo<o:p></o:p></p>
<p class="MsoPlainText">> rmat=raw,serial=75134ccc-b74e-4955-90a5-95d4ceff403b,cache=none,werror<o:p></o:p></p>
<p class="MsoPlainText">> =stop,rerror=stop,aio=threads<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -device<o:p></o:p></p>
<p class="MsoPlainText">> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk1,id<o:p></o:p></p>
<p class="MsoPlainText">> =virtio-disk1<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -netdev tap,fd=30,id=hostnet0,vhost=on,vhostfd=31<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -device<o:p></o:p></p>
<p class="MsoPlainText">> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:c7:72:0a,bus=pci.0<o:p></o:p></p>
<p class="MsoPlainText">> ,addr=0x3<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -chardev<o:p></o:p></p>
<p class="MsoPlainText">> socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/b75823d1-00<o:p></o:p></p>
<p class="MsoPlainText">> f0-457e-a692-8b95f73907db.com.redhat.rhevm.vdsm,server,nowait<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -device<o:p></o:p></p>
<p class="MsoPlainText">> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=chann<o:p></o:p></p>
<p class="MsoPlainText">> el0,name=com.redhat.rhevm.vdsm<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -chardev<o:p></o:p></p>
<p class="MsoPlainText">> socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/b75823d1-00<o:p></o:p></p>
<p class="MsoPlainText">> f0-457e-a692-8b95f73907db.org.qemu.guest_agent.0,server,nowait<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -device<o:p></o:p></p>
<p class="MsoPlainText">> virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=chann<o:p></o:p></p>
<p class="MsoPlainText">> el1,name=org.qemu.guest_agent.0<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -device usb-tablet,id=input0<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -vnc IP-ADDRESS:2,password<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -k de<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -device cirrus-vga,id=video0,bus=pci.0,addr=0x2<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -incoming tcp:[::]:49152<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> -msg timestamp=on<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> ßsnip à<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Here is the content of the .meta file in the directories of the 2
<o:p></o:p></p>
<p class="MsoPlainText">> images above<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <--snip--><o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> DOMAIN=276e9ba7-e19a-49c5-8ad7-26711934d5e4<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> VOLTYPE=LEAF<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> CTIME=1424445378<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> FORMAT=RAW<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> IMAGE=c37bfa94-718c-4125-9202-bc299535eca5<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> DISKTYPE=2<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> PUUID=00000000-0000-0000-0000-000000000000<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> LEGALITY=LEGAL<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> MTIME=0<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> POOL_UUID=<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> DESCRIPTION={"DiskAlias":"al-exchange-01_Disk1","DiskDescription":"Sys<o:p></o:p></p>
<p class="MsoPlainText">> tem"}<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> TYPE=SPARSE<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> SIZE=146800640<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> EOF<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <--snip--><o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <--snip--><o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> DOMAIN=276e9ba7-e19a-49c5-8ad7-26711934d5e4<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> CTIME=1424785898<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> FORMAT=RAW<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> DISKTYPE=2<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> LEGALITY=LEGAL<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> SIZE=419430400<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> VOLTYPE=LEAF<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> DESCRIPTION={"DiskAlias":"al-exchange-01_Disk2","DiskDescription":"Dat<o:p></o:p></p>
<p class="MsoPlainText">> a"}<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> IMAGE=75134ccc-b74e-4955-90a5-95d4ceff403b<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> PUUID=00000000-0000-0000-0000-000000000000<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> MTIME=0<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> POOL_UUID=<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> TYPE=SPARSE<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> EOF<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <--snip--><o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText"><span lang="EN-US">> *From:*Roy Golan [</span><a href="mailto:rgolan@redhat.com"><span lang="EN-US" style="color:windowtext;text-decoration:none">mailto:rgolan@redhat.com</span></a><span lang="EN-US">]<o:p></o:p></span></p>
<p class="MsoPlainText">> *Sent:* Wednesday, February 18, 2015 12:12 PM<o:p></o:p></p>
<p class="MsoPlainText"><span lang="EN-US">> *To:* Soeren Malchow; </span><a href="mailto:users@ovirt.org"><span lang="EN-US" style="color:windowtext;text-decoration:none">users@ovirt.org</span></a><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoPlainText">> *Subject:* Re: [ovirt-users] Hosted Engine Migration fails<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> On 02/16/2015 04:55 AM, Soeren Malchow wrote:<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Dear all,<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> we ahve a setup with several hosts running fedora 10 with the<o:p></o:p></p>
<p class="MsoPlainText">> virt-preview packages installed (for snapshot live merge) and a<o:p></o:p></p>
<p class="MsoPlainText">> hosted engine running centos 6.6-<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> We are experiencing a problem with the Live Migration of the Hosted<o:p></o:p></p>
<p class="MsoPlainText">> Engine, in the case of setting the host for the Engine into<o:p></o:p></p>
<p class="MsoPlainText">> maintenance as well as a manual migration.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> I tried this on the “ovirtmgmt” network and when that failed I did<o:p></o:p></p>
<p class="MsoPlainText">> some research and tried to use another network interface (separate<o:p></o:p></p>
<p class="MsoPlainText">> from ovirtmgmt on layer 2), this also fails.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> It looks as if the migration is still going through the ovirtmgmt<o:p></o:p></p>
<p class="MsoPlainText">> interface, at least judging from the network traffic, and I think<o:p></o:p></p>
<p class="MsoPlainText">> the error that I found (RHBZ#919201) is actually the right one.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> --<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-6745::WARNING::2015-02-16<o:p></o:p></p>
<p class="MsoPlainText">> 03:22:25,743::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`XXX`::Migration stalling: remaining (35MiB) > lowmark (15MiB).<o:p></o:p></p>
<p class="MsoPlainText">> Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-6745::WARNING::2015-02-16<o:p></o:p></p>
<p class="MsoPlainText">> 03:22:35,745::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`XXX`::Migration stalling: remaining (129MiB) > lowmark<o:p></o:p></p>
<p class="MsoPlainText">> (15MiB). Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-6745::WARNING::2015-02-16<o:p></o:p></p>
<p class="MsoPlainText">> 03:22:45,747::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`XXX`::Migration stalling: remaining (42MiB) > lowmark (15MiB).<o:p></o:p></p>
<p class="MsoPlainText">> Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> vdsm.log:Thread-6745::WARNING::2015-02-16<o:p></o:p></p>
<p class="MsoPlainText">> 03:22:55,749::migration::458::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">> vmId=`XXX`::Migration stalling: remaining (88MiB) > lowmark (15MiB).<o:p></o:p></p>
<p class="MsoPlainText">> Refer to RHBZ#919201.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> --<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> The ovirtmgmt interface is 2 x 1Gbit (LACP connected to Dell<o:p></o:p></p>
<p class="MsoPlainText">> Switches with MLAG) and by far not fully utilized.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Can anyone help me where to go form here ?<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> still need the whole log. probably the guest (engine) is doing lots of
<o:p></o:p></p>
<p class="MsoPlainText">> memory i/o if you got a fair amount of running VMs and Hosts. that
<o:p></o:p></p>
<p class="MsoPlainText">> will stalls the migration because the guest pages are getting dirty
<o:p></o:p></p>
<p class="MsoPlainText">> faster than qemu can copy.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> you have 2 options:<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> 1. try several more times.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> 2. shutdown the engine vm, it should start on another host<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Regards<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Soeren<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Hi Soeren,<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">the key issue seems to be this one-<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">"<o:p></o:p></p>
<p class="MsoPlainText">vdsm.log:Thread-49551::WARNING::2015-02-26<o:p></o:p></p>
<p class="MsoPlainText">15:50:27,359::migration::445::vm.Vm::(monitor_migration)<o:p></o:p></p>
<p class="MsoPlainText">vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration is stuck: Hasn't progressed in 150.069458961 seconds. Aborting.<o:p></o:p></p>
<p class="MsoPlainText">"<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Basically it means that the migration process is not converging since moving the data from one host to the other is too slow.<o:p></o:p></p>
<p class="MsoPlainText">The reason for this could be-<o:p></o:p></p>
<p class="MsoPlainText">1. Extremely slow or busy network.<o:p></o:p></p>
<p class="MsoPlainText">2. Something inside your VM is changing memory very fast (faster than the copying rate).<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">In order to rule out (2), you can start an empty VM with nothing inside but a minimal OS. If you're not sure you can use tiny Linux[1].<o:p></o:p></p>
<p class="MsoPlainText">Such a minimal VM should have no problem with migrating from one machine to the other. If it has an issue, it means that you have a problem in your network that causes the migration process to take longer than it should.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Give it a try and let us know how it goes.<o:p></o:p></p>
<p class="MsoPlainText">Doron<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">[1] <a href="http://distro.ibiblio.org/tinycorelinux/"><span style="color:windowtext;text-decoration:none">http://distro.ibiblio.org/tinycorelinux/</span></a><o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
</div>
</body>
</html>