<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;
mso-fareast-language:EN-US;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.gwt-inlinelabel
{mso-style-name:gwt-inlinelabel;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:70.85pt 70.85pt 56.7pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="DE" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">we tried this (Roys Mail below) , and yes, shutdown always works.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">But now this problem comes up with regular machines now as well.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">The environment is setup like this<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">Engine: Ovirt
</span></span><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">3.5.1.1-1.el6 – on CentOS6
<o:p></o:p></span></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">The Storage Backend is gluster 3.6.2-1.el7 on CentOS 7<o:p></o:p></span></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">Compute hosts: Libvirt Version: libvirt-1.2.9.1-2.fc20, kvm
2.1.2 - 7.fc20, vdsm vdsm-4.16.10-8.gitc937927.fc20<o:p></o:p></span></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">All compute Servers are 100% identical.<o:p></o:p></span></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">The storage cluster was tested manually and works just fine.<o:p></o:p></span></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">The network interfaces are not fully utilized, more like 15%.<o:p></o:p></span></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">Log output ist as below. The thing in the log output I do not
understand is this<o:p></o:p></span></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">“</span></span></span><span lang="EN-US" style="color:#1F497D">2015-02-26T14:50:26.650595Z
qemu-system-x86_64: load of migration failed: Input/output error” <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">From the qemu log.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Also if I shut down machines, put their host into maintenance and start them somewhere else, everything works just fine.</span><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><o:p></o:p></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">Can someone help with this ? Any idea where to look ?<o:p></o:p></span></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><o:p> </o:p></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">Regards<o:p></o:p></span></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US" style="color:#1F4E79;mso-style-textfill-fill-color:#1F4E79;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">Soeren
<o:p></o:p></span></span></span></p>
<p class="MsoNormal"><span class="gwt-inlinelabel"><span lang="EN-US"><o:p> </o:p></span></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:12.0pt;color:#1F497D">From VDSM Log<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">I just tried to migrate a machine, this here happens on the source
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Wingdings;color:#1F497D">ß</span><span lang="EN-US" style="color:#1F497D"> snip
</span><span lang="EN-US" style="font-family:Wingdings;color:#1F497D">à</span><span lang="EN-US" style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49548::DEBUG::2015-02-26 15:42:26,692::__init__::469::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'VM.migrate' in bridge with {u'params': {u'tunneled': u'false', u'dstqemu': u'172.19.2.31',
u'src': u'compute04', u'dst': u'compute01:54321', u'vmId': u'b75823d1-00f0-457e-a692-8b95f73907db', u'abortOnError': u'true', u'method': u'online'}, u'vmID': u'b75823d1-00f0-457e-a692-8b95f73907db'}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49548::DEBUG::2015-02-26 15:42:26,694::API::510::vds::(migrate) {u'tunneled': u'false', u'dstqemu': u'IPADDR', u'src': u'compute04', u'dst': u'compute01:54321', u'vmId': u'b75823d1-00f0-457e-a692-8b95f73907db',
u'abortOnError': u'true', u'method': u'online'}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49549::DEBUG::2015-02-26 15:42:26,699::migration::103::vm.Vm::(_setupVdsConnection) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Destination server is: compute01:54321<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49549::DEBUG::2015-02-26 15:42:26,702::migration::105::vm.Vm::(_setupVdsConnection) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Initiating connection with destination<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49549::DEBUG::2015-02-26 15:42:26,733::migration::155::vm.Vm::(_prepareGuest) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration started<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49549::DEBUG::2015-02-26 15:42:26,755::migration::238::vm.Vm::(run) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::migration semaphore acquired after 0 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49549::DEBUG::2015-02-26 15:42:27,211::migration::298::vm.Vm::(_startUnderlyingMigration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::starting migration to qemu+tls://compute01/system
with miguri tcp://IPADDR<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49550::DEBUG::2015-02-26 15:42:27,213::migration::361::vm.Vm::(run) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::migration downtime thread started<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::DEBUG::2015-02-26 15:42:27,216::migration::410::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::starting migration monitor thread<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49550::DEBUG::2015-02-26 15:43:42,218::migration::370::vm.Vm::(run) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime to 50<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49550::DEBUG::2015-02-26 15:44:57,222::migration::370::vm.Vm::(run) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime to 100<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49550::DEBUG::2015-02-26 15:46:12,227::migration::370::vm.Vm::(run) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime to 150<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:47:07,279::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (1791MiB) > lowmark
(203MiB). Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:47:17,281::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (1398MiB) > lowmark
(203MiB). Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49550::DEBUG::2015-02-26 15:47:27,233::migration::370::vm.Vm::(run) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime to 200<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:47:27,283::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (1066MiB) > lowmark
(203MiB). Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:47:37,285::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (701MiB) > lowmark (203MiB).
Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:47:47,287::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (361MiB) > lowmark (203MiB).
Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:07,291::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (683MiB) > lowmark (13MiB).
Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:17,292::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (350MiB) > lowmark (13MiB).
Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:27,294::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (18MiB) > lowmark (13MiB).
Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:37,296::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (646MiB) > lowmark (13MiB).
Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49550::DEBUG::2015-02-26 15:48:42,238::migration::370::vm.Vm::(run) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime to 250<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:47,334::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (317MiB) > lowmark (13MiB).
Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:48:57,342::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (1018MiB) > lowmark
(13MiB). Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:07,344::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (679MiB) > lowmark (13MiB).
Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:17,346::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (357MiB) > lowmark (13MiB).
Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:27,348::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (31MiB) > lowmark (13MiB).
Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:37,349::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (854MiB) > lowmark (13MiB).
Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:47,351::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (525MiB) > lowmark (13MiB).
Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49550::DEBUG::2015-02-26 15:49:57,242::migration::370::vm.Vm::(run) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::setting migration downtime to 300<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:49:57,353::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (183MiB) > lowmark (13MiB).
Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:50:07,355::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (785MiB) > lowmark (13MiB).
Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:50:17,357::migration::458::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration stalling: remaining (457MiB) > lowmark (13MiB).
Refer to RHBZ#919201.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::WARNING::2015-02-26 15:50:27,359::migration::445::vm.Vm::(monitor_migration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Migration is stuck: Hasn't progressed in 150.069458961
seconds. Aborting.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49551::DEBUG::2015-02-26 15:50:27,362::migration::470::vm.Vm::(stop) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::stopping migration monitor thread<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49549::DEBUG::2015-02-26 15:50:27,852::migration::376::vm.Vm::(cancel) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::canceling migration downtime thread<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49549::DEBUG::2015-02-26 15:50:27,852::migration::470::vm.Vm::(stop) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::stopping migration monitor thread<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49550::DEBUG::2015-02-26 15:50:27,853::migration::373::vm.Vm::(run) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::migration downtime thread exiting<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49549::ERROR::2015-02-26 15:50:27,855::migration::161::vm.Vm::(_recover) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::operation aborted: migration job: canceled by client<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49549::ERROR::2015-02-26 15:50:28,049::migration::260::vm.Vm::(run) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Failed to migrate<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-49690::DEBUG::2015-02-26 15:50:33,284::__init__::469::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'VM.getMigrationStatus' in bridge with {u'vmID': u'b75823d1-00f0-457e-a692-8b95f73907db'}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Wingdings;color:#1F497D">ß</span><span lang="EN-US" style="color:#1F497D">snip
</span><span lang="EN-US" style="font-family:Wingdings;color:#1F497D">à</span><span lang="EN-US" style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">At the same time on the destination server<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Wingdings;color:#1F497D">ß</span><span lang="EN-US" style="color:#1F497D"> snip
</span><span lang="EN-US" style="font-family:Wingdings;color:#1F497D">à</span><span lang="EN-US" style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350501::DEBUG::2015-02-26 15:42:25,923::BindingXMLRPC::1133::vds::(wrapper) client [172.19.2.34]::call vmGetStats with ('b75823d1-00f0-457e-a692-8b95f73907db',) {}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350502::DEBUG::2015-02-26 15:42:26,018::BindingXMLRPC::1133::vds::(wrapper) client [172.19.2.34]::call vmMigrationCreate with ALL THE MACHINE START PARAMETERS here<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350502::INFO::2015-02-26 15:42:26,019::clientIF::394::vds::(createVm) vmContainerLock acquired by vm b75823d1-00f0-457e-a692-8b95f73907db<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350502::DEBUG::2015-02-26 15:42:26,033::clientIF::407::vds::(createVm) Total desktops after creation of b75823d1-00f0-457e-a692-8b95f73907db is 3<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350503::DEBUG::2015-02-26 15:42:26,033::vm::2264::vm.Vm::(_startUnderlyingVm) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Start<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350502::DEBUG::2015-02-26 15:42:26,036::vm::5658::vm.Vm::(waitForMigrationDestinationPrepare) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::migration destination: waiting for VM creation<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350503::DEBUG::2015-02-26 15:42:26,036::vm::2268::vm.Vm::(_startUnderlyingVm) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::_ongoingCreations acquired<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350502::DEBUG::2015-02-26 15:42:26,038::vm::5663::vm.Vm::(waitForMigrationDestinationPrepare) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::migration destination: waiting 48s for path
preparation<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350503::INFO::2015-02-26 15:42:26,038::vm::3261::vm.Vm::(_run) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::VM wrapper has started<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350503::WARNING::2015-02-26 15:42:26,041::vm::2056::vm.Vm::(buildConfDevices) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Unknown type found, device: '{'device': 'unix', 'alias':
'channel0', 'type': 'channel', 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '1'}}' found<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350503::WARNING::2015-02-26 15:42:26,041::vm::2056::vm.Vm::(buildConfDevices) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Unknown type found, device: '{'device': 'unix', 'alias':
'channel1', 'type': 'channel', 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '2'}}' found<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350503::DEBUG::2015-02-26 15:42:26,349::vm::1058::vm.Vm::(__init__) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Ignoring param (target, 10485760) in BalloonDevice<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350503::DEBUG::2015-02-26 15:42:26,350::vm::2294::vm.Vm::(_startUnderlyingVm) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::_ongoingCreations released<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350502::ERROR::2015-02-26 15:42:26,351::vm::5638::vm.Vm::(_updateDevicesDomxmlCache) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Alias not found for device type graphics during migration
at destination host<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350503::DEBUG::2015-02-26 15:42:26,353::vm::4128::vm.Vm::(_waitForUnderlyingMigration) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Waiting 21600 seconds for end of migration<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350502::DEBUG::2015-02-26 15:42:26,377::BindingXMLRPC::1140::vds::(wrapper) return vmMigrationCreate with {ALL THE MACHINE PARAMETERS HERE<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:libvirtEventLoop::DEBUG::2015-02-26 15:42:27,195::vm::5571::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::event Started detail 1 opaque None<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:libvirtEventLoop::DEBUG::2015-02-26 15:50:27,037::vm::5571::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::event Stopped detail 5 opaque None<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:libvirtEventLoop::INFO::2015-02-26 15:50:27,038::vm::2366::vm.Vm::(_onQemuDeath) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::underlying process disconnected<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:libvirtEventLoop::INFO::2015-02-26 15:50:27,038::vm::4952::vm.Vm::(releaseVm) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Release VM resources<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350503::DEBUG::2015-02-26 15:50:27,044::libvirtconnection::143::root::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Domain not found: no domain with matching
uuid 'b75823d1-00f0-457e-a692-8b95f73907db'<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350503::ERROR::2015-02-26 15:50:27,047::vm::2325::vm.Vm::(_startUnderlyingVm) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Failed to start a migration destination vm<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:MigrationError: Domain not found: no domain with matching uuid 'b75823d1-00f0-457e-a692-8b95f73907db'<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-350503::DEBUG::2015-02-26 15:50:27,058::vm::2786::vm.Vm::(setDownStatus) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Changed state to Down: VM failed to migrate (code=8)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-351517::DEBUG::2015-02-26 15:50:27,089::BindingXMLRPC::1133::vds::(wrapper) client [172.19.2.34]::call vmDestroy with ('b75823d1-00f0-457e-a692-8b95f73907db',) {}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-351517::INFO::2015-02-26 15:50:27,092::API::332::vds::(destroy) vmContainerLock acquired by vm b75823d1-00f0-457e-a692-8b95f73907db<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-351517::DEBUG::2015-02-26 15:50:27,154::vm::5026::vm.Vm::(destroy) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::destroy Called<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:libvirtEventLoop::WARNING::2015-02-26 15:50:27,162::utils::129::root::(rmFile) File: /var/lib/libvirt/qemu/channels/b75823d1-00f0-457e-a692-8b95f73907db.com.redhat.rhevm.vdsm already removed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:libvirtEventLoop::WARNING::2015-02-26 15:50:27,163::utils::129::root::(rmFile) File: /var/lib/libvirt/qemu/channels/b75823d1-00f0-457e-a692-8b95f73907db.org.qemu.guest_agent.0 already removed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:libvirtEventLoop::INFO::2015-02-26 15:50:27,164::logUtils::44::dispatcher::(wrapper) Run and protect: inappropriateDevices(thiefId='b75823d1-00f0-457e-a692-8b95f73907db')<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:Thread-351517::DEBUG::2015-02-26 15:50:27,170::vm::5020::vm.Vm::(deleteVm) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::Total desktops after destroy of b75823d1-00f0-457e-a692-8b95f73907db
is 2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">vdsm.log:libvirtEventLoop::WARNING::2015-02-26 15:50:27,170::vm::1953::vm.Vm::(_set_lastStatus) vmId=`b75823d1-00f0-457e-a692-8b95f73907db`::trying to set state to Down when already Down<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Wingdings;color:#1F497D">ß</span><span lang="EN-US" style="color:#1F497D"> snip
</span><span lang="EN-US" style="font-family:Wingdings;color:#1F497D">à</span><span lang="EN-US" style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">And I can see the error “Domain not found: no domain with matching uuid 'b75823d1-00f0-457e-a692-8b95f73907db'” in the middle, but I am not entirely sure what this means.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">In the qemu log I can see this<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Wingdings;color:#1F497D">ß</span><span lang="EN-US" style="color:#1F497D"> snip
</span><span lang="EN-US" style="font-family:Wingdings;color:#1F497D">à</span><span lang="EN-US" style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:12.0pt;color:#1F497D">From QEMU LOG<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">2015-02-26 14:42:26.859+0000: starting up<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Domain id=8 is tainted: hook-script<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">2015-02-26T14:50:26.650595Z qemu-system-x86_64: load of migration failed: Input/output error<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">2015-02-26 14:50:26.684+0000: shutting down<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Wingdings;color:#1F497D">ß</span><span lang="EN-US" style="color:#1F497D"> snip
</span><span lang="EN-US" style="font-family:Wingdings;color:#1F497D">à</span><span lang="EN-US" style="color:#1F497D">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">the time here seems to be logged in GMT and in the snippets above it is CET – might this be the problem ???<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Here is the qemu commandline (that was removed above in cleaned
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Wingdings;color:#1F497D">ß</span><span style="color:#1F497D"> snip
</span><span lang="EN-US" style="font-family:Wingdings;color:#1F497D">à</span><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">/usr/bin/qemu-kvm <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">-name SERVERNAME <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-S <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-machine pc-1.0,accel=kvm,usb=off
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-cpu SandyBridge <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-m 10240 <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-realtime mlock=off
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-smp 4,maxcpus=64,sockets=16,cores=4,threads=1
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-uuid b75823d1-00f0-457e-a692-8b95f73907db
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-smbios type=1,manufacturer=oVirt,product=oVirt Node,version=20-3,serial=4C4C4544-0038-3210-8048-B7C04F303232,uuid=b75823d1-00f0-457e-a692-8b95f73907db
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-no-user-config <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-nodefaults <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/al-exchange-01.monitor,server,nowait
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-mon chardev=charmonitor,id=monitor,mode=control
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-rtc base=2015-02-26T15:42:26,driftfix=slew
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-global kvm-pit.lost_tick_policy=discard
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-no-hpet <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-no-shutdown <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-boot strict=on <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-drive file=/rhev/data-center/mnt/XXXX:_data_export_iso/85821683-bc7c-429b-bff2-ce25f56fc16f/images/11111111-1111-1111-1111-111111111111/XXXX.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=2 <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-drive file=/rhev/data-center/0f954891-b1cd-4f09-99ae-75d404d95f9d/276e9ba7-e19a-49c5-8ad7-26711934d5e4/images/c37bfa94-718c-4125-9202-bc299535eca5/299bd6d1-f4a0-4296-9944-176b2252a886,if=none,id=drive-virtio-disk0,format=raw,serial=c37bfa94-718c-4125-9202-bc299535eca5,cache=none,werror=stop,rerror=stop,aio=threads
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-drive file=/rhev/data-center/0f954891-b1cd-4f09-99ae-75d404d95f9d/276e9ba7-e19a-49c5-8ad7-26711934d5e4/images/75134ccc-b74e-4955-90a5-95d4ceff403b/1e5b5b86-c31d-476c-acb2-a5bd6a65490b,if=none,id=drive-virtio-disk1,format=raw,serial=75134ccc-b74e-4955-90a5-95d4ceff403b,cache=none,werror=stop,rerror=stop,aio=threads
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk1,id=virtio-disk1
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-netdev tap,fd=30,id=hostnet0,vhost=on,vhostfd=31
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:c7:72:0a,bus=pci.0,addr=0x3
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/b75823d1-00f0-457e-a692-8b95f73907db.com.redhat.rhevm.vdsm,server,nowait
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/b75823d1-00f0-457e-a692-8b95f73907db.org.qemu.guest_agent.0,server,nowait
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-device usb-tablet,id=input0
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-vnc IP-ADDRESS:2,password
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-k de <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-device cirrus-vga,id=video0,bus=pci.0,addr=0x2
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-incoming tcp:[::]:49152
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-msg timestamp=on<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Wingdings;color:#1F497D">ß</span><span lang="EN-US" style="color:#1F497D"> snip
</span><span lang="EN-US" style="font-family:Wingdings;color:#1F497D">à</span><span lang="EN-US" style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Here is the content of the .meta file in the directories of the 2 images above<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><--snip--> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">DOMAIN=276e9ba7-e19a-49c5-8ad7-26711934d5e4<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">VOLTYPE=LEAF<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">CTIME=1424445378<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">FORMAT=RAW<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">IMAGE=c37bfa94-718c-4125-9202-bc299535eca5<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">DISKTYPE=2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">PUUID=00000000-0000-0000-0000-000000000000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">LEGALITY=LEGAL<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">MTIME=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">POOL_UUID=<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">DESCRIPTION={"DiskAlias":"al-exchange-01_Disk1","DiskDescription":"System"}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">TYPE=SPARSE<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">SIZE=146800640<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">EOF<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><--snip--><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><--snip--> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">DOMAIN=276e9ba7-e19a-49c5-8ad7-26711934d5e4<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">CTIME=1424785898<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">FORMAT=RAW<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">DISKTYPE=2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">LEGALITY=LEGAL<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">SIZE=419430400<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">VOLTYPE=LEAF<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">DESCRIPTION={"DiskAlias":"al-exchange-01_Disk2","DiskDescription":"Data"}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">IMAGE=75134ccc-b74e-4955-90a5-95d4ceff403b<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">PUUID=00000000-0000-0000-0000-000000000000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">MTIME=0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">POOL_UUID=<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">TYPE=SPARSE<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">EOF<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><--snip--><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></a></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span lang="EN-US" style="color:windowtext;mso-fareast-language:DE">From:</span></b><span lang="EN-US" style="color:windowtext;mso-fareast-language:DE"> Roy Golan [mailto:rgolan@redhat.com]
<br>
<b>Sent:</b> Wednesday, February 18, 2015 12:12 PM<br>
<b>To:</b> Soeren Malchow; users@ovirt.org<br>
<b>Subject:</b> Re: [ovirt-users] Hosted Engine Migration fails<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 02/16/2015 04:55 AM, Soeren Malchow wrote:<span style="font-size:12.0pt;mso-fareast-language:DE"><o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Dear all,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">we ahve a setup with several hosts running fedora 10 with the virt-preview packages installed (for snapshot live merge) and a hosted engine running centos 6.6-</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">We are experiencing a problem with the Live Migration of the Hosted Engine, in the case of setting the host for the Engine into maintenance as well as a manual migration.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">I tried this on the “ovirtmgmt” network and when that failed I did some research and tried to use another network interface (separate from ovirtmgmt on layer 2), this also fails.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">It looks as if the migration is still going through the ovirtmgmt interface, at least judging from the network traffic, and I think the error that I found (RHBZ#919201) is actually the right one.
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:12.0pt"> --</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:12.0pt">vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:25,743::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining
(35MiB) > lowmark (15MiB). Refer to RHBZ#919201.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:12.0pt">vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:35,745::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining
(129MiB) > lowmark (15MiB). Refer to RHBZ#919201.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:12.0pt">vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:45,747::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining
(42MiB) > lowmark (15MiB). Refer to RHBZ#919201.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:12.0pt">vdsm.log:Thread-6745::WARNING::2015-02-16 03:22:55,749::migration::458::vm.Vm::(monitor_migration) vmId=`XXX`::Migration stalling: remaining
(88MiB) > lowmark (15MiB). Refer to RHBZ#919201.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:12.0pt">--</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:12.0pt">The ovirtmgmt interface is 2 x 1Gbit (LACP connected to Dell Switches with MLAG) and by far not fully utilized.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:12.0pt">Can anyone help me where to go form here ?</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif;mso-fareast-language:DE"><br>
still need the whole log. probably the guest (engine) is doing lots of memory i/o if you got a fair amount of running VMs and Hosts. that will stalls the migration because
<br>
the guest pages are getting dirty faster than qemu can copy.<br>
<br>
<br>
you have 2 options:<br>
<br>
1. try several more times.<br>
<br>
2. shutdown the engine vm, it should start on another host<br>
<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:12.0pt">Regards</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:12.0pt">Soeren
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif;mso-fareast-language:DE"><br>
<br>
<br>
<o:p></o:p></span></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Users mailing list<o:p></o:p></pre>
<pre><a href="mailto:Users@ovirt.org">Users@ovirt.org</a><o:p></o:p></pre>
<pre><a href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif;mso-fareast-language:DE"><o:p> </o:p></span></p>
</div>
</body>
</html>