<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi folks,<br>
<br>
Whilst testing migration, I found an oddness that others might
benefit from knowing about.<br>
<br>
<b>My environment:</b><br>
Two hosts, CentOS underneath, Ovirt 3.4.1 hosted-engine (clean
build) on top.<br>
<br>
<b>What happened:</b><br>
Doing migration testing (it is so nice to have reached this stage),
I found that VMs could migrate from the first host to the second,
but not the other way.<br>
To be clear, by "first host", I mean the first one that Ovirt was
installed on. Host ID 1.<br>
<br>
The Ovirt GUI's messages are uninformative (it failed; I knew that):<br>
Migration failed due to Error: Fatal error during migration
(VM:testvm1 , Source: host2.ovirt, Destination:host1.ovirt ).<br>
<br>
<b>Logs to look for:</b><br>
vdsm log on the source host (host 2) is informative, and includes
this clue:<br>
<tt>libvirtError: operation failed: Failed to connect to remote
libvirt URI qemu+tls://10.0.14.9/system</tt><br>
<br>
which leads us to <b>libvirtd</b>. Let's have a look at it:<br>
<br>
This is host 2, the one that's working, I can migrate to it:<br>
<tt>[root@host2 ~]# grep -v ^# /etc/libvirt/libvirtd.conf | grep -v
^$</tt><tt><br>
</tt><tt> listen_addr="0.0.0.0"</tt><tt><br>
</tt><tt> unix_sock_group="qemu"</tt><tt><br>
</tt><tt> unix_sock_rw_perms="0770"</tt><tt><br>
</tt><tt> auth_unix_rw="sasl"</tt><tt><br>
</tt><tt> host_uuid="a1e1b804-1b88-4380-8ccc-7802b3a025bb"</tt><tt><br>
</tt><tt> keepalive_interval=-1</tt><tt><br>
</tt><tt> log_outputs="1:</tt><tt><a class="moz-txt-link-freetext"
href="file:/var/log/libvirt/libvirtd.log">file:/var/log/libvirt/libvirtd.log</a></tt><tt>"</tt><tt><br>
</tt><tt> log_filters="3:virobject 3:virfile 2:virnetlink
3:cgroup 3:event 3:json 1:libvirt 1:util 1:qemu"</tt><tt><br>
</tt><tt> ca_file="/etc/pki/vdsm/certs/cacert.pem"</tt><tt><br>
</tt><tt> cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"</tt><tt><br>
</tt><tt> key_file="/etc/pki/vdsm/keys/vdsmkey.pem"</tt><tt><br>
</tt><tt> </tt><tt><br>
</tt><tt> [root@host2 ~]# lsof | grep libvirtd | grep TCP</tt><tt><br>
</tt><tt> libvirtd 22244 root 13u IPv4
99175113 0t0 TCP *:16514 (LISTEN)</tt><tt><br>
</tt><br>
Now the one that's not working, with differences highlighted.<br>
<br>
[root@host1 ~]# grep -v ^# /etc/libvirt/libvirtd.conf | grep -v ^$<br>
<b>listen_tcp = 0</b><br>
<b>auth_tcp = "sasl"</b><br>
<i># lines below here were added by vdsm</i><br>
listen_addr="0.0.0.0"<br>
unix_sock_group="qemu"<br>
unix_sock_rw_perms="0770"<br>
auth_unix_rw="sasl"<br>
host_uuid="a9fd3fe1-b3a7-48f7-8870-cff3ee4c1f62"<br>
keepalive_interval=-1<br>
log_outputs="1:<a class="moz-txt-link-freetext"
href="file:/var/log/libvirt/libvirtd.log">file:/var/log/libvirt/libvirtd.log</a>"<br>
log_filters="3:virobject 3:virfile 2:virnetlink 3:cgroup 3:event
3:json 1:libvirt 1:util 1:qemu"<br>
<b>listen_tcp=0</b><br>
<b>listen_tls=0</b><br>
ca_file="/etc/pki/vdsm/certs/cacert.pem"<br>
cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"<br>
key_file="/etc/pki/vdsm/keys/vdsmkey.pem"<br>
<br>
For the sake of text mail clients, the different lines were:<br>
<b>listen_tcp = 0</b><br>
<b>auth_tcp = "sasl"</b><br>
<b>listen_tcp=0</b><br>
<b>listen_tls=0</b><br>
<br>
and, owing to the listen_tcp=0 (yes, it appears twice in the config
file), there was no libvirt tcp socket listening here.<br>
<br>
Comment out those four lines to make it the same as the working one,
restart libvirtd, and now I can migrate both ways.<br>
<br>
WARNING: If you are in production already, make sure to migrate your
VMs off host 1 BEFORE restarting libvirtd. I found that restarting
libvirtd caused guest VMs to die, which doesn't bother me - I've no
production VMs on there yet, though we're not far off, touch wood.<br>
<br>
Now, the command to restart libvirtd:<br>
<br>
[root@host1 ~]# initctl restart libvirtd<br>
libvirtd start/running, process 16279<br>
[root@host1 ~]# lsof | grep ^libvirtd | grep TCP<br>
libvirtd 16279 root 13u IPv4 104492880
0t0 TCP *:16514 (LISTEN)<br>
<br>
At this point, host1 becomes unresponsive in the Ovirt GUI, and when
it comes back all by itself, we can migrate to it. In the first
couple of minutes, you might see a reinitializing message when you
start migration; it'll queue, and it'll work.<br>
<br>
End of that problem, for me at least.<br>
<br>
Others who've more experience than I may critique the libvirtd
config. This is the only manual intervention I've done with it.<br>
<br>
For developers:<br>
- I didn't have this problem with 3.3, but then, I didn't have
hosted-engine then either.<br>
- My 3.4.0 environment (my first hosted-engine) is mostly dead, so I
can't say whether this problem was there or not.<br>
- vdsm had configured both machines' libvirtd.conf, though slightly
differently.<br>
- I can see that some of the config above the vdsm section was
slightly different; I put this down to, maybe, I would usually yum
-y update before installing ovirt; I might have done it the other
way around; my libvirtds are the same version now, but could have
been different earlier.<br>
<br>
Here is what vdsm had added to ach libvirtd.conf<br>
<br>
host1:<br>
## beginning of configuration section by vdsm-4.13.0<br>
listen_addr="0.0.0.0"<br>
unix_sock_group="qemu"<br>
unix_sock_rw_perms="0770"<br>
auth_unix_rw="sasl"<br>
host_uuid="a9fd3fe1-b3a7-48f7-8870-cff3ee4c1f62"<br>
keepalive_interval=-1<br>
log_outputs="1:<a class="moz-txt-link-freetext" href="file:/var/log/libvirt/libvirtd.log">file:/var/log/libvirt/libvirtd.log</a>"<br>
log_filters="3:virobject 3:virfile 2:virnetlink 3:cgroup 3:event
3:json 1:libvirt 1:util 1:qemu"<br>
#auth_tcp="none"<br>
#MyComment#listen_tcp=0<br>
#MyComment#listen_tls=0<br>
## end of configuration section by vdsm-4.13.0<br>
<br>
host2:<br>
## beginning of configuration section by vdsm-4.13.0<br>
listen_addr="0.0.0.0"<br>
unix_sock_group="qemu"<br>
unix_sock_rw_perms="0770"<br>
auth_unix_rw="sasl"<br>
host_uuid="a1e1b804-1b88-4380-8ccc-7802b3a025bb"<br>
keepalive_interval=-1<br>
log_outputs="1:<a class="moz-txt-link-freetext" href="file:/var/log/libvirt/libvirtd.log">file:/var/log/libvirt/libvirtd.log</a>"<br>
log_filters="3:virobject 3:virfile 2:virnetlink 3:cgroup 3:event
3:json 1:libvirt 1:util 1:qemu"<br>
ca_file="/etc/pki/vdsm/certs/cacert.pem"<br>
cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"<br>
key_file="/etc/pki/vdsm/keys/vdsmkey.pem"<br>
## end of configuration section by vdsm-4.13.0<br>
<br>
Hope this helps someone else.<br>
<br>
cheers,<br>
<br>
<br>
<div class="moz-signature">-- <br>
<p class="MsoNormal"><font face="Verdana" size="2">
<span style="font-size:10.0pt;font-family:Verdana"><br>
<br>
<b>
<font color="#46166c"><span
style="color:#46166C;font-weight:bold"><!-- *** NAME *** -->Garry
Tiedemann<!-- *** END NAME *** --><br>
</span>
</font>
</b>
</span>
</font>
<font face="Verdana" size="1">
<span style="font-size:9.0pt;font-family:Verdana"><!-- *** TITLE *** -->IT
Manager<br>
<span style="font-family: Arial, Helvetica, sans-serif;
font-size: 8px; font-weight: bold;"> </span><br>
IT Division | <b><!-- *** END TITLE *** -->
<font color="#46166c"><span
style="color:#46166C;font-weight:bold">The Network
Group </span></font></b>| <a
href="http://maps.google.com.au/maps?q=334+Queensberry+St,+North+Melbourne,+Victoria,+3051&hl=en&sll=-37.813187,144.96298&sspn=2.111209,3.348083&vpsrc=0&z=17"><st1:address
w:st="on"><st1:street w:st="on">334 Queensberry St,
North</st1:street> <st1:city w:st="on">Melbourne</st1:city>,
<st1:state w:st="on">Victoria</st1:state></st1:address>,
3051, <st1:place w:st="on"><st1:country-region w:st="on">Australia</st1:country-region></st1:place></a><br>
<b><font color="#46166c"><span
style="color:#46166C;font-weight:bold">Phone</span></font></b>
<font color="#7f007f"><span style="color:#7F007F"> </span></font>(03)
9329 0933 |
<!-- *** DELETE *** --><b><font color="#46166c"><span
style="color:#46166C;font-weight:bold">Email </span></font></b><u>
<font color="blue">
<span style="color:blue"><!-- *** EMAIL *** --><a
href="mailto:garrytiedemann@networkvideo.com.au"><!-- *** END EMAIL *** --><!-- *** EMAIL ADDRESS *** -->
garrytiedemann@networkvideo.com.au<!-- *** END EMAIL ADDRESS *** --></a></span></font></u>
|
<!-- *** END DELETE *** --><b><font color="#46166c"><span
style="color:#46166C;font-weight:bold">Website</span></font></b><font
color="#50166c"><span style="color:#50166C"> </span></font><a
href="http://www.thenetworkgroup.com.au/">www.thenetworkgroup.com.au</a><br>
</span></font><font face="Verdana" size="1"><span
style="font-size:10.0pt;
font-family:Verdana"><br>
<img id="_x0000_i1025"
src="cid:part6.01060501.03000508@networkvideo.com.au"
alt="The Network Group email banner" width="700"
border="0" height="70"><br>
<span style="color:#46166C;font-weight:bold">The Network
Group: One of BRW's 50 most innovative companies for 2013,<br>
and winner of the 2013 Rental Group/Business of the Year
Award<br>
</span>
</span></font></p>
<table class="MsoTableGrid" style="border-collapse:collapse"
cellpadding="0" cellspacing="0" border="0">
<tbody>
<tr>
<td style="padding:0cm 5.4pt 0cm 5.4pt" valign="top"
width="680">
<p class="MsoNormal"><font face="Verdana" size="1"><span
style="font-size:7.0pt; font-family:Verdana"><br>
This email message and any accompanying attachments
may contain information that is confidential and is
subject to legal privilege. If you are not the
intended recipient, do not read, use, disseminate,
distribute or copy this message or attachments. If
you have received this message in error, please
notify the sender immediately and delete this
message. Network Video Home Entertainment Experts
accepts no liability for any damage caused by this
email or any electronic transmission. Any views
expressed in this message are those of the
individual sender, except where the sender
expressly, and with authority, states them to be the
views of 'Network Video Home Entertainment Experts'.
</span></font><font color="navy" face="Arial" size="2"><span
style="font-size:10.0pt;font-family:Arial;
color:navy"><o:p></o:p></span></font></p>
</td>
</tr>
</tbody>
</table>
<font face="Verdana" size="1">
</font><font face="Verdana" size="2"><span
style="font-size:10.0pt;font-family:Verdana"><br>
</span></font><br>
</div>
</body>
</html>