[ovirt-users] Problems migrating to host ID 1?
Garry Tiedemann
garrytiedemann at networkvideo.com.au
Wed May 28 02:09:52 EDT 2014
Hi folks,
Whilst testing migration, I found an oddness that others might benefit
from knowing about.
*My environment:*
Two hosts, CentOS underneath, Ovirt 3.4.1 hosted-engine (clean build)
on top.
*What happened:*
Doing migration testing (it is so nice to have reached this stage), I
found that VMs could migrate from the first host to the second, but not
the other way.
To be clear, by "first host", I mean the first one that Ovirt was
installed on. Host ID 1.
The Ovirt GUI's messages are uninformative (it failed; I knew that):
Migration failed due to Error: Fatal error during migration (VM:testvm1
, Source: host2.ovirt, Destination:host1.ovirt ).
*Logs to look for:*
vdsm log on the source host (host 2) is informative, and includes this clue:
libvirtError: operation failed: Failed to connect to remote libvirt URI
qemu+tls://10.0.14.9/system
which leads us to *libvirtd*. Let's have a look at it:
This is host 2, the one that's working, I can migrate to it:
[root at host2 ~]# grep -v ^# /etc/libvirt/libvirtd.conf | grep -v ^$
listen_addr="0.0.0.0"
unix_sock_group="qemu"
unix_sock_rw_perms="0770"
auth_unix_rw="sasl"
host_uuid="a1e1b804-1b88-4380-8ccc-7802b3a025bb"
keepalive_interval=-1
log_outputs="1:file:/var/log/libvirt/libvirtd.log"
log_filters="3:virobject 3:virfile 2:virnetlink 3:cgroup 3:event 3:json
1:libvirt 1:util 1:qemu"
ca_file="/etc/pki/vdsm/certs/cacert.pem"
cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
[root at host2 ~]# lsof | grep libvirtd | grep TCP
libvirtd 22244 root 13u IPv4 99175113
0t0 TCP *:16514 (LISTEN)
Now the one that's not working, with differences highlighted.
[root at host1 ~]# grep -v ^# /etc/libvirt/libvirtd.conf | grep -v ^$
*listen_tcp = 0*
*auth_tcp = "sasl"*
/# lines below here were added by vdsm/
listen_addr="0.0.0.0"
unix_sock_group="qemu"
unix_sock_rw_perms="0770"
auth_unix_rw="sasl"
host_uuid="a9fd3fe1-b3a7-48f7-8870-cff3ee4c1f62"
keepalive_interval=-1
log_outputs="1:file:/var/log/libvirt/libvirtd.log"
log_filters="3:virobject 3:virfile 2:virnetlink 3:cgroup 3:event
3:json 1:libvirt 1:util 1:qemu"
*listen_tcp=0*
*listen_tls=0*
ca_file="/etc/pki/vdsm/certs/cacert.pem"
cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
For the sake of text mail clients, the different lines were:
*listen_tcp = 0*
*auth_tcp = "sasl"*
*listen_tcp=0*
*listen_tls=0*
and, owing to the listen_tcp=0 (yes, it appears twice in the config
file), there was no libvirt tcp socket listening here.
Comment out those four lines to make it the same as the working one,
restart libvirtd, and now I can migrate both ways.
WARNING: If you are in production already, make sure to migrate your VMs
off host 1 BEFORE restarting libvirtd. I found that restarting libvirtd
caused guest VMs to die, which doesn't bother me - I've no production
VMs on there yet, though we're not far off, touch wood.
Now, the command to restart libvirtd:
[root at host1 ~]# initctl restart libvirtd
libvirtd start/running, process 16279
[root at host1 ~]# lsof | grep ^libvirtd | grep TCP
libvirtd 16279 root 13u IPv4 104492880
0t0 TCP *:16514 (LISTEN)
At this point, host1 becomes unresponsive in the Ovirt GUI, and when it
comes back all by itself, we can migrate to it. In the first couple of
minutes, you might see a reinitializing message when you start
migration; it'll queue, and it'll work.
End of that problem, for me at least.
Others who've more experience than I may critique the libvirtd config.
This is the only manual intervention I've done with it.
For developers:
- I didn't have this problem with 3.3, but then, I didn't have
hosted-engine then either.
- My 3.4.0 environment (my first hosted-engine) is mostly dead, so I
can't say whether this problem was there or not.
- vdsm had configured both machines' libvirtd.conf, though slightly
differently.
- I can see that some of the config above the vdsm section was slightly
different; I put this down to, maybe, I would usually yum -y update
before installing ovirt; I might have done it the other way around; my
libvirtds are the same version now, but could have been different earlier.
Here is what vdsm had added to ach libvirtd.conf
host1:
## beginning of configuration section by vdsm-4.13.0
listen_addr="0.0.0.0"
unix_sock_group="qemu"
unix_sock_rw_perms="0770"
auth_unix_rw="sasl"
host_uuid="a9fd3fe1-b3a7-48f7-8870-cff3ee4c1f62"
keepalive_interval=-1
log_outputs="1:file:/var/log/libvirt/libvirtd.log"
log_filters="3:virobject 3:virfile 2:virnetlink 3:cgroup 3:event
3:json 1:libvirt 1:util 1:qemu"
#auth_tcp="none"
#MyComment#listen_tcp=0
#MyComment#listen_tls=0
## end of configuration section by vdsm-4.13.0
host2:
## beginning of configuration section by vdsm-4.13.0
listen_addr="0.0.0.0"
unix_sock_group="qemu"
unix_sock_rw_perms="0770"
auth_unix_rw="sasl"
host_uuid="a1e1b804-1b88-4380-8ccc-7802b3a025bb"
keepalive_interval=-1
log_outputs="1:file:/var/log/libvirt/libvirtd.log"
log_filters="3:virobject 3:virfile 2:virnetlink 3:cgroup 3:event
3:json 1:libvirt 1:util 1:qemu"
ca_file="/etc/pki/vdsm/certs/cacert.pem"
cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
## end of configuration section by vdsm-4.13.0
Hope this helps someone else.
cheers,
--
*Garry Tiedemann
* IT Manager
IT Division | *The Network Group *| 334 Queensberry St, North Melbourne,
Victoria, 3051, Australia
<http://maps.google.com.au/maps?q=334+Queensberry+St,+North+Melbourne,+Victoria,+3051&hl=en&sll=-37.813187,144.96298&sspn=2.111209,3.348083&vpsrc=0&z=17>
*Phone* (03) 9329 0933 | *Email *_garrytiedemann at networkvideo.com.au
<mailto:garrytiedemann at networkvideo.com.au>_ |
*Website*www.thenetworkgroup.com.au <http://www.thenetworkgroup.com.au/>
The Network Group email banner
The Network Group: One of BRW's 50 most innovative companies for 2013,
and winner of the 2013 Rental Group/Business of the Year Award
This email message and any accompanying attachments may contain
information that is confidential and is subject to legal privilege. If
you are not the intended recipient, do not read, use, disseminate,
distribute or copy this message or attachments. If you have received
this message in error, please notify the sender immediately and delete
this message. Network Video Home Entertainment Experts accepts no
liability for any damage caused by this email or any electronic
transmission. Any views expressed in this message are those of the
individual sender, except where the sender expressly, and with
authority, states them to be the views of 'Network Video Home
Entertainment Experts'.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20140528/d7fe75e2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2013_BRW_signature.gif
Type: image/gif
Size: 69864 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20140528/d7fe75e2/attachment-0001.gif>
More information about the Users
mailing list