[ovirt-users] Problems migrating to host ID 1?

Garry Tiedemann garrytiedemann at networkvideo.com.au
Wed May 28 02:09:52 EDT 2014


Hi folks,

Whilst testing migration, I found an oddness that others might benefit 
from knowing about.

*My environment:*
Two hosts, CentOS underneath, Ovirt 3.4.1  hosted-engine (clean build) 
on top.

*What happened:*
Doing migration testing (it is so nice to have reached this stage), I 
found that VMs could migrate from the first host to the second, but not 
the other way.
To be clear, by "first host", I mean the first one that Ovirt was 
installed on. Host ID 1.

The Ovirt GUI's messages are uninformative (it failed; I knew that):
Migration failed due to Error: Fatal error during migration (VM:testvm1 
, Source: host2.ovirt, Destination:host1.ovirt ).

*Logs to look for:*
vdsm log on the source host (host 2) is informative, and includes this clue:
libvirtError: operation failed: Failed to connect to remote libvirt URI 
qemu+tls://10.0.14.9/system

which leads us to *libvirtd*. Let's have a look at it:

This is host 2, the one that's working, I can migrate to it:
[root at host2 ~]# grep -v ^# /etc/libvirt/libvirtd.conf | grep -v ^$
listen_addr="0.0.0.0"
unix_sock_group="qemu"
unix_sock_rw_perms="0770"
auth_unix_rw="sasl"
host_uuid="a1e1b804-1b88-4380-8ccc-7802b3a025bb"
keepalive_interval=-1
log_outputs="1:file:/var/log/libvirt/libvirtd.log"
log_filters="3:virobject 3:virfile 2:virnetlink 3:cgroup 3:event 3:json 
1:libvirt 1:util 1:qemu"
ca_file="/etc/pki/vdsm/certs/cacert.pem"
cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
key_file="/etc/pki/vdsm/keys/vdsmkey.pem"

[root at host2 ~]# lsof | grep libvirtd | grep TCP
libvirtd  22244      root   13u     IPv4 99175113          
0t0                  TCP *:16514 (LISTEN)

Now the one that's not working, with differences highlighted.

[root at host1 ~]# grep -v ^# /etc/libvirt/libvirtd.conf | grep -v ^$
*listen_tcp = 0*
*auth_tcp = "sasl"*
/# lines below here were added by vdsm/
listen_addr="0.0.0.0"
unix_sock_group="qemu"
unix_sock_rw_perms="0770"
auth_unix_rw="sasl"
host_uuid="a9fd3fe1-b3a7-48f7-8870-cff3ee4c1f62"
keepalive_interval=-1
log_outputs="1:file:/var/log/libvirt/libvirtd.log"
log_filters="3:virobject 3:virfile 2:virnetlink     3:cgroup 3:event 
3:json 1:libvirt 1:util 1:qemu"
*listen_tcp=0*
*listen_tls=0*
ca_file="/etc/pki/vdsm/certs/cacert.pem"
cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
key_file="/etc/pki/vdsm/keys/vdsmkey.pem"

For the sake of text mail clients, the different lines were:
*listen_tcp = 0*
*auth_tcp = "sasl"*
*listen_tcp=0*
*listen_tls=0*

and, owing to the listen_tcp=0 (yes, it appears twice in the config 
file), there was no libvirt tcp socket listening here.

Comment out those four lines to make it the same as the working one, 
restart libvirtd, and now I can migrate both ways.

WARNING: If you are in production already, make sure to migrate your VMs 
off host 1 BEFORE restarting libvirtd. I found that restarting libvirtd 
caused guest VMs to die, which doesn't bother me - I've no production 
VMs on there yet, though we're not far off, touch wood.

Now, the command to restart libvirtd:

[root at host1 ~]# initctl restart libvirtd
libvirtd start/running, process 16279
[root at host1 ~]# lsof | grep ^libvirtd | grep TCP
libvirtd  16279      root   13u     IPv4          104492880 
0t0                  TCP *:16514 (LISTEN)

At this point, host1 becomes unresponsive in the Ovirt GUI, and when it 
comes back all by itself, we can migrate to it. In the first couple of 
minutes, you might see a reinitializing message when you start 
migration; it'll queue, and it'll work.

End of that problem, for me at least.

Others who've more experience than I may critique the libvirtd config. 
This is the only manual intervention I've done with it.

For developers:
- I didn't have this problem with 3.3, but then, I didn't have 
hosted-engine then either.
- My 3.4.0 environment (my first hosted-engine) is mostly dead, so I 
can't say whether this problem was there or not.
- vdsm had configured both machines' libvirtd.conf, though slightly 
differently.
- I can see that some of the config above the vdsm section was slightly 
different; I put this down to, maybe, I would usually yum -y update 
before installing ovirt; I might have done it the other way around; my 
libvirtds are the same version now, but could have been different earlier.

Here is what vdsm had added to ach libvirtd.conf

host1:
## beginning of configuration section by vdsm-4.13.0
listen_addr="0.0.0.0"
unix_sock_group="qemu"
unix_sock_rw_perms="0770"
auth_unix_rw="sasl"
host_uuid="a9fd3fe1-b3a7-48f7-8870-cff3ee4c1f62"
keepalive_interval=-1
log_outputs="1:file:/var/log/libvirt/libvirtd.log"
log_filters="3:virobject 3:virfile 2:virnetlink     3:cgroup 3:event 
3:json 1:libvirt 1:util 1:qemu"
#auth_tcp="none"
#MyComment#listen_tcp=0
#MyComment#listen_tls=0
## end of configuration section by vdsm-4.13.0

host2:
## beginning of configuration section by vdsm-4.13.0
listen_addr="0.0.0.0"
unix_sock_group="qemu"
unix_sock_rw_perms="0770"
auth_unix_rw="sasl"
host_uuid="a1e1b804-1b88-4380-8ccc-7802b3a025bb"
keepalive_interval=-1
log_outputs="1:file:/var/log/libvirt/libvirtd.log"
log_filters="3:virobject 3:virfile 2:virnetlink     3:cgroup 3:event 
3:json 1:libvirt 1:util 1:qemu"
ca_file="/etc/pki/vdsm/certs/cacert.pem"
cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
## end of configuration section by vdsm-4.13.0

Hope this helps someone else.

cheers,


-- 



*Garry Tiedemann
* IT Manager

IT Division | *The Network Group *| 334 Queensberry St, North Melbourne, 
Victoria, 3051, Australia 
<http://maps.google.com.au/maps?q=334+Queensberry+St,+North+Melbourne,+Victoria,+3051&hl=en&sll=-37.813187,144.96298&sspn=2.111209,3.348083&vpsrc=0&z=17>
*Phone* (03) 9329 0933 | *Email *_garrytiedemann at networkvideo.com.au 
<mailto:garrytiedemann at networkvideo.com.au>_  | 
*Website*www.thenetworkgroup.com.au <http://www.thenetworkgroup.com.au/>

The Network Group email banner
The Network Group: One of BRW's 50 most innovative companies for 2013,
and winner of the 2013 Rental Group/Business of the Year Award


This email message and any accompanying attachments may contain 
information that is confidential and is subject to legal privilege. If 
you are not the intended recipient, do not read, use, disseminate, 
distribute or copy this message or attachments. If you have received 
this message in error, please notify the sender immediately and delete 
this message. Network Video Home Entertainment Experts accepts no 
liability for any damage caused by this email or any electronic 
transmission. Any views expressed in this message are those of the 
individual sender, except where the sender expressly, and with 
authority, states them to be the views of 'Network Video Home 
Entertainment Experts'.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20140528/d7fe75e2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2013_BRW_signature.gif
Type: image/gif
Size: 69864 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20140528/d7fe75e2/attachment-0001.gif>


More information about the Users mailing list