On Mon, Feb 25, 2019 at 7:15 PM Guillaume Pavese <guillaume.pavese@interactiv-group.com> wrote:

No, as indicated previously, still :

[root@vs-inf-int-kvm-fr-301-210 ~]# virsh -r net-dhcp-leases default

 Expiry Time          MAC address        Protocol  IP address                Hostname        Client ID or DUID

-------------------------------------------------------------------------------------------------------------------

[root@vs-inf-int-kvm-fr-301-210 ~]# 

I did not see any relevant log on the HE vm. Is there something I should look for there?

This smells really bad: I'd suggest to check /var/log/messages and /var/log/libvirt/qemu/HostedEngineLocal.log for libvirt errors;

if nothing is there can I ask you to try reexecuting with libvirt debug logs (edit /etc/libvirt/libvirtd.conf).

Honestly I'm not able to reproduce it on my side.

 

Guillaume Pavese
Ingénieur Système et Réseau

Interactiv-Group

On Tue, Feb 26, 2019 at 3:12 AM Simone Tiraboschi <stirabos@redhat.com> wrote:

On Mon, Feb 25, 2019 at 7:04 PM Guillaume Pavese <guillaume.pavese@interactiv-group.com> wrote:

I still can't connect with VNC remotely but locally with X forwarding it works.

However my connection has too high latency for that to be usable (I'm in Japan, my hosts in France, ~250 ms ping)

But I could see that the VM is booted!

and in Hosts logs there is :

févr. 25 18:51:12 vs-inf-int-kvm-fr-301-210.hostics.fr python[14719]: ansible-command Invoked with warn=True executable=None _uses_shell=True _raw_params=virsh -r net-dhcp-leases default | grep -i 00:16:3e:1d:4b:b6 | awk '{ print $5 }' | cut -f1 -d'/' removes=None argv=None creates=None chdir=None stdin=None

févr. 25 18:51:30 vs-inf-int-kvm-fr-301-210.hostics.fr dnsmasq-dhcp[6310]: DHCPDISCOVER(virbr0) 00:16:3e:1d:4b:b6

févr. 25 18:51:30 vs-inf-int-kvm-fr-301-210.hostics.fr dnsmasq-dhcp[6310]: DHCPOFFER(virbr0) 192.168.122.14 00:16:3e:1d:4b:b6

févr. 25 18:51:30 vs-inf-int-kvm-fr-301-210.hostics.fr dnsmasq-dhcp[6310]: DHCPREQUEST(virbr0) 192.168.122.14 00:16:3e:1d:4b:b6

févr. 25 18:51:30 vs-inf-int-kvm-fr-301-210.hostics.fr dnsmasq-dhcp[6310]: DHCPACK(virbr0) 192.168.122.14 00:16:3e:1d:4b:b6 vs-inf-int-ovt-fr-301-210

févr. 25 18:51:42 vs-inf-int-kvm-fr-301-210.hostics.fr python[14757]: ansible-command Invoked with warn=True executable=None _uses_shell=True _raw_params=virsh -r net-dhcp-leases default | grep -i 00:16:3e:1d:4b:b6 | awk '{ print $5 }' | cut -f1 -d'/' removes=None argv=None creates=None chdir=None stdin=None

févr. 25 18:52:12 vs-inf-int-kvm-fr-301-210.hostics.fr python[14789]: ansible-command Invoked with warn=True executable=None _uses_shell=True _raw_params=virsh -r net-dhcp-leases default | grep -i 00:16:3e:1d:4b:b6 | awk '{ print $5 }' | cut -f1 -d'/' removes=None argv=None creates=None chdir=None stdin=None

févr. 25 18:52:43 vs-inf-int-kvm-fr-301-210.hostics.fr python[14818]: ansible-command Invoked with warn=True executable=None _uses_shell=True _raw_params=virsh -r net-dhcp-leases default | grep -i 00:16:3e:1d:4b:b6 | awk '{ print $5 }' | cut -f1 -d'/' removes=None argv=None creates=None chdir=None stdin=None

....

ssh to the vm works too : 

[root@vs-inf-int-kvm-fr-301-210 ~]# ssh root@192.168.122.14

The authenticity of host '192.168.122.14 (192.168.122.14)' can't be established.

ECDSA key fingerprint is SHA256:+/pUzTGVA4kCyICb7XgqrxWYYkqzmDjVmdAahiBFgOQ.

ECDSA key fingerprint is MD5:4b:ef:ff:4a:7c:1a:af:c2:af:4a:0f:14:a3:c5:31:fb.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '192.168.122.14' (ECDSA) to the list of known hosts.

root@192.168.122.14's password: 

[root@vs-inf-int-ovt-fr-301-210 ~]# 

But the test that the playbook tries still fails with empty result :

[root@vs-inf-int-kvm-fr-301-210 ~]# virsh -r net-dhcp-leases default

 Expiry Time          MAC address        Protocol  IP address                Hostname        Client ID or DUID

-------------------------------------------------------------------------------------------------------------------

[root@vs-inf-int-kvm-fr-301-210 ~]# 

This smells like a bug to me:

and nothing at all in the output of

virsh -r net-dhcp-leases default

?

 

Guillaume Pavese
Ingénieur Système et Réseau

Interactiv-Group

On Tue, Feb 26, 2019 at 1:54 AM Simone Tiraboschi <stirabos@redhat.com> wrote:

On Mon, Feb 25, 2019 at 5:50 PM Guillaume Pavese <guillaume.pavese@interactiv-group.com> wrote:

I did that but no success yet.

I see that "Get local VM IP" task tries the following :

virsh -r net-dhcp-leases default | grep -i {{ he_vm_mac_addr }} | awk '{ print $5 }' | cut -f1 -d'/'

However while the task is running, and vm running in qemu, "virsh -r net-dhcp-leases default" never returns anything :

Yes, I think that libvirt will never provide a DHCP lease since the appliance OS never correctly complete the boot process.

I'd suggest to connect to the running VM via vnc DURING the boot process and check what's wrong.

 

[root@vs-inf-int-kvm-fr-301-210 ~]# virsh -r net-dhcp-leases default

 Expiry Time          MAC address        Protocol  IP address                Hostname        Client ID or DUID

-------------------------------------------------------------------------------------------------------------------

[root@vs-inf-int-kvm-fr-301-210 ~]# 

Guillaume Pavese
Ingénieur Système et Réseau

Interactiv-Group

On Tue, Feb 26, 2019 at 12:33 AM Simone Tiraboschi <stirabos@redhat.com> wrote:

OK, try this:

temporary edit /usr/share/ansible/roles/ovirt.hosted_engine_setup/tasks/bootstrap_local_vm/02_create_local_vm.yml around line 120

and edit tasks "Get local VM IP"

changing from "retries: 50" to  "retries: 500" so that you have more time to debug it

On Mon, Feb 25, 2019 at 4:20 PM Guillaume Pavese <guillaume.pavese@interactiv-group.com> wrote:

I retried after killing the remaining qemu process and doing ovirt-hosted-engine-cleanup

The new attempt failed again at the same step. Then after it fails, it cleans the temporary files (and vm disk) but qemu still runs! :

[ INFO  ] TASK [ovirt.hosted_engine_setup : Get local VM IP]

[ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:6c:e8:f9 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.092436", "end": "2019-02-25 16:09:38.863263", "rc": 0, "start": "2019-02-25 16:09:38.770827", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

[ INFO  ] TASK [ovirt.hosted_engine_setup : include_tasks]

[ INFO  ] ok: [localhost]

[ INFO  ] TASK [ovirt.hosted_engine_setup : Remove local vm dir]

[ INFO  ] changed: [localhost]

[ INFO  ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM]

[ INFO  ] ok: [localhost]

[ INFO  ] TASK [ovirt.hosted_engine_setup : Notify the user about a failure]

[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"}

[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook

[ INFO  ] Stage: Clean up

[ INFO  ] Cleaning temporary resources

...

[ INFO  ] TASK [ovirt.hosted_engine_setup : Remove local vm dir]

[ INFO  ] ok: [localhost]

[ INFO  ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM]

[ INFO  ] ok: [localhost]

[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20190225161011.conf'

[ INFO  ] Stage: Pre-termination

[ INFO  ] Stage: Termination

[ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.

[root@vs-inf-int-kvm-fr-301-210 ~]# ps aux | grep qemu

root      4021  0.0  0.0  24844  1788 ?        Ss   févr.22   0:00 /usr/bin/qemu-ga --method=virtio-serial --path=/dev/virtio-ports/org.qemu.guest_agent.0 --blacklist=guest-file-open,guest-file-close,guest-file-read,guest-file-write,guest-file-seek,guest-file-flush,guest-exec,guest-exec-status -F/etc/qemu-ga/fsfreeze-hook

qemu     26463 22.9  4.8 17684512 1088844 ?    Sl   16:01   3:09 /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu Haswell-noTSX,+kvmclock -m 16384 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 316eca5f-81de-4a0b-af1f-58f910402a8e -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=27,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=off,strict=on -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/var/tmp/localvmdRIozH/images/15023c8a-e3a7-4851-a97d-3b90996b423b/07fdcff3-11ce-4f7c-af05-0a878593e78e,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/tmp/localvmdRIozH/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:6c:e8:f9,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=31,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -vnc 127.0.0.1:0 -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

root     28416  0.0  0.0 112712   980 pts/3    S+   16:14   0:00 grep --color=auto qemu

Before the first Error, while the vm was running for sure and the disk was there, I also unsuccessfuly tried to connect to it with VNC  and got the same error I got before : 

[root@vs-inf-int-kvm-fr-301-210 ~]# debug1: Connection to port 5900 forwarding to vs-inf-int-kvm-fr-301-210.hostics.fr port 5900 requested.

debug1: channel 3: new [direct-tcpip]

channel 3: open failed: connect failed: Connection refused

debug1: channel 3: free: direct-tcpip: listening port 5900 for vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from 127.0.0.1 port 37002 to 127.0.0.1 port 5900, nchannels 4

Guillaume Pavese
Ingénieur Système et Réseau

Interactiv-Group

On Mon, Feb 25, 2019 at 11:57 PM Guillaume Pavese <guillaume.pavese@interactiv-group.com> wrote:

Something was definitely wrong ; as indicated, qemu process for guest=HostedEngineLocal was running but the disk file did not exist anymore...

No surprise I could not connect

I am retrying

Guillaume Pavese
Ingénieur Système et Réseau

Interactiv-Group

On Mon, Feb 25, 2019 at 11:15 PM Guillaume Pavese <guillaume.pavese@interactiv-group.com> wrote:

It fails too :

I made sure PermitTunnel=yes in sshd config but when I try to connect to the forwarded port I get the following error on the openened host ssh session :

[gpavese@sheepora-X230 ~]$ ssh -v -L 5900:vs-inf-int-kvm-fr-301-210.hostics.fr:5900 root@vs-inf-int-kvm-fr-301-210.hostics.fr

...

[root@vs-inf-int-kvm-fr-301-210 ~]#

debug1: channel 3: free: direct-tcpip: listening port 5900 for vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from ::1 port 42144 to ::1 port 5900, nchannels 4

debug1: Connection to port 5900 forwarding to vs-inf-int-kvm-fr-301-210.hostics.fr port 5900 requested.

debug1: channel 3: new [direct-tcpip]

channel 3: open failed: connect failed: Connection refused

debug1: channel 3: free: direct-tcpip: listening port 5900 for vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from 127.0.0.1 port 32778 to 127.0.0.1 port 5900, nchannels 4

and in journalctl : 

févr. 25 14:55:38 vs-inf-int-kvm-fr-301-210.hostics.fr sshd[19595]: error: connect_to vs-inf-int-kvm-fr-301-210.hostics.fr port 5900: failed.

Guillaume Pavese
Ingénieur Système et Réseau

Interactiv-Group

On Mon, Feb 25, 2019 at 10:44 PM Simone Tiraboschi <stirabos@redhat.com> wrote:

On Mon, Feb 25, 2019 at 2:35 PM Guillaume Pavese <guillaume.pavese@interactiv-group.com> wrote:

I made sure of everything and even stopped firewalld but still can't connect :

[root@vs-inf-int-kvm-fr-301-210 ~]# cat /var/run/libvirt/qemu/HostedEngineLocal.xml

     <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>

        <listen type='address' address='127.0.0.1' fromConfig='1' autoGenerated='no'/>

[root@vs-inf-int-kvm-fr-301-210 ~]# netstat -pan | grep 59

tcp        0      0 127.0.0.1:5900          0.0.0.0:*               LISTEN      13376/qemu-kvm

I suggest to try ssh tunneling, run
ssh -L 5900:vs-inf-int-kvm-fr-301-210.hostics.fr:5900 root@vs-inf-int-kvm-fr-301-210.hostics.fr

and then

remote-viewer vnc://localhost:5900
 

 

[root@vs-inf-int-kvm-fr-301-210 ~]# systemctl status firewalld.service 

● firewalld.service - firewalld - dynamic firewall daemon

   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)

   Active: inactive (dead)

févr. 25 14:24:03 vs-inf-int-kvm-fr-301-210.hostics.fr systemd[1]: Stopped firewalld - dynamic firewall daemon.

From my laptop :

[gpavese@sheepora-X230 ~]$ telnet vs-inf-int-kvm-fr-301-210.hostics.fr 5900

Trying 10.199.210.11...

[nothing gets through...]

^C

For making sure :

[gpavese@sheepora-X230 ~]$ telnet vs-inf-int-kvm-fr-301-210.hostics.fr 9090

Trying 10.199.210.11...

Connected to vs-inf-int-kvm-fr-301-210.hostics.fr.

Escape character is '^]'.

Guillaume Pavese
Ingénieur Système et Réseau

Interactiv-Group

On Mon, Feb 25, 2019 at 10:24 PM Parth Dhanjal <dparth@redhat.com> wrote:

Hey!

You can check under /var/run/libvirt/qemu/HostedEngine.xml
Search for 'vnc'
From there you can look up the port on which the HE VM is available and connect to the same.

On Mon, Feb 25, 2019 at 6:47 PM Guillaume Pavese <guillaume.pavese@interactiv-group.com> wrote:

1) I am running in a Nested env, but under libvirt/kvm on remote Centos 7.4 Hosts

Please advise how to connect with VNC to the local HE vm. I see it's running, but this is on a remote host, not my local machine :

qemu     13376  100  3.7 17679424 845216 ?     Sl   12:46  85:08 /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu Haswell-noTSX,+kvmclock -m 16384 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 6fe7c1c3-ea93-4343-a385-0d9e14bb563a -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=27,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=off,strict=on -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/var/tmp/localvmgmyYik/images/15023c8a-e3a7-4851-a97d-3b90996b423b/07fdcff3-11ce-4f7c-af05-0a878593e78e,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/tmp/localvmgmyYik/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:3e:fe:28,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=31,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -vnc 127.0.0.1:0 -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

2) [root@vs-inf-int-kvm-fr-301-210 ~]# cat /etc/libvirt/qemu/networks/default.xml

<!--

WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE

OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:

  virsh net-edit default

or other application using the libvirt API.

-->

<network>

  <name>default</name>

  <uuid>ba7bbfc8-28b8-459e-a42d-c2d6218e2cb6</uuid>

  <forward mode='nat'/>

  <bridge name='virbr0' stp='on' delay='0'/>

  <mac address='52:54:00:e5:fe:3b'/>

  <ip address='192.168.122.1' netmask='255.255.255.0'>

    <dhcp>

      <range start='192.168.122.2' end='192.168.122.254'/>

    </dhcp>

  </ip>

</network>

You have new mail in /var/spool/mail/root

[root@vs-inf-int-kvm-fr-301-210 ~]

Guillaume Pavese
Ingénieur Système et Réseau

Interactiv-Group

On Mon, Feb 25, 2019 at 9:57 PM Simone Tiraboschi <stirabos@redhat.com> wrote:

On Mon, Feb 25, 2019 at 1:14 PM Guillaume Pavese <guillaume.pavese@interactiv-group.com> wrote:

He deployment with "hosted-engine --deploy" fails at TASK [ovirt.hosted_engine_setup : Get local VM IP]

See following Error :

2019-02-25 12:46:50,154+0100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup : Get local VM IP]

2019-02-25 12:55:26,823+0100 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:103 {u'_ansible_parsed': True, u'stderr_lines': [], u'cmd': u"virsh -r net-dhcp-leases default | grep -i 00

:16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", u'end': u'2019-02-25 12:55:26.666925', u'_ansible_no_log': False, u'stdout': u'', u'changed': True, u'invocation': {u'module_args': {u'warn': True, u'executable':

None, u'_uses_shell': True, u'_raw_params': u"virsh -r net-dhcp-leases default | grep -i 00:16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'std

in': None}}, u'start': u'2019-02-25 12:55:26.584686', u'attempts': 50, u'stderr': u'', u'rc': 0, u'delta': u'0:00:00.082239', u'stdout_lines': []}

2019-02-25 12:55:26,924+0100 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:107 fatal: [localhost]: FAILED! => {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.082239", "end": "2019-02-25 12:55:26.666925", "rc": 0, "start": "2019-02-25 12:55:26.584686", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

Here we are just waiting for the bootstrap engine VM to fetch an IP address from default libvirt network over DHCP but it your case it never happened.

Possible issues: something went wrong in the bootstrap process for the engine VM or the default libvirt network is not correctly configured.

1. can you try to reach the engine VM via VNC and check what's happening there? (another question, are you running it nested? AFAIK it will not work if nested over ESXi)

2. can you please share the output of 

cat /etc/libvirt/qemu/networks/default.xml

 

Guillaume Pavese
Ingénieur Système et Réseau

Interactiv-Group

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VXRMU3SQWTMB2YYNMOMD7I5NX7RZQ2IW/

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/45UR44ITQTV7YVE6GLJWEV2AV22TIGXK/