<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 11/06/2015 02:02 PM, Simone
Tiraboschi wrote:<br>
</div>
<blockquote
cite="mid:CAN8-ONp4EyCOCgGCHrqWD4Eu_F4NgGXjB-NcoeuM8P9NYDrC9g@mail.gmail.com"
type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Nov 6, 2015 at 5:21 PM, Jason
Keltz <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:jas@cse.yorku.ca" target="_blank">jas@cse.yorku.ca</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi.<br>
<br>
Last night, I upgraded my engine from 3.5 to 3.6. That
went flawlessly.<br>
Today, I'm trying to upgrade the vdsm on the hosts from
3.5 to 3.6 (along with applying other RHEL7.1 updates)
However, when I'm trying to put each host into maintenance
mode, and migrations start to occur, they all seem to FAIL
now! Even worse, when they fail, it leaves the hosts
DOWN! If there's a failure, I'd expect the host to simply
abort the migration.... Any help in debugging this would
be VERY much appreciated!<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
2015-11-06 10:09:16,065 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-4) [] Correlation ID:
658ba478, Job ID: 524e8c44-04e0-42d3-89f9-9f4e4d397583,
Call Stack: null, Custom Event ID: -1, Message:
Migration failed (VM: eportfolio, Source: virt1).<br>
2015-11-06 10:10:17,112 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-22) [2f0dee16]
Correlation ID: 7da3ac1b, Job ID:
93c0b1f2-4c8e-48cf-9e63-c1ba91be425f, Call Stack: null,
Custom Event ID: -1, Message: Migration failed (VM:
ftp1, Source: virt1).<br>
2015-11-06 10:15:08,273 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-45) [] Correlation ID:
5394ef76, Job ID: 994065fc-a142-4821-934a-c2297d86ec12,
Call Stack: null, Custom Event ID: -1, Message:
Migration failed while Host is in 'preparing for
maintenance' state.<br>
2015-11-06 10:19:13,712 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-36) [] Correlation ID:
6e422728, Job ID: 994065fc-a142-4821-934a-c2297d86ec12,
Call Stack: null, Custom Event ID: -1, Message:
Migration failed while Host is in 'preparing for
maintenance' state.<br>
2015-11-06 10:42:37,852 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-12) [] Correlation ID:
e7f6300, Job ID: 1ea16622-0fa0-4e92-89e5-9dc235c03ef8,
Call Stack: null, Custom Event ID: -1, Message:
Migration failed (VM: ipa, Source: virt1).<br>
2015-11-06 10:43:59,732 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-40) [] Correlation ID:
39cfdf9, Job ID: 72be29bc-a02b-4a90-b5ec-8b995c2fa692,
Call Stack: null, Custom Event ID: -1, Message:
Migration failed (VM: labtesteval, Source: virt1).<br>
2015-11-06 10:52:11,893 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-23) [] Correlation ID:
5c435149, Job ID: 1dcd1e14-baa6-44bc-a853-5d33107b759c,
Call Stack: null, Custom Event ID: -1, Message:
Migration failed (VM: www-vhost, Source: virt1).<br>
</blockquote>
<br>
<br>
The complete engine log, virt1, virt2, and virt3 vdsm logs
are here:<br>
<br>
<a moz-do-not-send="true"
href="http://www.eecs.yorku.ca/%7Ejas/ovirt-debug/11062015"
rel="noreferrer" target="_blank">http://www.eecs.yorku.ca/~jas/ovirt-debug/11062015</a><br>
<br>
</blockquote>
<div><br>
</div>
<div>Is vdsmd service still active on that hosts?</div>
<div> </div>
</div>
</div>
</div>
</blockquote>
<br>
Hi Simone..<br>
<br>
Yes..<br>
<br>
virt1:<br>
sh-4.2# systemctl -l status vdsmd<br>
vdsmd.service - Virtual Desktop Server Manager<br>
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)<br>
Active: active (running) since Thu 2015-11-05 22:47:46 EST; 15h
ago<br>
Main PID: 16520 (vdsm)<br>
CGroup: /system.slice/vdsmd.service<br>
├─16520 /usr/bin/python /usr/share/vdsm/vdsm<br>
├─30038 /usr/libexec/ioprocess --read-pipe-fd 67
--write-pipe-fd 66 --max-threads 10 --max-queued-requests 10<br>
├─30055 /usr/libexec/ioprocess --read-pipe-fd 76
--write-pipe-fd 75 --max-threads 10 --max-queued-requests 10<br>
└─30062 /usr/libexec/ioprocess --read-pipe-fd 81
--write-pipe-fd 84 --max-threads 10 --max-queued-requests 10<br>
<br>
Nov 06 10:09:15 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.com.redhat.rhevm.vdsm
already removed<br>
Nov 06 10:09:15 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.org.qemu.guest_agent.0
already removed<br>
Nov 06 10:10:15 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/aa487207-7ff4-465a-9d9b-2a103d50dc77.com.redhat.rhevm.vdsm
already removed<br>
Nov 06 10:10:15 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/aa487207-7ff4-465a-9d9b-2a103d50dc77.org.qemu.guest_agent.0
already removed<br>
Nov 06 10:42:36 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8.com.redhat.rhevm.vdsm
already removed<br>
Nov 06 10:42:36 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8.org.qemu.guest_agent.0
already removed<br>
Nov 06 10:43:57 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/77766cb4-0625-4432-a16f-def5e702102a.com.redhat.rhevm.vdsm
already removed<br>
Nov 06 10:43:57 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/77766cb4-0625-4432-a16f-def5e702102a.org.qemu.guest_agent.0
already removed<br>
Nov 06 10:52:08 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/3bb83ad5-dc00-445d-8708-2db18c53e0e6.com.redhat.rhevm.vdsm
already removed<br>
Nov 06 10:52:08 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/3bb83ad5-dc00-445d-8708-2db18c53e0e6.org.qemu.guest_agent.0
already removed<br>
<br>
virt2:<br>
<br>
# systemctl -l status vdsmd<br>
vdsmd.service - Virtual Desktop Server Manager<br>
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)<br>
Active: active (running) since Fri 2015-11-06 10:40:36 EST; 3h
55min ago<br>
Process: 5035 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
--pre-start (code=exited, status=0/SUCCESS)<br>
Main PID: 6427 (vdsm)<br>
CGroup: /system.slice/vdsmd.service<br>
├─ 6334 /usr/libexec/ioprocess --read-pipe-fd 39
--write-pipe-fd 38 --max-threads 10 --max-queued-requests 10<br>
├─ 6342 /usr/libexec/ioprocess --read-pipe-fd 48
--write-pipe-fd 46 --max-threads 10 --max-queued-requests 10<br>
├─ 6354 /usr/libexec/ioprocess --read-pipe-fd 55
--write-pipe-fd 54 --max-threads 10 --max-queued-requests 10<br>
├─ 6427 /usr/bin/python /usr/share/vdsm/vdsm<br>
├─15071 /usr/libexec/ioprocess --read-pipe-fd 65
--write-pipe-fd 63 --max-threads 10 --max-queued-requests 10<br>
├─15125 /usr/libexec/ioprocess --read-pipe-fd 73
--write-pipe-fd 72 --max-threads 10 --max-queued-requests 10<br>
└─15140 /usr/libexec/ioprocess --read-pipe-fd 83
--write-pipe-fd 82 --max-threads 10 --max-queued-requests 10<br>
<br>
Nov 06 10:40:37 virt2.eecs.yorku.ca python[6427]: DIGEST-MD5
parse_server_challenge()<br>
Nov 06 10:40:37 virt2.eecs.yorku.ca python[6427]: DIGEST-MD5
ask_user_info()<br>
Nov 06 10:40:37 virt2.eecs.yorku.ca python[6427]: DIGEST-MD5
make_client_response()<br>
Nov 06 10:40:37 virt2.eecs.yorku.ca python[6427]: DIGEST-MD5 client
step 3<br>
Nov 06 10:42:34 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8.com.redhat.rhevm.vdsm
already removed<br>
Nov 06 10:42:34 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8.org.qemu.guest_agent.0
already removed<br>
Nov 06 10:43:56 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/77766cb4-0625-4432-a16f-def5e702102a.com.redhat.rhevm.vdsm
already removed<br>
Nov 06 10:43:56 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/77766cb4-0625-4432-a16f-def5e702102a.org.qemu.guest_agent.0
already removed<br>
Nov 06 10:52:08 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/3bb83ad5-dc00-445d-8708-2db18c53e0e6.com.redhat.rhevm.vdsm
already removed<br>
Nov 06 10:52:08 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/3bb83ad5-dc00-445d-8708-2db18c53e0e6.org.qemu.guest_agent.0
already removed<br>
<br>
virt3:<br>
<br>
<br>
# systemctl -l status vdsmd<br>
vdsmd.service - Virtual Desktop Server Manager<br>
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)<br>
Active: active (running) since Fri 2015-11-06 10:06:28 EST; 4h
30min ago<br>
Process: 12442 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh
--post-stop (code=exited, status=0/SUCCESS)<br>
Process: 12446 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
--pre-start (code=exited, status=0/SUCCESS)<br>
Main PID: 12561 (vdsm)<br>
CGroup: /system.slice/vdsmd.service<br>
├─ 9066 /usr/libexec/ioprocess --read-pipe-fd 40
--write-pipe-fd 36 --max-threads 10 --max-queued-requests 10<br>
├─ 9074 /usr/libexec/ioprocess --read-pipe-fd 48
--write-pipe-fd 47 --max-threads 10 --max-queued-requests 10<br>
├─ 9085 /usr/libexec/ioprocess --read-pipe-fd 57
--write-pipe-fd 56 --max-threads 10 --max-queued-requests 10<br>
├─12561 /usr/bin/python /usr/share/vdsm/vdsm<br>
├─12726 /usr/libexec/ioprocess --read-pipe-fd 65
--write-pipe-fd 63 --max-threads 10 --max-queued-requests 10<br>
├─12744 /usr/libexec/ioprocess --read-pipe-fd 72
--write-pipe-fd 74 --max-threads 10 --max-queued-requests 10<br>
└─12750 /usr/libexec/ioprocess --read-pipe-fd 83
--write-pipe-fd 82 --max-threads 10 --max-queued-requests 10<br>
<br>
Nov 06 10:09:15 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.com.redhat.rhevm.vdsm
already removed<br>
Nov 06 10:09:15 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.org.qemu.guest_agent.0
already removed<br>
Nov 06 10:10:17 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/aa487207-7ff4-465a-9d9b-2a103d50dc77.com.redhat.rhevm.vdsm
already removed<br>
Nov 06 10:10:17 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/aa487207-7ff4-465a-9d9b-2a103d50dc77.org.qemu.guest_agent.0
already removed<br>
Nov 06 10:15:05 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/d66235b4-7c8d-4da2-a572-82ecc9c53a59.com.redhat.rhevm.vdsm
already removed<br>
Nov 06 10:15:05 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/d66235b4-7c8d-4da2-a572-82ecc9c53a59.org.qemu.guest_agent.0
already removed<br>
Nov 06 10:19:13 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/c64d80f9-a3bd-44f3-89c4-f1b9b509cdde.com.redhat.rhevm.vdsm
already removed<br>
Nov 06 10:19:13 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/c64d80f9-a3bd-44f3-89c4-f1b9b509cdde.org.qemu.guest_agent.0
already removed<br>
Nov 06 10:42:56 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.com.redhat.rhevm.vdsm
already removed<br>
Nov 06 10:42:56 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING
File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.org.qemu.guest_agent.0
already removed<br>
<br>
--- I don't know what those warnings mean...<br>
<br>
Jason.<br>
<br>
<br>
<br>
</body>
</html>