[ovirt-users] upgrade from 3.5 to 3.6 causing problems with migration
Jason Keltz
jas at cse.yorku.ca
Fri Nov 6 19:37:56 UTC 2015
On 11/06/2015 02:02 PM, Simone Tiraboschi wrote:
>
>
> On Fri, Nov 6, 2015 at 5:21 PM, Jason Keltz <jas at cse.yorku.ca
> <mailto:jas at cse.yorku.ca>> wrote:
>
> Hi.
>
> Last night, I upgraded my engine from 3.5 to 3.6. That went
> flawlessly.
> Today, I'm trying to upgrade the vdsm on the hosts from 3.5 to 3.6
> (along with applying other RHEL7.1 updates) However, when I'm
> trying to put each host into maintenance mode, and migrations
> start to occur, they all seem to FAIL now! Even worse, when they
> fail, it leaves the hosts DOWN! If there's a failure, I'd expect
> the host to simply abort the migration.... Any help in debugging
> this would be VERY much appreciated!
>
> 2015-11-06 10:09:16,065 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (org.ovirt.thread.pool-8-thread-4) [] Correlation ID:
> 658ba478, Job ID: 524e8c44-04e0-42d3-89f9-9f4e4d397583, Call
> Stack: null, Custom Event ID: -1, Message: Migration failed
> (VM: eportfolio, Source: virt1).
> 2015-11-06 10:10:17,112 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (org.ovirt.thread.pool-8-thread-22) [2f0dee16] Correlation ID:
> 7da3ac1b, Job ID: 93c0b1f2-4c8e-48cf-9e63-c1ba91be425f, Call
> Stack: null, Custom Event ID: -1, Message: Migration failed
> (VM: ftp1, Source: virt1).
> 2015-11-06 10:15:08,273 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (org.ovirt.thread.pool-8-thread-45) [] Correlation ID:
> 5394ef76, Job ID: 994065fc-a142-4821-934a-c2297d86ec12, Call
> Stack: null, Custom Event ID: -1, Message: Migration failed
> while Host is in 'preparing for maintenance' state.
> 2015-11-06 10:19:13,712 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (org.ovirt.thread.pool-8-thread-36) [] Correlation ID:
> 6e422728, Job ID: 994065fc-a142-4821-934a-c2297d86ec12, Call
> Stack: null, Custom Event ID: -1, Message: Migration failed
> while Host is in 'preparing for maintenance' state.
> 2015-11-06 10:42:37,852 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (org.ovirt.thread.pool-8-thread-12) [] Correlation ID:
> e7f6300, Job ID: 1ea16622-0fa0-4e92-89e5-9dc235c03ef8, Call
> Stack: null, Custom Event ID: -1, Message: Migration failed
> (VM: ipa, Source: virt1).
> 2015-11-06 10:43:59,732 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (org.ovirt.thread.pool-8-thread-40) [] Correlation ID:
> 39cfdf9, Job ID: 72be29bc-a02b-4a90-b5ec-8b995c2fa692, Call
> Stack: null, Custom Event ID: -1, Message: Migration failed
> (VM: labtesteval, Source: virt1).
> 2015-11-06 10:52:11,893 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (org.ovirt.thread.pool-8-thread-23) [] Correlation ID:
> 5c435149, Job ID: 1dcd1e14-baa6-44bc-a853-5d33107b759c, Call
> Stack: null, Custom Event ID: -1, Message: Migration failed
> (VM: www-vhost, Source: virt1).
>
>
>
> The complete engine log, virt1, virt2, and virt3 vdsm logs are here:
>
> http://www.eecs.yorku.ca/~jas/ovirt-debug/11062015
> <http://www.eecs.yorku.ca/%7Ejas/ovirt-debug/11062015>
>
>
> Is vdsmd service still active on that hosts?
Hi Simone..
Yes..
virt1:
sh-4.2# systemctl -l status vdsmd
vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)
Active: active (running) since Thu 2015-11-05 22:47:46 EST; 15h ago
Main PID: 16520 (vdsm)
CGroup: /system.slice/vdsmd.service
├─16520 /usr/bin/python /usr/share/vdsm/vdsm
├─30038 /usr/libexec/ioprocess --read-pipe-fd 67
--write-pipe-fd 66 --max-threads 10 --max-queued-requests 10
├─30055 /usr/libexec/ioprocess --read-pipe-fd 76
--write-pipe-fd 75 --max-threads 10 --max-queued-requests 10
└─30062 /usr/libexec/ioprocess --read-pipe-fd 81
--write-pipe-fd 84 --max-threads 10 --max-queued-requests 10
Nov 06 10:09:15 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.com.redhat.rhevm.vdsm
already removed
Nov 06 10:09:15 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.org.qemu.guest_agent.0
already removed
Nov 06 10:10:15 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/aa487207-7ff4-465a-9d9b-2a103d50dc77.com.redhat.rhevm.vdsm
already removed
Nov 06 10:10:15 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/aa487207-7ff4-465a-9d9b-2a103d50dc77.org.qemu.guest_agent.0
already removed
Nov 06 10:42:36 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8.com.redhat.rhevm.vdsm
already removed
Nov 06 10:42:36 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8.org.qemu.guest_agent.0
already removed
Nov 06 10:43:57 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/77766cb4-0625-4432-a16f-def5e702102a.com.redhat.rhevm.vdsm
already removed
Nov 06 10:43:57 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/77766cb4-0625-4432-a16f-def5e702102a.org.qemu.guest_agent.0
already removed
Nov 06 10:52:08 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/3bb83ad5-dc00-445d-8708-2db18c53e0e6.com.redhat.rhevm.vdsm
already removed
Nov 06 10:52:08 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/3bb83ad5-dc00-445d-8708-2db18c53e0e6.org.qemu.guest_agent.0
already removed
virt2:
# systemctl -l status vdsmd
vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)
Active: active (running) since Fri 2015-11-06 10:40:36 EST; 3h 55min ago
Process: 5035 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
--pre-start (code=exited, status=0/SUCCESS)
Main PID: 6427 (vdsm)
CGroup: /system.slice/vdsmd.service
├─ 6334 /usr/libexec/ioprocess --read-pipe-fd 39
--write-pipe-fd 38 --max-threads 10 --max-queued-requests 10
├─ 6342 /usr/libexec/ioprocess --read-pipe-fd 48
--write-pipe-fd 46 --max-threads 10 --max-queued-requests 10
├─ 6354 /usr/libexec/ioprocess --read-pipe-fd 55
--write-pipe-fd 54 --max-threads 10 --max-queued-requests 10
├─ 6427 /usr/bin/python /usr/share/vdsm/vdsm
├─15071 /usr/libexec/ioprocess --read-pipe-fd 65
--write-pipe-fd 63 --max-threads 10 --max-queued-requests 10
├─15125 /usr/libexec/ioprocess --read-pipe-fd 73
--write-pipe-fd 72 --max-threads 10 --max-queued-requests 10
└─15140 /usr/libexec/ioprocess --read-pipe-fd 83
--write-pipe-fd 82 --max-threads 10 --max-queued-requests 10
Nov 06 10:40:37 virt2.eecs.yorku.ca python[6427]: DIGEST-MD5
parse_server_challenge()
Nov 06 10:40:37 virt2.eecs.yorku.ca python[6427]: DIGEST-MD5 ask_user_info()
Nov 06 10:40:37 virt2.eecs.yorku.ca python[6427]: DIGEST-MD5
make_client_response()
Nov 06 10:40:37 virt2.eecs.yorku.ca python[6427]: DIGEST-MD5 client step 3
Nov 06 10:42:34 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8.com.redhat.rhevm.vdsm
already removed
Nov 06 10:42:34 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8.org.qemu.guest_agent.0
already removed
Nov 06 10:43:56 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/77766cb4-0625-4432-a16f-def5e702102a.com.redhat.rhevm.vdsm
already removed
Nov 06 10:43:56 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/77766cb4-0625-4432-a16f-def5e702102a.org.qemu.guest_agent.0
already removed
Nov 06 10:52:08 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/3bb83ad5-dc00-445d-8708-2db18c53e0e6.com.redhat.rhevm.vdsm
already removed
Nov 06 10:52:08 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/3bb83ad5-dc00-445d-8708-2db18c53e0e6.org.qemu.guest_agent.0
already removed
virt3:
# systemctl -l status vdsmd
vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)
Active: active (running) since Fri 2015-11-06 10:06:28 EST; 4h 30min ago
Process: 12442 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh
--post-stop (code=exited, status=0/SUCCESS)
Process: 12446 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
--pre-start (code=exited, status=0/SUCCESS)
Main PID: 12561 (vdsm)
CGroup: /system.slice/vdsmd.service
├─ 9066 /usr/libexec/ioprocess --read-pipe-fd 40
--write-pipe-fd 36 --max-threads 10 --max-queued-requests 10
├─ 9074 /usr/libexec/ioprocess --read-pipe-fd 48
--write-pipe-fd 47 --max-threads 10 --max-queued-requests 10
├─ 9085 /usr/libexec/ioprocess --read-pipe-fd 57
--write-pipe-fd 56 --max-threads 10 --max-queued-requests 10
├─12561 /usr/bin/python /usr/share/vdsm/vdsm
├─12726 /usr/libexec/ioprocess --read-pipe-fd 65
--write-pipe-fd 63 --max-threads 10 --max-queued-requests 10
├─12744 /usr/libexec/ioprocess --read-pipe-fd 72
--write-pipe-fd 74 --max-threads 10 --max-queued-requests 10
└─12750 /usr/libexec/ioprocess --read-pipe-fd 83
--write-pipe-fd 82 --max-threads 10 --max-queued-requests 10
Nov 06 10:09:15 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.com.redhat.rhevm.vdsm
already removed
Nov 06 10:09:15 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.org.qemu.guest_agent.0
already removed
Nov 06 10:10:17 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/aa487207-7ff4-465a-9d9b-2a103d50dc77.com.redhat.rhevm.vdsm
already removed
Nov 06 10:10:17 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/aa487207-7ff4-465a-9d9b-2a103d50dc77.org.qemu.guest_agent.0
already removed
Nov 06 10:15:05 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/d66235b4-7c8d-4da2-a572-82ecc9c53a59.com.redhat.rhevm.vdsm
already removed
Nov 06 10:15:05 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/d66235b4-7c8d-4da2-a572-82ecc9c53a59.org.qemu.guest_agent.0
already removed
Nov 06 10:19:13 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/c64d80f9-a3bd-44f3-89c4-f1b9b509cdde.com.redhat.rhevm.vdsm
already removed
Nov 06 10:19:13 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/c64d80f9-a3bd-44f3-89c4-f1b9b509cdde.org.qemu.guest_agent.0
already removed
Nov 06 10:42:56 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.com.redhat.rhevm.vdsm
already removed
Nov 06 10:42:56 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.org.qemu.guest_agent.0
already removed
--- I don't know what those warnings mean...
Jason.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20151106/ab28d4cc/attachment-0001.html>
More information about the Users
mailing list