[ovirt-users] upgrade from 3.5 to 3.6 causing problems with migration

Jason Keltz jas at cse.yorku.ca
Fri Nov 6 14:37:56 EST 2015


On 11/06/2015 02:02 PM, Simone Tiraboschi wrote:
>
>
> On Fri, Nov 6, 2015 at 5:21 PM, Jason Keltz <jas at cse.yorku.ca 
> <mailto:jas at cse.yorku.ca>> wrote:
>
>     Hi.
>
>     Last night, I upgraded my engine from 3.5 to 3.6.  That went
>     flawlessly.
>     Today, I'm trying to upgrade the vdsm on the hosts from 3.5 to 3.6
>     (along with applying other RHEL7.1 updates) However, when I'm
>     trying to put each host into maintenance mode, and migrations
>     start to occur, they all seem to FAIL now!  Even worse, when they
>     fail, it leaves the hosts DOWN!  If there's a failure, I'd expect
>     the host to simply abort the migration....  Any help in debugging
>     this would be VERY much appreciated!
>
>         2015-11-06 10:09:16,065 ERROR
>         [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>         (org.ovirt.thread.pool-8-thread-4) [] Correlation ID:
>         658ba478, Job ID: 524e8c44-04e0-42d3-89f9-9f4e4d397583, Call
>         Stack: null, Custom Event ID: -1, Message: Migration failed 
>         (VM: eportfolio, Source: virt1).
>         2015-11-06 10:10:17,112 ERROR
>         [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>         (org.ovirt.thread.pool-8-thread-22) [2f0dee16] Correlation ID:
>         7da3ac1b, Job ID: 93c0b1f2-4c8e-48cf-9e63-c1ba91be425f, Call
>         Stack: null, Custom Event ID: -1, Message: Migration failed 
>         (VM: ftp1, Source: virt1).
>         2015-11-06 10:15:08,273 ERROR
>         [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>         (org.ovirt.thread.pool-8-thread-45) [] Correlation ID:
>         5394ef76, Job ID: 994065fc-a142-4821-934a-c2297d86ec12, Call
>         Stack: null, Custom Event ID: -1, Message: Migration failed 
>         while Host is in 'preparing for maintenance' state.
>         2015-11-06 10:19:13,712 ERROR
>         [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>         (org.ovirt.thread.pool-8-thread-36) [] Correlation ID:
>         6e422728, Job ID: 994065fc-a142-4821-934a-c2297d86ec12, Call
>         Stack: null, Custom Event ID: -1, Message: Migration failed 
>         while Host is in 'preparing for maintenance' state.
>         2015-11-06 10:42:37,852 ERROR
>         [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>         (org.ovirt.thread.pool-8-thread-12) [] Correlation ID:
>         e7f6300, Job ID: 1ea16622-0fa0-4e92-89e5-9dc235c03ef8, Call
>         Stack: null, Custom Event ID: -1, Message: Migration failed 
>         (VM: ipa, Source: virt1).
>         2015-11-06 10:43:59,732 ERROR
>         [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>         (org.ovirt.thread.pool-8-thread-40) [] Correlation ID:
>         39cfdf9, Job ID: 72be29bc-a02b-4a90-b5ec-8b995c2fa692, Call
>         Stack: null, Custom Event ID: -1, Message: Migration failed 
>         (VM: labtesteval, Source: virt1).
>         2015-11-06 10:52:11,893 ERROR
>         [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>         (org.ovirt.thread.pool-8-thread-23) [] Correlation ID:
>         5c435149, Job ID: 1dcd1e14-baa6-44bc-a853-5d33107b759c, Call
>         Stack: null, Custom Event ID: -1, Message: Migration failed 
>         (VM: www-vhost, Source: virt1).
>
>
>
>     The complete engine log, virt1, virt2, and virt3 vdsm logs are here:
>
>     http://www.eecs.yorku.ca/~jas/ovirt-debug/11062015
>     <http://www.eecs.yorku.ca/%7Ejas/ovirt-debug/11062015>
>
>
> Is vdsmd service still active on that hosts?

Hi Simone..

Yes..

virt1:
sh-4.2# systemctl -l status vdsmd
vdsmd.service - Virtual Desktop Server Manager
    Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)
    Active: active (running) since Thu 2015-11-05 22:47:46 EST; 15h ago
  Main PID: 16520 (vdsm)
    CGroup: /system.slice/vdsmd.service
            ├─16520 /usr/bin/python /usr/share/vdsm/vdsm
            ├─30038 /usr/libexec/ioprocess --read-pipe-fd 67 
--write-pipe-fd 66 --max-threads 10 --max-queued-requests 10
            ├─30055 /usr/libexec/ioprocess --read-pipe-fd 76 
--write-pipe-fd 75 --max-threads 10 --max-queued-requests 10
            └─30062 /usr/libexec/ioprocess --read-pipe-fd 81 
--write-pipe-fd 84 --max-threads 10 --max-queued-requests 10

Nov 06 10:09:15 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.com.redhat.rhevm.vdsm 
already removed
Nov 06 10:09:15 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.org.qemu.guest_agent.0 
already removed
Nov 06 10:10:15 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/aa487207-7ff4-465a-9d9b-2a103d50dc77.com.redhat.rhevm.vdsm 
already removed
Nov 06 10:10:15 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/aa487207-7ff4-465a-9d9b-2a103d50dc77.org.qemu.guest_agent.0 
already removed
Nov 06 10:42:36 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8.com.redhat.rhevm.vdsm 
already removed
Nov 06 10:42:36 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8.org.qemu.guest_agent.0 
already removed
Nov 06 10:43:57 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/77766cb4-0625-4432-a16f-def5e702102a.com.redhat.rhevm.vdsm 
already removed
Nov 06 10:43:57 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/77766cb4-0625-4432-a16f-def5e702102a.org.qemu.guest_agent.0 
already removed
Nov 06 10:52:08 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/3bb83ad5-dc00-445d-8708-2db18c53e0e6.com.redhat.rhevm.vdsm 
already removed
Nov 06 10:52:08 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/3bb83ad5-dc00-445d-8708-2db18c53e0e6.org.qemu.guest_agent.0 
already removed

virt2:

# systemctl -l status vdsmd
vdsmd.service - Virtual Desktop Server Manager
    Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)
    Active: active (running) since Fri 2015-11-06 10:40:36 EST; 3h 55min ago
   Process: 5035 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh 
--pre-start (code=exited, status=0/SUCCESS)
  Main PID: 6427 (vdsm)
    CGroup: /system.slice/vdsmd.service
            ├─ 6334 /usr/libexec/ioprocess --read-pipe-fd 39 
--write-pipe-fd 38 --max-threads 10 --max-queued-requests 10
            ├─ 6342 /usr/libexec/ioprocess --read-pipe-fd 48 
--write-pipe-fd 46 --max-threads 10 --max-queued-requests 10
            ├─ 6354 /usr/libexec/ioprocess --read-pipe-fd 55 
--write-pipe-fd 54 --max-threads 10 --max-queued-requests 10
            ├─ 6427 /usr/bin/python /usr/share/vdsm/vdsm
            ├─15071 /usr/libexec/ioprocess --read-pipe-fd 65 
--write-pipe-fd 63 --max-threads 10 --max-queued-requests 10
            ├─15125 /usr/libexec/ioprocess --read-pipe-fd 73 
--write-pipe-fd 72 --max-threads 10 --max-queued-requests 10
            └─15140 /usr/libexec/ioprocess --read-pipe-fd 83 
--write-pipe-fd 82 --max-threads 10 --max-queued-requests 10

Nov 06 10:40:37 virt2.eecs.yorku.ca python[6427]: DIGEST-MD5 
parse_server_challenge()
Nov 06 10:40:37 virt2.eecs.yorku.ca python[6427]: DIGEST-MD5 ask_user_info()
Nov 06 10:40:37 virt2.eecs.yorku.ca python[6427]: DIGEST-MD5 
make_client_response()
Nov 06 10:40:37 virt2.eecs.yorku.ca python[6427]: DIGEST-MD5 client step 3
Nov 06 10:42:34 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8.com.redhat.rhevm.vdsm 
already removed
Nov 06 10:42:34 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8.org.qemu.guest_agent.0 
already removed
Nov 06 10:43:56 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/77766cb4-0625-4432-a16f-def5e702102a.com.redhat.rhevm.vdsm 
already removed
Nov 06 10:43:56 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/77766cb4-0625-4432-a16f-def5e702102a.org.qemu.guest_agent.0 
already removed
Nov 06 10:52:08 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/3bb83ad5-dc00-445d-8708-2db18c53e0e6.com.redhat.rhevm.vdsm 
already removed
Nov 06 10:52:08 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/3bb83ad5-dc00-445d-8708-2db18c53e0e6.org.qemu.guest_agent.0 
already removed

virt3:


# systemctl -l status vdsmd
vdsmd.service - Virtual Desktop Server Manager
    Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)
    Active: active (running) since Fri 2015-11-06 10:06:28 EST; 4h 30min ago
   Process: 12442 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh 
--post-stop (code=exited, status=0/SUCCESS)
   Process: 12446 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh 
--pre-start (code=exited, status=0/SUCCESS)
  Main PID: 12561 (vdsm)
    CGroup: /system.slice/vdsmd.service
            ├─ 9066 /usr/libexec/ioprocess --read-pipe-fd 40 
--write-pipe-fd 36 --max-threads 10 --max-queued-requests 10
            ├─ 9074 /usr/libexec/ioprocess --read-pipe-fd 48 
--write-pipe-fd 47 --max-threads 10 --max-queued-requests 10
            ├─ 9085 /usr/libexec/ioprocess --read-pipe-fd 57 
--write-pipe-fd 56 --max-threads 10 --max-queued-requests 10
            ├─12561 /usr/bin/python /usr/share/vdsm/vdsm
            ├─12726 /usr/libexec/ioprocess --read-pipe-fd 65 
--write-pipe-fd 63 --max-threads 10 --max-queued-requests 10
            ├─12744 /usr/libexec/ioprocess --read-pipe-fd 72 
--write-pipe-fd 74 --max-threads 10 --max-queued-requests 10
            └─12750 /usr/libexec/ioprocess --read-pipe-fd 83 
--write-pipe-fd 82 --max-threads 10 --max-queued-requests 10

Nov 06 10:09:15 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.com.redhat.rhevm.vdsm 
already removed
Nov 06 10:09:15 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.org.qemu.guest_agent.0 
already removed
Nov 06 10:10:17 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/aa487207-7ff4-465a-9d9b-2a103d50dc77.com.redhat.rhevm.vdsm 
already removed
Nov 06 10:10:17 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/aa487207-7ff4-465a-9d9b-2a103d50dc77.org.qemu.guest_agent.0 
already removed
Nov 06 10:15:05 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/d66235b4-7c8d-4da2-a572-82ecc9c53a59.com.redhat.rhevm.vdsm 
already removed
Nov 06 10:15:05 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/d66235b4-7c8d-4da2-a572-82ecc9c53a59.org.qemu.guest_agent.0 
already removed
Nov 06 10:19:13 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/c64d80f9-a3bd-44f3-89c4-f1b9b509cdde.com.redhat.rhevm.vdsm 
already removed
Nov 06 10:19:13 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/c64d80f9-a3bd-44f3-89c4-f1b9b509cdde.org.qemu.guest_agent.0 
already removed
Nov 06 10:42:56 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.com.redhat.rhevm.vdsm 
already removed
Nov 06 10:42:56 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File: 
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.org.qemu.guest_agent.0 
already removed

--- I don't know what those warnings mean...

Jason.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20151106/ab28d4cc/attachment-0001.html>


More information about the Users mailing list