Re: [Users] Problem with live migration.

Tuesday, 17 April 2012

This is a multi-part message in MIME format.
--------------000509020206090803080704
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

On 04/17/2012 03:45 PM, Martin Hovmöller wrote:
...

 On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim <iheim(a)redhat.com 
 <mailto:iheim@redhat.com>> wrote:

     On 04/17/2012 12:47 PM, Martin Hovmöller wrote:

         I can't live migrate the vm's in my cluster:

         012-04-17 04:06:39,728 INFO
          [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-48)
         Running command: MigrateVmCommand internal: false. Entities
         affected :
          ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM
         2012-04-17 04:06:39,743 INFO
          [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
         (pool-5-thread-48)
         START, MigrateVDSCommand(vdsId =
         f33ffc14-87ba-11e1-b610-e3aeca1e8008,
         vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,
         dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,
         dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>
         <http://10.23.30.110:54321>;,

         migrationMethod=ONLINE), log id: 7c7201be
         2012-04-17 04:06:39,748 INFO
          [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
         (pool-5-thread-48) VdsBroker::migrate::Entered
         (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
         srcHost=10.23.30.130,
         dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>
         <http://10.23.30.110:54321>;,  method=online

         2012-04-17 04:06:39,749 INFO
          [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
         (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId =
         f33ffc14-87ba-11e1-b610-e3aeca1e8008,
         vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,
         dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,
         dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>
         <http://10.23.30.110:54321>;,

         migrationMethod=ONLINE), log id: 3c40c2de
         2012-04-17 04:06:39,834 INFO
          [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
         (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id:
         3c40c2de
         2012-04-17 04:06:39,840 INFO
          [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
         (pool-5-thread-48)
         FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be
         2012-04-17 04:06:42,166 INFO
          [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
         (QuartzScheduler_Worker-18) vds::refreshVmList vm id
         ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds
         rhevh1.<domain>
         ignoring it in the refresh till migration is done
         [...]
         2012-04-17 04:07:33,852 INFO
          [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
         (QuartzScheduler_Worker-34) vds::refreshVmList vm id
         ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on vds
         rhevh1.<domain> ignoring it in the refresh till migration is done
         2012-04-17 04:07:37,407 ERROR
         [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
         (QuartzScheduler_Worker-71) Rerun vm
         ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds
         rhevh2.<domain>
         2012-04-17 04:07:37,414 INFO
          [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
         (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId =
         f33ffc14-87ba-11e1-b610-e3aeca1e8008,
         vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id: 8dfce1b
         2012-04-17 04:07:37,496 ERROR
         [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
         (pool-5-thread-47) Failed in MigrateStatusVDS method
         2012-04-17 04:07:37,497 ERROR
         [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
         (pool-5-thread-47) Error code createErr and error message
         VDSGenericException: VDSErrorException: Failed to
         MigrateStatusVDS,
         error = Error creating the requested virtual machine
         2012-04-17 04:07:37,497 INFO
          [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
         (pool-5-thread-47) Command
         org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand
         return
         value
          Class Name:
         org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc
         mStatus                       Class Name:
         org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
         mCode                         9
         mMessage                      Error creating the requested
         virtual machine

         2012-04-17 04:07:37,497 INFO
          [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
         (pool-5-thread-47) Vds: rhevh2.<domain>
         2012-04-17 04:07:37,497 ERROR
         [org.ovirt.engine.core.vdsbroker.VDSCommandBase]
         (pool-5-thread-47)
         Command MigrateStatusVDS execution failed. Exception:
         VDSErrorException:
         VDSGenericException: VDSErrorException: Failed to
         MigrateStatusVDS,
         error = Error creating the requested virtual machine
         2012-04-17 04:07:37,497 INFO
          [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
         (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id:
         8dfce1b
         2012-04-17 04:07:37,528 WARN
          [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47)
         CanDoAction of action MigrateVm failed.
         Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM

         There is no problem regarding storage or anything like that.
         If I shut
         down the vm I can start it on the other host without any problems
         whatsoever.
         Do I need to do something to make live migration work? Trying
         to dig in
         the logs on the hypervisors, but there's so much stuff being
         logged there...

         _______________________________________________
         Users mailing list
         Users(a)ovirt.org <mailto:Users@ovirt.org>
         http://lists.ovirt.org/mailman/listinfo/users

     vdsm log from the host that failed to create the VM?

 The vdsm log is frighteningly long, so I put it in pastebin: 
 http://pastebin.com/BEeqFCCb
 I don't really see anything worthwhile myself, but on the other hand 
 I'm not sure what I'm looking for, so it's mostly a massive wall of 
 text to me, but it should be somewhere in that time frame. Here is the failure,
"code 9" is not too informative, can you attach the 
libvirtd.log file too?

Thread-49287::DEBUG::2012-04-17 
14:25:39,834::clientIF::81::vds::(wrapper) return migrationCreate with 
{'status': {'message': 'Error creating the requested virtual
machine', 
'code': 9}}
...

 _______________________________________________
 Users mailing list
 Users(a)ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users 

-- 

Thanks,

Rami Vaknin, QE @ Red Hat, TLV, IL.

--------------000509020206090803080704
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    On 04/17/2012 03:45 PM, Martin Hovm&ouml;ller wrote:
    <blockquote
cite="mid:CAGKvcKXMhdcCdui97Y6MPCk8F2912ji8rfVEFhRKasrESycjAQ@mail.gmail.com"
      type="cite"><br>
      <br>
      <div class="gmail_quote">On Tue, Apr 17, 2012 at 2:11 PM, Itamar
        Heim <span dir="ltr">&lt;<a
moz-do-not-send="true"

href="mailto:iheim@redhat.com">iheim@redhat.com</a>&gt;</span>
        wrote:<br>
        <blockquote class="gmail_quote" style="margin:0 0 0
          .8ex;border-left:1px #ccc solid;padding-left:1ex">
          <div class="im">On 04/17/2012 12:47 PM, Martin
Hovm&ouml;ller
            wrote:<br>
          </div>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div class="im">
              I can't live migrate the vm's in my cluster:<br>
              <br>
              012-04-17 04:06:39,728 INFO<br>
              &nbsp;[org.ovirt.engine.core.bll.MigrateVmCommand]
              (pool-5-thread-48)<br>
              Running command: MigrateVmCommand internal: false.
              Entities affected :<br>
              &nbsp;ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM<br>
              2012-04-17 04:06:39,743 INFO<br>
              &nbsp;[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
              (pool-5-thread-48)<br>
              START, MigrateVDSCommand(vdsId =
f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br>
              vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
              srcHost=10.23.30.130,<br>
              dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br>
            </div>
            dstHost=<a moz-do-not-send="true"
              href="http://10.23.30.110:54321"
target="_blank">10.23.30.110:54321</a>
            &lt;<a moz-do-not-send="true"
              href="http://10.23.30.110:54321"
target="_blank">http://10.23.30.110:54321</a>&gt;,
            <div class="im"><br>
              migrationMethod=ONLINE), log id: 7c7201be<br>
              2012-04-17 04:06:39,748 INFO<br>

&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br>
              (pool-5-thread-48) VdsBroker::migrate::Entered<br>
              (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
              srcHost=10.23.30.130,<br>
            </div>
            dstHost=<a moz-do-not-send="true"
              href="http://10.23.30.110:54321"
target="_blank">10.23.30.110:54321</a>
            &lt;<a moz-do-not-send="true"
              href="http://10.23.30.110:54321"
target="_blank">http://10.23.30.110:54321</a>&gt;,
            &nbsp;method=online
            <div class="im"><br>
              2012-04-17 04:06:39,749 INFO<br>

&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br>
              (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId =<br>
              f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br>
              vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
              srcHost=10.23.30.130,<br>
              dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br>
            </div>
            dstHost=<a moz-do-not-send="true"
              href="http://10.23.30.110:54321"
target="_blank">10.23.30.110:54321</a>
            &lt;<a moz-do-not-send="true"
              href="http://10.23.30.110:54321"
target="_blank">http://10.23.30.110:54321</a>&gt;,
            <div>
              <div class="h5"><br>
                migrationMethod=ONLINE), log id: 3c40c2de<br>
                2012-04-17 04:06:39,834 INFO<br>

&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br>
                (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log
                id: 3c40c2de<br>
                2012-04-17 04:06:39,840 INFO<br>
                &nbsp;[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
                (pool-5-thread-48)<br>
                FINISH, MigrateVDSCommand, return: MigratingFrom, log
                id: 7c7201be<br>
                2012-04-17 04:06:42,166 INFO<br>

&nbsp;[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br>
                (QuartzScheduler_Worker-18) vds::refreshVmList vm id<br>
                ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds
                rhevh1.&lt;domain&gt;<br>
                ignoring it in the refresh till migration is done<br>
                [...]<br>
                2012-04-17 04:07:33,852 INFO<br>

&nbsp;[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br>
                (QuartzScheduler_Worker-34) vds::refreshVmList vm id<br>
                ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on
                vds<br>
                rhevh1.&lt;domain&gt; ignoring it in the refresh till
                migration is done<br>
                2012-04-17 04:07:37,407 ERROR<br>
                [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br>
                (QuartzScheduler_Worker-71) Rerun vm<br>
                ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds
                rhevh2.&lt;domain&gt;<br>
                2012-04-17 04:07:37,414 INFO<br>

&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br>
                (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId
                =<br>
                f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br>
                vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id:
                8dfce1b<br>
                2012-04-17 04:07:37,496 ERROR<br>
                [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
                (pool-5-thread-47) Failed in MigrateStatusVDS method<br>
                2012-04-17 04:07:37,497 ERROR<br>
                [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
                (pool-5-thread-47) Error code createErr and error
                message<br>
                VDSGenericException: VDSErrorException: Failed to
                MigrateStatusVDS,<br>
                error = Error creating the requested virtual machine<br>
                2012-04-17 04:07:37,497 INFO<br>

&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
                (pool-5-thread-47) Command<br>
                org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand
                return<br>
                value<br>
                &nbsp;Class Name:<br>

org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc<br>
                mStatus &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Class Name:<br>
                org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc<br>
                mCode &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 9<br>
                mMessage &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Error creating the
                requested virtual machine<br>
                <br>
                <br>
                2012-04-17 04:07:37,497 INFO<br>

&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
                (pool-5-thread-47) Vds: rhevh2.&lt;domain&gt;<br>
                2012-04-17 04:07:37,497 ERROR<br>
                [org.ovirt.engine.core.vdsbroker.VDSCommandBase]
                (pool-5-thread-47)<br>
                Command MigrateStatusVDS execution failed. Exception:
                VDSErrorException:<br>
                VDSGenericException: VDSErrorException: Failed to
                MigrateStatusVDS,<br>
                error = Error creating the requested virtual machine<br>
                2012-04-17 04:07:37,497 INFO<br>

&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br>
                (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log
                id: 8dfce1b<br>
                2012-04-17 04:07:37,528 WARN<br>
                &nbsp;[org.ovirt.engine.core.bll.MigrateVmCommand]
                (pool-5-thread-47)<br>
                CanDoAction of action MigrateVm failed.<br>

Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM<br>
                <br>
                <br>
                There is no problem regarding storage or anything like
                that. If I shut<br>
                down the vm I can start it on the other host without any
                problems<br>
                whatsoever.<br>
                Do I need to do something to make live migration work?
                Trying to dig in<br>
                the logs on the hypervisors, but there's so much stuff
                being logged there...<br>
                <br>
                <br>
              </div>
            </div>
            _______________________________________________<br>
            Users mailing list<br>
            <a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
              target=&quot;_blank&quot;&gt;Users(a)ovirt.org&lt;/a&gt;&lt;br&gt;
            <a moz-do-not-send="true"
              href="http://lists.ovirt.org/mailman/listinfo/users"

target="_blank">http://lists.ovirt.org/mailman/listinfo/user...
          </blockquote>
          <br>
          vdsm log from the host that failed to create the VM?<br>
        </blockquote>
      </div>
      <br>
      <div>The vdsm log is&nbsp;frighteningly long, so I put it in
pastebin:&nbsp;<a
          moz-do-not-send="true"
href="http://pastebin.com/BEeqFCCb">http://pastebin.com/BEeq...
      <div>I don't really see anything worthwhile myself, but on the
        other hand I'm not sure what I'm looking for, so it's mostly a
        massive wall of text to me, but it should be somewhere in that
        time frame.</div>
    </blockquote>
    Here is the failure, "code 9" is not too informative, can you attach
    the libvirtd.log file too?<br>
    <br>
    Thread-49287::DEBUG::2012-04-17
    14:25:39,834::clientIF::81::vds::(wrapper) return migrationCreate
    with {'status': {'message': 'Error creating the requested virtual
    machine', 'code': 9}}<br>
    <blockquote
cite="mid:CAGKvcKXMhdcCdui97Y6MPCk8F2912ji8rfVEFhRKasrESycjAQ@mail.gmail.com"
      type="cite">
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
    </blockquote>
    <br>
    <br>
    <pre class="moz-signature" cols="72">-- 

Thanks,

Rami Vaknin, QE @ Red Hat, TLV, IL.
</pre>
  </body>
</html>

--------------000509020206090803080704--

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [Users] Problem with live migration.