<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    On 04/18/2012 11:30 AM, Martin Hovm&ouml;ller wrote:
    <blockquote
cite="mid:CAGKvcKXco_=isKREAGyq+CTV_-=xVUruBHfBHDmkREJLc-W+7w@mail.gmail.com"
      type="cite"><br>
      <br>
      <div class="gmail_quote">On Tue, Apr 17, 2012 at 4:38 PM, Rami
        Vaknin <span dir="ltr">&lt;<a moz-do-not-send="true"
            href="mailto:rvaknin@redhat.com">rvaknin@redhat.com</a>&gt;</span>
        wrote:<br>
        <blockquote class="gmail_quote" style="margin:0 0 0
          .8ex;border-left:1px #ccc solid;padding-left:1ex">
          <div bgcolor="#FFFFFF" text="#000000">
            <div>
              <div class="h5"> On 04/17/2012 05:36 PM, Martin Hovm&ouml;ller
                wrote:
                <blockquote type="cite"><br>
                  <br>
                  <div class="gmail_quote">On Tue, Apr 17, 2012 at 2:55
                    PM, Rami Vaknin <span dir="ltr">&lt;<a
                        moz-do-not-send="true"
                        href="mailto:rvaknin@redhat.com" target="_blank">rvaknin@redhat.com</a>&gt;</span>
                    wrote:<br>
                    <blockquote class="gmail_quote" style="margin:0 0 0
                      .8ex;border-left:1px #ccc solid;padding-left:1ex">
                      <div bgcolor="#FFFFFF" text="#000000">
                        <div>
                          <div> On 04/17/2012 03:45 PM, Martin Hovm&ouml;ller
                            wrote:
                            <blockquote type="cite"><br>
                              <br>
                              <div class="gmail_quote">On Tue, Apr 17,
                                2012 at 2:11 PM, Itamar Heim <span
                                  dir="ltr">&lt;<a
                                    moz-do-not-send="true"
                                    href="mailto:iheim@redhat.com"
                                    target="_blank">iheim@redhat.com</a>&gt;</span>
                                wrote:<br>
                                <blockquote class="gmail_quote"
                                  style="margin:0 0 0
                                  .8ex;border-left:1px #ccc
                                  solid;padding-left:1ex">
                                  <div>On 04/17/2012 12:47 PM, Martin
                                    Hovm&ouml;ller wrote:<br>
                                  </div>
                                  <blockquote class="gmail_quote"
                                    style="margin:0 0 0
                                    .8ex;border-left:1px #ccc
                                    solid;padding-left:1ex">
                                    <div> I can't live migrate the vm's
                                      in my cluster:<br>
                                      <br>
                                      012-04-17 04:06:39,728 INFO<br>
                                      &nbsp;[org.ovirt.engine.core.bll.MigrateVmCommand]

                                      (pool-5-thread-48)<br>
                                      Running command: MigrateVmCommand
                                      internal: false. Entities affected
                                      :<br>
                                      &nbsp;ID:
                                      ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
                                      Type: VM<br>
                                      2012-04-17 04:06:39,743 INFO<br>
                                      &nbsp;[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]


                                      (pool-5-thread-48)<br>
                                      START, MigrateVDSCommand(vdsId =
                                      f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br>
                                      vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,

                                      srcHost=10.23.30.130,<br>
dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br>
                                    </div>
                                    dstHost=<a moz-do-not-send="true"
                                      href="http://10.23.30.110:54321"
                                      target="_blank">10.23.30.110:54321</a>
                                    &lt;<a moz-do-not-send="true"
                                      href="http://10.23.30.110:54321"
                                      target="_blank">http://10.23.30.110:54321</a>&gt;,


                                    <div><br>
                                      migrationMethod=ONLINE), log id:
                                      7c7201be<br>
                                      2012-04-17 04:06:39,748 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br>
                                      (pool-5-thread-48)
                                      VdsBroker::migrate::Entered<br>
                                      (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,

                                      srcHost=10.23.30.130,<br>
                                    </div>
                                    dstHost=<a moz-do-not-send="true"
                                      href="http://10.23.30.110:54321"
                                      target="_blank">10.23.30.110:54321</a>
                                    &lt;<a moz-do-not-send="true"
                                      href="http://10.23.30.110:54321"
                                      target="_blank">http://10.23.30.110:54321</a>&gt;,


                                    &nbsp;method=online
                                    <div><br>
                                      2012-04-17 04:06:39,749 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br>
                                      (pool-5-thread-48) START,
                                      MigrateBrokerVDSCommand(vdsId =<br>
f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br>
                                      vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,

                                      srcHost=10.23.30.130,<br>
dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br>
                                    </div>
                                    dstHost=<a moz-do-not-send="true"
                                      href="http://10.23.30.110:54321"
                                      target="_blank">10.23.30.110:54321</a>
                                    &lt;<a moz-do-not-send="true"
                                      href="http://10.23.30.110:54321"
                                      target="_blank">http://10.23.30.110:54321</a>&gt;,


                                    <div>
                                      <div><br>
                                        migrationMethod=ONLINE), log id:
                                        3c40c2de<br>
                                        2012-04-17 04:06:39,834 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br>
                                        (pool-5-thread-48) FINISH,
                                        MigrateBrokerVDSCommand, log id:
                                        3c40c2de<br>
                                        2012-04-17 04:06:39,840 INFO<br>
                                        &nbsp;[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]


                                        (pool-5-thread-48)<br>
                                        FINISH, MigrateVDSCommand,
                                        return: MigratingFrom, log id:
                                        7c7201be<br>
                                        2012-04-17 04:06:42,166 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br>
                                        (QuartzScheduler_Worker-18)
                                        vds::refreshVmList vm id<br>
                                        ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
                                        is migrating to vds
                                        rhevh1.&lt;domain&gt;<br>
                                        ignoring it in the refresh till
                                        migration is done<br>
                                        [...]<br>
                                        2012-04-17 04:07:33,852 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br>
                                        (QuartzScheduler_Worker-34)
                                        vds::refreshVmList vm id<br>
                                        ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
                                        status = Paused on vds<br>
                                        rhevh1.&lt;domain&gt; ignoring
                                        it in the refresh till migration
                                        is done<br>
                                        2012-04-17 04:07:37,407 ERROR<br>
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br>
                                        (QuartzScheduler_Worker-71)
                                        Rerun vm<br>
                                        ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a.
                                        Called from vds
                                        rhevh2.&lt;domain&gt;<br>
                                        2012-04-17 04:07:37,414 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br>
                                        (pool-5-thread-47) START,
                                        MigrateStatusVDSCommand(vdsId =<br>
f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br>
                                        vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a),

                                        log id: 8dfce1b<br>
                                        2012-04-17 04:07:37,496 ERROR<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
                                        (pool-5-thread-47) Failed in
                                        MigrateStatusVDS method<br>
                                        2012-04-17 04:07:37,497 ERROR<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
                                        (pool-5-thread-47) Error code
                                        createErr and error message<br>
                                        VDSGenericException:
                                        VDSErrorException: Failed to
                                        MigrateStatusVDS,<br>
                                        error = Error creating the
                                        requested virtual machine<br>
                                        2012-04-17 04:07:37,497 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
                                        (pool-5-thread-47) Command<br>
                                        org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand


                                        return<br>
                                        value<br>
                                        &nbsp;Class Name:<br>
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc<br>
                                        mStatus &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
                                        Class Name:<br>
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc<br>
                                        mCode &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 9<br>
                                        mMessage &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
                                        &nbsp;Error creating the requested
                                        virtual machine<br>
                                        <br>
                                        <br>
                                        2012-04-17 04:07:37,497 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
                                        (pool-5-thread-47) Vds:
                                        rhevh2.&lt;domain&gt;<br>
                                        2012-04-17 04:07:37,497 ERROR<br>
                                        [org.ovirt.engine.core.vdsbroker.VDSCommandBase]


                                        (pool-5-thread-47)<br>
                                        Command MigrateStatusVDS
                                        execution failed. Exception:
                                        VDSErrorException:<br>
                                        VDSGenericException:
                                        VDSErrorException: Failed to
                                        MigrateStatusVDS,<br>
                                        error = Error creating the
                                        requested virtual machine<br>
                                        2012-04-17 04:07:37,497 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br>
                                        (pool-5-thread-47) FINISH,
                                        MigrateStatusVDSCommand, log id:
                                        8dfce1b<br>
                                        2012-04-17 04:07:37,528 WARN<br>
                                        &nbsp;[org.ovirt.engine.core.bll.MigrateVmCommand]


                                        (pool-5-thread-47)<br>
                                        CanDoAction of action MigrateVm
                                        failed.<br>
Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM<br>
                                        <br>
                                        <br>
                                        There is no problem regarding
                                        storage or anything like that.
                                        If I shut<br>
                                        down the vm I can start it on
                                        the other host without any
                                        problems<br>
                                        whatsoever.<br>
                                        Do I need to do something to
                                        make live migration work? Trying
                                        to dig in<br>
                                        the logs on the hypervisors, but
                                        there's so much stuff being
                                        logged there...<br>
                                        <br>
                                        <br>
                                      </div>
                                    </div>
_______________________________________________<br>
                                    Users mailing list<br>
                                    <a moz-do-not-send="true"
                                      href="mailto:Users@ovirt.org"
                                      target="_blank">Users@ovirt.org</a><br>
                                    <a moz-do-not-send="true"
                                      href="http://lists.ovirt.org/mailman/listinfo/users"
                                      target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
                                  </blockquote>
                                  <br>
                                  vdsm log from the host that failed to
                                  create the VM?<br>
                                </blockquote>
                              </div>
                              <br>
                              <div>The vdsm log is&nbsp;frighteningly long,
                                so I put it in pastebin:&nbsp;<a
                                  moz-do-not-send="true"
                                  href="http://pastebin.com/BEeqFCCb"
                                  target="_blank">http://pastebin.com/BEeqFCCb</a></div>
                              <div>I don't really see anything
                                worthwhile myself, but on the other hand
                                I'm not sure what I'm looking for, so
                                it's mostly a massive wall of text to
                                me, but it should be somewhere in that
                                time frame.</div>
                            </blockquote>
                          </div>
                        </div>
                        Here is the failure, "code 9" is not too
                        informative, can you attach the libvirtd.log
                        file too?<br>
                        <br>
                        Thread-49287::DEBUG::2012-04-17
                        14:25:39,834::clientIF::81::vds::(wrapper)
                        return migrationCreate with {'status':
                        {'message': 'Error creating the requested
                        virtual machine', 'code': 9}}
                        <div><br>
                          <blockquote type="cite"> <br>
                            <fieldset></fieldset>
                            <br>
                            <pre>_______________________________________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>
<a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
                          </blockquote>
                          <br>
                          <br>
                        </div>
                        <span><font color="#888888">
                            <pre cols="72">-- 

Thanks,

Rami Vaknin, QE @ Red Hat, TLV, IL.
</pre>
                          </font></span></div>
                    </blockquote>
                  </div>
                  <div><br>
                  </div>
                  <div>I think this is the relevant info from the
                    libvirt.log:</div>
                  <div>
                    <div>2012-04-17 14:30:53.768+0000: 8506: debug :
                      virDomainLookupByUUIDString:1984 :
                      conn=0x7fa9a00bcde0,
                      uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a</div>
                    <div>2012-04-17 14:30:53.768+0000: 8506: debug :
                      virDomainLookupByUUID:1941 : conn=0x7fa9a00bcde0,
                      uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a</div>
                    <div>2012-04-17 14:30:53.771+0000: 8506: error :
                      virNetClientProgramDispatchError:170 : Domain not
                      found: no domain with matching uuid
                      'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a'</div>
                  </div>
                  <div><br>
                  </div>
                  <div>What domain is this, is it the storage domain?</div>
                </blockquote>
                <br>
              </div>
            </div>
            Domain in libvirt is simply a vm.
            <div class="im"><br>
              <blockquote type="cite"> <br>
                <fieldset></fieldset>
                <br>
                <pre>_______________________________________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>
<a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
              </blockquote>
              <br>
              <br>
              <pre cols="72">-- 

Thanks,

Rami Vaknin, QE @ Red Hat, TLV, IL.
</pre>
            </div>
          </div>
        </blockquote>
      </div>
      <br>
      <div>Alright, so what does this mean then? The machine I'm
        migrating to can't see the vm?<br>
        That doesn't make sense since I can manually start it on that
        host..</div>
    </blockquote>
    <br>
    On second look at the vdsm.log I see that disk preparation in the
    destination host failed (timed out), try to take a look at vdsm.log
    at the destination.<br>
    In addition, for some reason getVmStats returns "Virtual machine
    does not exist" right before the migration.<br>
    <br>
    <br>
    Thread-49286::DEBUG::2012-04-17
    14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call
    getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}<br>
    Thread-49286::DEBUG::2012-04-17
    14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with
    {'status': {'message': 'Virtual machine does not exist', 'code': 1}}<br>
    Thread-49287::DEBUG::2012-04-17
    14:25:07,381::clientIF::76::vds::(wrapper) [10.23.30.130]::call
    migrationCreate with ({'bridge': 'ovirtmgmt', 'acpiEnable':
    'true',.....<br>
    Thread-49287::DEBUG::2012-04-17
    14:25:08,156::libvirtvm::1504::vm.Vm::(waitForMigrationDestinationPrepare)
    vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::migration destination:
    waiting 31s for path preparation<br>
    <br>
    Thread-49287::DEBUG::2012-04-17
    14:25:39,458::libvirtvm::1507::vm.Vm::(waitForMigrationDestinationPrepare)
    vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::Timeout while waiting
    for path preparation<br>
    <br>
    14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call
    getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}<br>
    Thread-49286::DEBUG::2012-04-17
    14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with
    {'status': {'message': 'Virtual machine does not exist', 'code': 1}}<br>
    <blockquote
cite="mid:CAGKvcKXco_=isKREAGyq+CTV_-=xVUruBHfBHDmkREJLc-W+7w@mail.gmail.com"
      type="cite">
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
    </blockquote>
    <br>
    <br>
    <pre class="moz-signature" cols="72">-- 

Thanks,

Rami Vaknin, QE @ Red Hat, TLV, IL.
</pre>
  </body>
</html>