<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    On 04/19/2012 05:17 PM, Martin Hovm&ouml;ller wrote:
    <blockquote
cite="mid:CAGKvcKUOXkNPKM1qGe2seVKWp2Z98JO5ehoDYs8vcgLF5p16Ww@mail.gmail.com"
      type="cite"><br>
      <br>
      <div class="gmail_quote">On Wed, Apr 18, 2012 at 11:06 AM, Rami
        Vaknin <span dir="ltr">&lt;<a moz-do-not-send="true"
            href="mailto:rvaknin@redhat.com">rvaknin@redhat.com</a>&gt;</span>
        wrote:<br>
        <blockquote class="gmail_quote" style="margin:0 0 0
          .8ex;border-left:1px #ccc solid;padding-left:1ex">
          <div bgcolor="#FFFFFF" text="#000000">
            <div>
              <div class="h5"> On 04/18/2012 11:30 AM, Martin Hovm&ouml;ller
                wrote:
                <blockquote type="cite"><br>
                  <br>
                  <div class="gmail_quote">On Tue, Apr 17, 2012 at 4:38
                    PM, Rami Vaknin <span dir="ltr">&lt;<a
                        moz-do-not-send="true"
                        href="mailto:rvaknin@redhat.com" target="_blank">rvaknin@redhat.com</a>&gt;</span>
                    wrote:<br>
                    <blockquote class="gmail_quote" style="margin:0 0 0
                      .8ex;border-left:1px #ccc solid;padding-left:1ex">
                      <div bgcolor="#FFFFFF" text="#000000">
                        <div>
                          <div> On 04/17/2012 05:36 PM, Martin Hovm&ouml;ller
                            wrote:
                            <blockquote type="cite"><br>
                              <br>
                              <div class="gmail_quote">On Tue, Apr 17,
                                2012 at 2:55 PM, Rami Vaknin <span
                                  dir="ltr">&lt;<a
                                    moz-do-not-send="true"
                                    href="mailto:rvaknin@redhat.com"
                                    target="_blank">rvaknin@redhat.com</a>&gt;</span>
                                wrote:<br>
                                <blockquote class="gmail_quote"
                                  style="margin:0 0 0
                                  .8ex;border-left:1px #ccc
                                  solid;padding-left:1ex">
                                  <div bgcolor="#FFFFFF" text="#000000">
                                    <div>
                                      <div> On 04/17/2012 03:45 PM,
                                        Martin Hovm&ouml;ller wrote:
                                        <blockquote type="cite"><br>
                                          <br>
                                          <div class="gmail_quote">On
                                            Tue, Apr 17, 2012 at 2:11
                                            PM, Itamar Heim <span
                                              dir="ltr">&lt;<a
                                                moz-do-not-send="true"
                                                href="mailto:iheim@redhat.com"
                                                target="_blank">iheim@redhat.com</a>&gt;</span>
                                            wrote:<br>
                                            <blockquote
                                              class="gmail_quote"
                                              style="margin:0 0 0
                                              .8ex;border-left:1px #ccc
                                              solid;padding-left:1ex">
                                              <div>On 04/17/2012 12:47
                                                PM, Martin Hovm&ouml;ller
                                                wrote:<br>
                                              </div>
                                              <blockquote
                                                class="gmail_quote"
                                                style="margin:0 0 0
                                                .8ex;border-left:1px
                                                #ccc
                                                solid;padding-left:1ex">
                                                <div> I can't live
                                                  migrate the vm's in my
                                                  cluster:<br>
                                                  <br>
                                                  012-04-17 04:06:39,728
                                                  INFO<br>
                                                  &nbsp;[org.ovirt.engine.core.bll.MigrateVmCommand]


                                                  (pool-5-thread-48)<br>
                                                  Running command:
                                                  MigrateVmCommand
                                                  internal: false.
                                                  Entities affected :<br>
                                                  &nbsp;ID:
                                                  ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
                                                  Type: VM<br>
                                                  2012-04-17
                                                  04:06:39,743 INFO<br>
                                                  &nbsp;[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]



                                                  (pool-5-thread-48)<br>
                                                  START,
                                                  MigrateVDSCommand(vdsId
                                                  =
                                                  f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br>
                                                  vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,


                                                  srcHost=10.23.30.130,<br>
dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br>
                                                </div>
                                                dstHost=<a
                                                  moz-do-not-send="true"
href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a>
                                                &lt;<a
                                                  moz-do-not-send="true"
href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>&gt;,



                                                <div><br>
                                                  migrationMethod=ONLINE),
                                                  log id: 7c7201be<br>
                                                  2012-04-17
                                                  04:06:39,748 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br>
                                                  (pool-5-thread-48)
                                                  VdsBroker::migrate::Entered<br>
                                                  (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,


                                                  srcHost=10.23.30.130,<br>
                                                </div>
                                                dstHost=<a
                                                  moz-do-not-send="true"
href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a>
                                                &lt;<a
                                                  moz-do-not-send="true"
href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>&gt;,



                                                &nbsp;method=online
                                                <div><br>
                                                  2012-04-17
                                                  04:06:39,749 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br>
                                                  (pool-5-thread-48)
                                                  START,
                                                  MigrateBrokerVDSCommand(vdsId
                                                  =<br>
f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br>
                                                  vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,


                                                  srcHost=10.23.30.130,<br>
dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br>
                                                </div>
                                                dstHost=<a
                                                  moz-do-not-send="true"
href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a>
                                                &lt;<a
                                                  moz-do-not-send="true"
href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>&gt;,



                                                <div>
                                                  <div><br>
                                                    migrationMethod=ONLINE),
                                                    log id: 3c40c2de<br>
                                                    2012-04-17
                                                    04:06:39,834 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br>
                                                    (pool-5-thread-48)
                                                    FINISH,
                                                    MigrateBrokerVDSCommand,
                                                    log id: 3c40c2de<br>
                                                    2012-04-17
                                                    04:06:39,840 INFO<br>
                                                    &nbsp;[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]



                                                    (pool-5-thread-48)<br>
                                                    FINISH,
                                                    MigrateVDSCommand,
                                                    return:
                                                    MigratingFrom, log
                                                    id: 7c7201be<br>
                                                    2012-04-17
                                                    04:06:42,166 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br>
                                                    (QuartzScheduler_Worker-18)

                                                    vds::refreshVmList
                                                    vm id<br>
                                                    ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a

                                                    is migrating to vds
rhevh1.&lt;domain&gt;<br>
                                                    ignoring it in the
                                                    refresh till
                                                    migration is done<br>
                                                    [...]<br>
                                                    2012-04-17
                                                    04:07:33,852 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br>
                                                    (QuartzScheduler_Worker-34)

                                                    vds::refreshVmList
                                                    vm id<br>
                                                    ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a

                                                    status = Paused on
                                                    vds<br>
                                                    rhevh1.&lt;domain&gt;
                                                    ignoring it in the
                                                    refresh till
                                                    migration is done<br>
                                                    2012-04-17
                                                    04:07:37,407 ERROR<br>
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br>
                                                    (QuartzScheduler_Worker-71)

                                                    Rerun vm<br>
                                                    ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a.

                                                    Called from vds
                                                    rhevh2.&lt;domain&gt;<br>
                                                    2012-04-17
                                                    04:07:37,414 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br>
                                                    (pool-5-thread-47)
                                                    START,
                                                    MigrateStatusVDSCommand(vdsId
                                                    =<br>
f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br>
                                                    vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a),


                                                    log id: 8dfce1b<br>
                                                    2012-04-17
                                                    04:07:37,496 ERROR<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
                                                    (pool-5-thread-47)
                                                    Failed in
                                                    MigrateStatusVDS
                                                    method<br>
                                                    2012-04-17
                                                    04:07:37,497 ERROR<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
                                                    (pool-5-thread-47)
                                                    Error code createErr
                                                    and error message<br>
                                                    VDSGenericException:
                                                    VDSErrorException:
                                                    Failed to
                                                    MigrateStatusVDS,<br>
                                                    error = Error
                                                    creating the
                                                    requested virtual
                                                    machine<br>
                                                    2012-04-17
                                                    04:07:37,497 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
                                                    (pool-5-thread-47)
                                                    Command<br>
                                                    org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand



                                                    return<br>
                                                    value<br>
                                                    &nbsp;Class Name:<br>
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc<br>
                                                    mStatus &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
                                                    &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Class
                                                    Name:<br>
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc<br>
                                                    mCode &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
                                                    &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 9<br>
                                                    mMessage &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
                                                    &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Error
                                                    creating the
                                                    requested virtual
                                                    machine<br>
                                                    <br>
                                                    <br>
                                                    2012-04-17
                                                    04:07:37,497 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
                                                    (pool-5-thread-47)
                                                    Vds:
                                                    rhevh2.&lt;domain&gt;<br>
                                                    2012-04-17
                                                    04:07:37,497 ERROR<br>
                                                    [org.ovirt.engine.core.vdsbroker.VDSCommandBase]



                                                    (pool-5-thread-47)<br>
                                                    Command
                                                    MigrateStatusVDS
                                                    execution failed.
                                                    Exception:
                                                    VDSErrorException:<br>
                                                    VDSGenericException:
                                                    VDSErrorException:
                                                    Failed to
                                                    MigrateStatusVDS,<br>
                                                    error = Error
                                                    creating the
                                                    requested virtual
                                                    machine<br>
                                                    2012-04-17
                                                    04:07:37,497 INFO<br>
&nbsp;[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br>
                                                    (pool-5-thread-47)
                                                    FINISH,
                                                    MigrateStatusVDSCommand,
                                                    log id: 8dfce1b<br>
                                                    2012-04-17
                                                    04:07:37,528 WARN<br>
                                                    &nbsp;[org.ovirt.engine.core.bll.MigrateVmCommand]



                                                    (pool-5-thread-47)<br>
                                                    CanDoAction of
                                                    action MigrateVm
                                                    failed.<br>
Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM<br>
                                                    <br>
                                                    <br>
                                                    There is no problem
                                                    regarding storage or
                                                    anything like that.
                                                    If I shut<br>
                                                    down the vm I can
                                                    start it on the
                                                    other host without
                                                    any problems<br>
                                                    whatsoever.<br>
                                                    Do I need to do
                                                    something to make
                                                    live migration work?
                                                    Trying to dig in<br>
                                                    the logs on the
                                                    hypervisors, but
                                                    there's so much
                                                    stuff being logged
                                                    there...<br>
                                                    <br>
                                                    <br>
                                                  </div>
                                                </div>
_______________________________________________<br>
                                                Users mailing list<br>
                                                <a
                                                  moz-do-not-send="true"
href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
                                                <a
                                                  moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
                                              </blockquote>
                                              <br>
                                              vdsm log from the host
                                              that failed to create the
                                              VM?<br>
                                            </blockquote>
                                          </div>
                                          <br>
                                          <div>The vdsm log
                                            is&nbsp;frighteningly long, so I
                                            put it in pastebin:&nbsp;<a
                                              moz-do-not-send="true"
                                              href="http://pastebin.com/BEeqFCCb"
                                              target="_blank">http://pastebin.com/BEeqFCCb</a></div>
                                          <div>I don't really see
                                            anything worthwhile myself,
                                            but on the other hand I'm
                                            not sure what I'm looking
                                            for, so it's mostly a
                                            massive wall of text to me,
                                            but it should be somewhere
                                            in that time frame.</div>
                                        </blockquote>
                                      </div>
                                    </div>
                                    Here is the failure, "code 9" is not
                                    too informative, can you attach the
                                    libvirtd.log file too?<br>
                                    <br>
                                    Thread-49287::DEBUG::2012-04-17
                                    14:25:39,834::clientIF::81::vds::(wrapper)
                                    return migrationCreate with
                                    {'status': {'message': 'Error
                                    creating the requested virtual
                                    machine', 'code': 9}}
                                    <div><br>
                                      <blockquote type="cite"> <br>
                                        <fieldset></fieldset>
                                        <br>
                                        <pre>_______________________________________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>
<a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
                                      </blockquote>
                                      <br>
                                      <br>
                                    </div>
                                    <span><font color="#888888">
                                        <pre cols="72">-- 

Thanks,

Rami Vaknin, QE @ Red Hat, TLV, IL.
</pre>
                                      </font></span></div>
                                </blockquote>
                              </div>
                              <div><br>
                              </div>
                              <div>I think this is the relevant info
                                from the libvirt.log:</div>
                              <div>
                                <div>2012-04-17 14:30:53.768+0000: 8506:
                                  debug :
                                  virDomainLookupByUUIDString:1984 :
                                  conn=0x7fa9a00bcde0,
                                  uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a</div>
                                <div>2012-04-17 14:30:53.768+0000: 8506:
                                  debug : virDomainLookupByUUID:1941 :
                                  conn=0x7fa9a00bcde0,
                                  uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a</div>
                                <div>2012-04-17 14:30:53.771+0000: 8506:
                                  error :
                                  virNetClientProgramDispatchError:170 :
                                  Domain not found: no domain with
                                  matching uuid
                                  'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a'</div>
                              </div>
                              <div><br>
                              </div>
                              <div>What domain is this, is it the
                                storage domain?</div>
                            </blockquote>
                            <br>
                          </div>
                        </div>
                        Domain in libvirt is simply a vm.
                        <div><br>
                          <blockquote type="cite"> <br>
                            <fieldset></fieldset>
                            <br>
                            <pre>_______________________________________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>
<a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
                          </blockquote>
                          <br>
                          <br>
                          <pre cols="72">-- 

Thanks,

Rami Vaknin, QE @ Red Hat, TLV, IL.
</pre>
                        </div>
                      </div>
                    </blockquote>
                  </div>
                  <br>
                  <div>Alright, so what does this mean then? The machine
                    I'm migrating to can't see the vm?<br>
                    That doesn't make sense since I can manually start
                    it on that host..</div>
                </blockquote>
                <br>
              </div>
            </div>
            On second look at the vdsm.log I see that disk preparation
            in the destination host failed (timed out), try to take a
            look at vdsm.log at the destination.<br>
            In addition, for some reason getVmStats returns "Virtual
            machine does not exist" right before the migration.<br>
            <br>
            <br>
            Thread-49286::DEBUG::2012-04-17
            14:25:07,054::clientIF::76::vds::(wrapper)
            [10.23.30.130]::call getVmStats with
            ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}<br>
            Thread-49286::DEBUG::2012-04-17
            14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats
            with {'status': {'message': 'Virtual machine does not
            exist', 'code': 1}}<br>
            Thread-49287::DEBUG::2012-04-17
            14:25:07,381::clientIF::76::vds::(wrapper)
            [10.23.30.130]::call migrationCreate with ({'bridge':
            'ovirtmgmt', 'acpiEnable': 'true',.....<br>
            Thread-49287::DEBUG::2012-04-17
            14:25:08,156::libvirtvm::1504::vm.Vm::(waitForMigrationDestinationPrepare)

            vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::migration
            destination: waiting 31s for path preparation<br>
            <br>
            Thread-49287::DEBUG::2012-04-17
            14:25:39,458::libvirtvm::1507::vm.Vm::(waitForMigrationDestinationPrepare)

            vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::Timeout while
            waiting for path preparation<br>
            <br>
            14:25:07,054::clientIF::76::vds::(wrapper)
            [10.23.30.130]::call getVmStats with
            ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}<br>
            Thread-49286::DEBUG::2012-04-17
            14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats
            with {'status': {'message': 'Virtual machine does not
            exist', 'code': 1}}
            <div class="im"><br>
              <blockquote type="cite"> <br>
                <fieldset></fieldset>
                <br>
                <pre>_______________________________________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>
<a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
              </blockquote>
              <br>
              <br>
              <pre cols="72">-- 

Thanks,

Rami Vaknin, QE @ Red Hat, TLV, IL.
</pre>
            </div>
          </div>
        </blockquote>
      </div>
      The vdsm log is already included earlier in the conversation;&nbsp;<a
        moz-do-not-send="true" href="http://pastebin.com/BEeqFCCb">http://pastebin.com/BEeqFCCb</a>
      <div>I don't really know where to go from here. :/</div>
    </blockquote>
    <br>
    The exact issue that caused the prepareVolume to fail is not
    mentioned in the attached log.<br>
    <br>
    Can you please tell whether the attached vdsm log is from the
    migration source host or from the migration destination host, and
    attach as well the second missing log? The vdsm log from the host
    where the vm should run after the migration should contain a
    failed/hanged lvm command on the vm's disk.<br>
    <br>
    <br>
    <br>
    <blockquote
cite="mid:CAGKvcKUOXkNPKM1qGe2seVKWp2Z98JO5ehoDYs8vcgLF5p16Ww@mail.gmail.com"
      type="cite">
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
    </blockquote>
    <br>
    <br>
    <pre class="moz-signature" cols="72">-- 

Thanks,

Rami Vaknin, QE @ Red Hat, TLV, IL.
</pre>
  </body>
</html>