Re: [ovirt-users] VM Live Migration issues

27 Aug 2016

      This is a multi-part message in MIME format.
--------------AC0BFBC82D080F054CD895A0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

The migration is working fine with Cluster Switch set to "Legacy" Mode.

But I have run into a different problem. Yesterday, for some reason, I 
had to restart the hosts. But the Hosts never came online. Hosts were 
constantly in Non Responsive state. I manually fenced it, and Vds 
command timed out.

I found that NIC was down for some reason. I brought it up and again 
fenced the hosts. Again the Vds command timed out. I manually attempted 
to restart the VDSM service, but host reported there is no such service. 
Starting VDSMD service resulted in failure.

Finally, I had to forcefully remove the hosts, reinstalled base (CentOS 
7) on hosts and re-added them to oVirt with the storage domain intact. 
All hosts, storage domains went online and all VMs were intact.

Post reinstallation, I set the Cluster Switch mode to "Legacy" and tried 
migration and it worked fine.

I fail to understand what went wrong while rebooting the hosts.

-- 

Thanks & Regards,

Anantha Raghava

eXza Technology Consulting & Services

Do not print this e-mail unless required. Save Paper & trees.
On Friday 26 August 2016 10:12 PM, Anantha Raghava wrote:
...
Hi,
Then, I will switch the cluster switch type back to Legacy and then 
try to migrate. Will post the results here.
Any other suggestions?
--
Thanks & Regards,
Anantha Raghava
eXza Technology Consulting & Services
Do not print this e-mail unless required. Save Paper & trees.
On Friday 26 August 2016 09:58 PM, Yaniv Dary wrote:
...
OVS is experimental and there is a open item on making migration work:
https://bugzilla.redhat.com/show_bug.cgi?id=1362495
Yaniv Dary Technical Product Manager Red Hat Israel Ltd. 34 Jerusalem 
Road Building A, 4th floor Ra'anana, Israel 4350109 Tel : +972 (9) 
7692306 8272306 Email: ydary@redhat.com <mailto:ydary@redhat.com> IRC 
: ydary
On Fri, Aug 26, 2016 at 11:25 AM, Anantha Raghava 
<raghav@exzatechconsulting.com 
<mailto:raghav@exzatechconsulting.com>> wrote:
Hi,
Not at all.
Just newly created Virtual Machine. No user is accessing the VM
    yet. Only Ubuntu Trusty Tahr (14.04) OS with oVirt Guest Agent is
    installed. VM and both hosts are in the same network as well.
Applied migration policy is "Minimum Downtime". Bandwidth limit
    is set to Auto. Cluster switch is set to "OVS" I can send the
    screen shot tomorrow.
Host Hardware configuration. Intel Xeon CPU, 2 Socket X 16 Core
    in each socket. Installed Memory is 256 GB on each host. I have 4
    X 10Gbps NIC, 2 given for VM traffic and 2 given for iSCSI
    traffic. oVirt Management is on a separate 1 X 1Gbps NIC.
Yet, migration fails.
--
Thanks & Regards,
Anantha Raghava
eXza Technology Consulting & Services
Do not print this e-mail unless required. Save Paper & trees.
    On Friday 26 August 2016 09:31 PM, Yaniv Dary wrote:
...
Is the VM very busy?
    Did you apply the new cluster migration policies?
Yaniv Dary Technical Product Manager Red Hat Israel Ltd. 34
    Jerusalem Road Building A, 4th floor Ra'anana, Israel 4350109
    Tel : +972 (9) 7692306 <tel:%2B972%20%289%29%207692306> 8272306
    Email: ydary@redhat.com <mailto:ydary@redhat.com> IRC : ydary
On Fri, Aug 26, 2016 at 1:54 AM, Anantha Raghava
    <raghav@exzatechconsulting.com
    <mailto:raghav@exzatechconsulting.com>> wrote:
Hi,
In our setup we have configured two hosts, both of same CPU
        type, same amount of memory and Master storage domain is
        created on iSCSI storage and is live.
I created a single VM with Ubuntu Trusty as OS. It installed
        properly and when I attempted to migrate the running VM, the
        migration failed.
Engine log, Host 1 log and Host 2 logs are attached for your
        reference. Since logs are running into several MBs, I have
        compressed them and attached here.
Can someone help us to solve this issue?
--
Thanks & Regards,
Anantha Raghava
_______________________________________________
        Users mailing list
        Users@ovirt.org <mailto:Users@ovirt.org>
        http://lists.ovirt.org/mailman/listinfo/users
        <http://lists.ovirt.org/mailman/listinfo/users>
--------------AC0BFBC82D080F054CD895A0
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <p>Hi,</p>
    <p>The migration is working fine with Cluster Switch set to "Legacy"
      Mode.<br>
    </p>
    <div class="moz-signature">
      <meta http-equiv="content-type" content="text/html; charset=utf-8">
      <title></title>
      <meta name="generator" content="LibreOffice 5.0.3.2 (Linux)">
      <meta name="created" content="00:00:00">
      <meta name="changedby" content="Anantha Raghava">
      <meta name="changed" content="2016-01-05T17:20:50.677541300">
      <meta name="created" content="00:00:00">
      <meta name="changedby" content="Anantha Raghava">
      <meta name="changed" content="2015-12-20T09:03:26.251763811">
      <meta name="created" content="2015-02-21T00:00:00">
      <meta name="changedby" content="Anantha Raghava">
      <meta name="changed" content="2015-12-20T09:02:11.666821134">
      <style type="text/css">
		@page { margin: 2cm }
		p { margin-bottom: 0.25cm; color: #000000; line-height: 120% }
		address { color: #000000 }
		a:link { so-language: zxx }
	</style>
      <p style="margin-bottom: 0cm">But I have run into a different
        problem. Yesterday, for some reason, I had to restart the hosts.
        But the Hosts never came online. Hosts were constantly in Non
        Responsive state. I manually fenced it, and Vds command timed
        out.</p>
      <p style="margin-bottom: 0cm">I found that NIC was down for some
        reason. I brought it up and again fenced the hosts. Again the
        Vds command timed out. I manually attempted to restart the VDSM
        service, but host reported there is no such service. Starting
        VDSMD service resulted in failure. <br>
      </p>
      <p style="margin-bottom: 0cm">Finally, I had to forcefully remove
        the hosts, reinstalled base (CentOS 7) on hosts and re-added
        them to oVirt with the storage domain intact. All hosts, storage
        domains went online and all VMs were intact.</p>
      <p style="margin-bottom: 0cm">Post reinstallation, I set the
        Cluster Switch mode to "Legacy" and tried migration and it
        worked fine.<br>
      </p>
      <p style="margin-bottom: 0cm">I fail to understand what went wrong
        while rebooting the hosts.<br>
      </p>
      <p>-- </p>
      <p style="margin-bottom: 0cm; line-height: 100%"><font face="Times
          New Roman, serif">Thanks
          & Regards,</font></p>
      <p style="margin-bottom: 0cm; line-height: 100%"><br>
      </p>
      <address style="line-height: 100%"><font face="Times New Roman,
          serif">Anantha
          Raghava</font></address>
      <address style="line-height: 100%"><font face="Times New Roman,
          serif">eXza
          Technology Consulting & Services</font></address>
      <font color="#66cc00"><font face="Times New Roman, serif"><br>
          Do
          not print this e-mail unless required. Save Paper & trees.</font></font>
    </div>
    <div class="moz-cite-prefix">On Friday 26 August 2016 10:12 PM,
      Anantha Raghava wrote:<br>
    </div>
    <blockquote
      cite="mid:ea203138-3169-7211-b306-66d82e9bc7a0@exzatechconsulting.com"
      type="cite">
      <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
      <p>Hi,</p>
      <p>Then, I will switch the cluster switch type back to Legacy and
        then try to migrate. Will post the results here.</p>
      <p>Any other suggestions?<br>
      </p>
      <div class="moz-signature">
        <meta http-equiv="content-type" content="text/html;
          charset=utf-8">
        <title></title>
        <meta name="generator" content="LibreOffice 5.0.3.2 (Linux)">
        <meta name="created" content="00:00:00">
        <meta name="changedby" content="Anantha Raghava">
        <meta name="changed" content="2016-01-05T17:20:50.677541300">
        <meta name="created" content="00:00:00">
        <meta name="changedby" content="Anantha Raghava">
        <meta name="changed" content="2015-12-20T09:03:26.251763811">
        <meta name="created" content="2015-02-21T00:00:00">
        <meta name="changedby" content="Anantha Raghava">
        <meta name="changed" content="2015-12-20T09:02:11.666821134">
        <style type="text/css">
		@page { margin: 2cm }
		p { margin-bottom: 0.25cm; color: #000000; line-height: 120% }
		address { color: #000000 }
		a:link { so-language: zxx }
	</style>
        <p>-- </p>
        <p style="margin-bottom: 0cm; line-height: 100%"><font
            face="Times New Roman, serif">Thanks & Regards,</font></p>
        <p style="margin-bottom: 0cm; line-height: 100%"><br>
        </p>
        <address style="line-height: 100%"><font face="Times New Roman,
            serif">Anantha Raghava</font></address>
        <address style="line-height: 100%"><font face="Times New Roman,
            serif">eXza Technology Consulting & Services</font></address>
        <br>
        <p style="margin-bottom: 0cm; line-height: 100%"><br>
        </p>
        <p style="margin-bottom: 0cm; line-height: 100%"><font
            color="#66cc00"><font face="Times New Roman, serif">Do not
              print this e-mail unless required. Save Paper & trees.</font></font></p>
      </div>
      <div class="moz-cite-prefix">On Friday 26 August 2016 09:58 PM,
        Yaniv Dary wrote:<br>
      </div>
      <blockquote
cite="mid:CACKMAy8yvBUnJGrEtWXkP_VH=7vE+sC8pAQoKm3v0b0ZYi_jwg@mail.gmail.com"
        type="cite">
        <div dir="ltr">OVS is experimental and there is a open item on
          making migration work:
          <div><a moz-do-not-send="true"
              href="https://bugzilla.redhat.com/show_bug.cgi?id=1362495">https://bugzilla.redhat.com/show_bug.cgi?id=1362495</a> </div>
        </div>
        <div class="gmail_extra"><br clear="all">
          <div>
            <div class="gmail_signature"
              data-smartmail="gmail_signature">
              <div dir="ltr">
                <div>
                  <div dir="ltr">
                    <pre cols="72"><span style="font-family:arial,helvetica,sans-serif">Yaniv Dary
Technical Product Manager
Red Hat Israel Ltd.
34 Jerusalem Road
Building A, 4th floor
Ra'anana, Israel 4350109

Tel : +972 (9) 7692306
        8272306
Email: <a moz-do-not-send="true" href="mailto:ydary@redhat.com" target="_blank">ydary@redhat.com</a>
IRC : ydary</span></pre>
                  </div>
                </div>
              </div>
            </div>
          </div>
          <br>
          <div class="gmail_quote">On Fri, Aug 26, 2016 at 11:25 AM,
            Anantha Raghava <span dir="ltr"><<a
                moz-do-not-send="true"
                href="mailto:raghav@exzatechconsulting.com"
                target="_blank">raghav@exzatechconsulting.com</a>></span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div bgcolor="#FFFFFF" text="#000000">
                <p><font face="Liberation Serif">Hi,</font></p>
                <p><font face="Liberation Serif">Not at all.</font></p>
                <p><font face="Liberation Serif">Just newly created
                    Virtual Machine. No user is accessing the VM yet.
                    Only Ubuntu Trusty Tahr (14.04) OS with oVirt Guest
                    Agent is installed. VM and both hosts are in the
                    same network as well. <br>
                  </font></p>
                <p><font face="Liberation Serif">Applied migration
                    policy is "Minimum Downtime". Bandwidth limit is set
                    to Auto. Cluster switch is set to "OVS" I can send
                    the screen shot tomorrow. <br>
                  </font></p>
                <p><font face="Liberation Serif">Host Hardware
                    configuration. Intel Xeon CPU, 2 Socket X 16 Core in
                    each socket. Installed Memory is 256 GB on each
                    host. I have 4 X 10Gbps NIC, 2 given for VM traffic
                    and 2 given for iSCSI traffic. oVirt Management is
                    on a separate 1 X 1Gbps NIC.</font></p>
                <p><font face="Liberation Serif">Yet, migration fails.</font><br>
                </p>
                <div>
                  <p>-- </p>
                  <p style="margin-bottom:0cm;line-height:100%"><font
                      face="Times New Roman, serif">Thanks &
                      Regards,</font></p>
                  <p style="margin-bottom:0cm;line-height:100%"><br>
                  </p>
                  <address style="line-height:100%"><font face="Times
                      New Roman, serif">Anantha Raghava</font></address>
                  <address style="line-height:100%"><font face="Times
                      New Roman, serif">eXza Technology Consulting &
                      Services<br>
                      <br>
                    </font></address>
                  <font color="#66cc00"><font face="Times New Roman,
                      serif">Do not print this e-mail unless required.
                      Save Paper & trees.</font></font> </div>
                <div>
                  <div class="h5">
                    <div>On Friday 26 August 2016 09:31 PM, Yaniv Dary
                      wrote:<br>
                    </div>
                    <blockquote type="cite">
                      <div dir="ltr">Is the VM very busy?
                        <div>Did you apply the new cluster migration
                          policies?</div>
                      </div>
                      <div class="gmail_extra"><br clear="all">
                        <div>
                          <div data-smartmail="gmail_signature">
                            <div dir="ltr">
                              <div>
                                <div dir="ltr">
                                  <pre cols="72"><span style="font-family:arial,helvetica,sans-serif">Yaniv Dary
Technical Product Manager
Red Hat Israel Ltd.
34 Jerusalem Road
Building A, 4th floor
Ra'anana, Israel 4350109

Tel : <a moz-do-not-send="true" href="tel:%2B972%20%289%29%207692306" value="+97297692306" target="_blank">+972 (9) 7692306</a>
        8272306
Email: <a moz-do-not-send="true" href="mailto:ydary@redhat.com" target="_blank">ydary@redhat.com</a>
IRC : ydary</span></pre>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                        <br>
                        <div class="gmail_quote">On Fri, Aug 26, 2016 at
                          1:54 AM, Anantha Raghava <span dir="ltr"><<a
                              moz-do-not-send="true"
                              href="mailto:raghav@exzatechconsulting.com"
                              target="_blank">raghav@exzatechconsulting.com</a><wbr>></span>
                          wrote:<br>
                          <blockquote class="gmail_quote"
                            style="margin:0 0 0 .8ex;border-left:1px
                            #ccc solid;padding-left:1ex">
                            <div bgcolor="#FFFFFF" text="#000000">
                              <p><font face="Liberation Serif">Hi,</font></p>
                              <p><font face="Liberation Serif">In our
                                  setup we have configured two hosts,
                                  both of same CPU type, same amount of
                                  memory and Master storage domain is
                                  created on iSCSI storage and is live.</font></p>
                              <p><font face="Liberation Serif">I created
                                  a single VM with Ubuntu Trusty as OS.
                                  It installed properly and when I
                                  attempted to migrate the running VM,
                                  the migration failed. <br>
                                </font></p>
                              <p><font face="Liberation Serif">Engine
                                  log, Host 1 log and Host 2 logs are
                                  attached for your reference. Since
                                  logs are running into several MBs, I
                                  have compressed them and attached
                                  here.<br>
                                </font></p>
                              <p><font face="Liberation Serif">Can
                                  someone help us to solve this issue?</font><br>
                              </p>
                              <br>
                              <div>--
                                <p
                                  style="margin-bottom:0cm;line-height:100%"><font
                                    face="Times New Roman, serif">Thanks
                                    & Regards,</font></p>
                                <address style="line-height:100%"><font
                                    face="Times New Roman, serif">Anantha
                                    Raghava</font></address>
                                <br>
                              </div>
                            </div>
                            <br>
                            ______________________________<wbr>_________________<br>
                            Users mailing list<br>
                            <a moz-do-not-send="true"
                              href="mailto:Users@ovirt.org"
                              target="_blank">Users@ovirt.org</a><br>
                            <a moz-do-not-send="true"
                              href="http://lists.ovirt.org/mailman/listinfo/users"
                              rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
                            <br>
                          </blockquote>
                        </div>
                        <br>
                      </div>
                    </blockquote>
                    <br>
                  </div>
                </div>
              </div>
            </blockquote>
          </div>
          <br>
        </div>
      </blockquote>
      <br>
    </blockquote>
    <br>
  </body>
</html>

--------------AC0BFBC82D080F054CD895A0--