Re: [ovirt-devel] [ OST Failure Report ] [ oVirtMaster ] [ 07-11-2017 ] [007_sd_reattach.deactivate_storage_domain ]

Friday, 8 December 2017

This is a multi-part message in MIME format.
--------------78AFAE881715BC23A10C5F16
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

I opened a bug: https://bugzilla.redhat.com/show_bug.cgi?id=1523813
Allon, can you please assign someone to help fix this test?
Please let me know if you need help from me.

Thanks!
Dafna

On 12/07/2017 11:59 AM, Yaniv Kaul wrote:
...

 On Thu, Dec 7, 2017 at 1:30 PM, Eyal Shenitzky <eshenitz(a)redhat.com
 <mailto:eshenitz@redhat.com>> wrote:

     I think that the maybe the QE can share their methods on how to
     avoid those issues. 
     From what I remember, before deactivating storage domain they make
     sure that there are no running tasks related to
     the storage domain.

 Looks like an easy fix is to wrap it with try, except sdk4.Error and
 let it sit within the testlib.assert_true_within_short() loop.
 Y.

     On Thu, Dec 7, 2017 at 1:22 PM, Yaniv Kaul <ykaul(a)redhat.com
     <mailto:ykaul@redhat.com>> wrote:

         On Thu, Dec 7, 2017 at 1:12 PM, Dafna Ron <dron(a)redhat.com
         <mailto:dron@redhat.com>> wrote:

             Maor, I either need to get new glasses or a magnifier
             glass to read what you wrote :-P
             when you say running tasks - these are actually running
             tasks that may be running because of other tests in ost -
             correct? wouldn't killing or blocking those can cause
             other tests to fail?

         It might well be the OVF update. How can we, from the API,
         wait for those tasks to complete? Or should we catch exception
         and retry?
         Y.

             On 12/07/2017 11:06 AM, Maor Lipchuk wrote:
>
>             CANNOT_DEACTIVATE_DOMAIN_WITH_TASKS is a known issue, the
>             problem is that we might have tasks which will start
>             running internally using scheduling (like OVF_UPDATE) and
>             we can't really know how much time every task will take
>             until it will end.
>
>             Even if we check that there are no running tasks it will
>             not guaranteethat no task will start until you deactivate
>             the storage domain.
>
>             I think that the best solution for it is an engine
>             support to cancel running tasks or block tasks from running.
>
>
>             On Thu, Dec 7, 2017 at 12:14 PM, Dafna Ron
>             <dron(a)redhat.com <mailto:dron@redhat.com>> wrote:
>
>                 Hi,
>
>                 We had a failure on master basic suite for test
>                 007_sd_reattach.deactivate_storage_domain.
>
>                 The failure was that we failed to deactivate domain
>                 due to running tasks.
>
>                 It does not seem to be related to the patch it was
>                 testing and I think that the test itself needs to be
>                 modified to check there are no running tasks.
>
>                 Is there perhaps a way to query if there are running
>                 tasks before running the command? can you please take
>                 a look at the test on OST?
>
>                 **
>
>                 *Link and headline of suspected patches:**Not related
>                 to error***
>
>                 **
>
>                 **
>
>                 *Link to Job:
>                 http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4319/
>                
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4319/>*
>
>                 ***
>
>                 Link to all logs:
>                
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4319/artifact/
>                
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4319/artifa...
>
>
>                 (Relevant) error snippet from the log:
>
>                 <error>
>
>                 *2017-12-06 20:13:23,166-05 WARN
>                
[org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainWithOvfUpdateCommand]
>                 (default task-7)
>                 [d82880e8-1d40-4a3b-a1ba-3362f2f130a0] Validation of
>                 action 'DeactivateStorageDomainWithOvfUpdate' fa iled
>                 for user admin@internal-authz. Reasons:
>                
VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ERROR_CANNOT_DEACTIVATE_DOMAIN_WITH_TASKS
>                 2017-12-06 20:13:23,167-05 INFO
>                
[org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainWithOvfUpdateCommand]
>                 (default task-7)
>                 [d82880e8-1d40-4a3b-a1ba-3362f2f130a0] Lock freed to
>                 object 'EngineLock:{exclusiveLocks='[ea2fd992-8a
>                 a4-44fe-aa43-e96754a975ba=STORAGE]',
>                
sharedLocks='[5e0a0183-0e25-4f43-b5b0-0cfb5510248e=POOL]'}'
>                 2017-12-06 20:13:23,172-05 DEBUG
>                
[org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
>                 (default task-7)
>                 [d82880e8-1d40-4a3b-a1ba-3362f2f130a0] method:
>                 runAction, params:
>                 [DeactivateStorageDomainWithOvfUpdate, DeactivateSto
>                
rageDomainWithOvfUpdateParameters:{commandId='630c28e1-41ab-43db-9755-a2bb870dbcb3',
>                 user='null', commandType='Unknown'}], timeElapsed:
>                 65ms 2017-12-06 20:13:23,176-05 ERROR
>                 [org.ovirt.engine.api.restapi.resource.AbstractBackendResource]
>                 (default task-7) [] Operation Failed: [Cannot
>                 deactivate Storage while there are running tasks on
>                 this Storage. -Please wait until tasks will finish
>                 and try again.] *
>
>                 *<error>
>
>
>
>
>
>                 _______________________________________________
>                 Devel mailing list
>                 Devel(a)ovirt.org <mailto:Devel@ovirt.org>
>                 http://lists.ovirt.org/mailman/listinfo/devel
>                 <http://lists.ovirt.org/mailman/listinfo/devel>
>
>

             _______________________________________________
             Devel mailing list
             Devel(a)ovirt.org <mailto:Devel@ovirt.org>
             http://lists.ovirt.org/mailman/listinfo/devel
             <http://lists.ovirt.org/mailman/listinfo/devel>

         _______________________________________________
         Devel mailing list
         Devel(a)ovirt.org <mailto:Devel@ovirt.org>
         http://lists.ovirt.org/mailman/listinfo/devel
         <http://lists.ovirt.org/mailman/listinfo/devel>

     -- 
     Regards,
     Eyal Shenitzky

--------------78AFAE881715BC23A10C5F16
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">I opened a bug:
      <a class="moz-txt-link-freetext"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1523813">h...
      Allon, can you please assign someone to help fix this test? <br>
      Please let me know if you need help from me. <br>
      <br>
      Thanks!<br>
      Dafna<br>
      <br>
      <br>
      On 12/07/2017 11:59 AM, Yaniv Kaul wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAJgorsZ0iXzc0WpWZHBe0=NtY53cs_wGRY4X8KQqS3oa0YCX0A@mail.gmail.com">
      <div dir="ltr"><br>
        <div class="gmail_extra"><br>
          <div class="gmail_quote">On Thu, Dec 7, 2017 at 1:30 PM, Eyal
            Shenitzky <span dir="ltr">&lt;<a
                href="mailto:eshenitz@redhat.com" target="_blank"

moz-do-not-send=&quot;true&quot;&gt;eshenitz(a)redhat.com&lt;/a&gt;&amp;gt;&lt;/span&gt;
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div dir="ltr">I think that the maybe the QE can share
                their methods on how to avoid those issues. 
                <div>From what I remember, before deactivating storage
                  domain they make sure that there are no running tasks
                  related to</div>
                <div>the storage domain.</div>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>Looks like an easy fix is to wrap it with try, except
              sdk4.Error and let it sit within the
              testlib.assert_true_within_short() loop.</div>
            <div>Y.</div>
            <div> </div>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div class="gmail_extra">
                <div>
                  <div class="h5"><br>
                    <div class="gmail_quote">On Thu, Dec 7, 2017 at 1:22
                      PM, Yaniv Kaul <span dir="ltr">&lt;<a
                          href="mailto:ykaul@redhat.com"
target="_blank"

moz-do-not-send=&quot;true&quot;&gt;ykaul(a)redhat.com&lt;/a&gt;&amp;gt;&lt;/span&gt;
                      wrote:<br>
                      <blockquote class="gmail_quote" style="margin:0 0
                        0 .8ex;border-left:1px #ccc
                        solid;padding-left:1ex">
                        <div dir="ltr"><br>
                          <div class="gmail_extra"><br>
                            <div class="gmail_quote">On Thu, Dec 7, 2017
                              at 1:12 PM, Dafna Ron <span
dir="ltr">&lt;<a
                                  href="mailto:dron@redhat.com"
                                  target="_blank"
moz-do-not-send=&quot;true&quot;&gt;dron(a)redhat.com&lt;/a&gt;&amp;gt;&lt;/span&gt;
                              wrote:<br>
                              <blockquote class="gmail_quote"
                                style="margin:0 0 0 .8ex;border-left:1px
                                #ccc solid;padding-left:1ex">
                                <div text="#000000"
bgcolor="#FFFFFF">
                                  <div
class="m_8942236351456563473m_-5014669676915136678m_3333631383734454308moz-cite-prefix">Maor,
                                    I either need to get new glasses or
                                    a magnifier glass to read what you
                                    wrote :-P<br>
                                    when you say running tasks - these
                                    are actually running tasks that may
                                    be running because of other tests in
                                    ost - correct? wouldn't killing or
                                    blocking those can cause other tests
                                    to fail? <br>
                                  </div>
                                </div>
                              </blockquote>
                              <div><br>
                              </div>
                              <div>It might well be the OVF update. How
                                can we, from the API, wait for those
                                tasks to complete? Or should we catch
                                exception and retry?</div>
                              <div>Y.</div>
                              <div> </div>
                              <blockquote class="gmail_quote"
                                style="margin:0 0 0 .8ex;border-left:1px
                                #ccc solid;padding-left:1ex">
                                <div text="#000000"
bgcolor="#FFFFFF">
                                  <div
class="m_8942236351456563473m_-5014669676915136678m_3333631383734454308moz-cite-prefix">
                                    <div>
                                      <div

class="m_8942236351456563473m_-5014669676915136678h5">
                                        <br>
                                        On 12/07/2017 11:06 AM, Maor
                                        Lipchuk wrote:<br>
                                      </div>
                                    </div>
                                  </div>
                                  <div>
                                    <div

class="m_8942236351456563473m_-5014669676915136678h5">
                                      <blockquote type="cite">
                                        <div dir="ltr"><span
id="m_8942236351456563473m_-5014669676915136678m_3333631383734454308gmail-m_-5329876534188665956docs-internal-guid-5859b7a1-3073-5414-18b7-2d31991889e8">
                                            <p

style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span
style="background-color:transparent;vertical-align:baseline"><span
style="color:rgb(0,0,0);font-family:monospace;white-space:pre-wrap">CANNOT_DEACTIVATE_DOMAIN_WITH_</span><span
style="color:rgb(0,0,0);font-family:monospace;white-space:pre-wrap"><wbr>TASKS
</span><font><font
                                                    color="#000000"
                                                    size="1"
                                                    face="Arial"><span
style="white-space:pre-wrap">is a known issue, the problem is that we might
have tasks which will start running internally using scheduling (like OVF_UPDATE) and we
can't really know how much time every task will take until it will
end.</span></font></font></span></p>
                                            <p

style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span
style="background-color:transparent;vertical-align:baseline"><font><font
                                                    color="#000000"
                                                    size="1"
                                                    face="Arial"><span
style="white-space:pre-wrap">Even if we check that there are no running tasks
it will not </span></font><font
                                                    color="#000000"><span
style="white-space:pre-wrap">guarantee</span></font><font
                                                    color="#000000"><font
                                                      size="1"
                                                      face="Arial"><span
style="white-space:pre-wrap"> that no task will start until you deactivate
the storage
domain.</span></font></font></font></span></p>
                                            <p

style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span
style="background-color:transparent;vertical-align:baseline"><font><font
                                                    color="#000000"><font
                                                      size="1"
                                                      face="Arial"><span
style="white-space:pre-wrap">I think that the best solution for it is an
engine support to cancel running tasks or block tasks from
running.</span></font></font></font></span></p>
                                          </span></div>
                                        <div
class="gmail_extra"><br>
                                          <div class="gmail_quote">On
                                            Thu, Dec 7, 2017 at 12:14
                                            PM, Dafna Ron <span
                                              dir="ltr">&lt;<a
                                                href="mailto:dron@redhat.com"
                                                target="_blank"

moz-do-not-send=&quot;true&quot;&gt;dron(a)redhat.com&lt;/a&gt;&amp;gt;&lt;/span&gt;
                                            wrote:<br>
                                            <blockquote
                                              class="gmail_quote"
                                              style="margin:0 0 0
                                              .8ex;border-left:1px #ccc
                                              solid;padding-left:1ex">
                                              <div text="#000000"
                                                bgcolor="#FFFFFF">
                                                <p>Hi, <br>
                                                </p>
                                                <p>We had a failure on
                                                  master basic suite for
                                                  test

007_sd_reattach.deactivate_sto<wbr>rage_domain.
                                                  <br>
                                                </p>
                                                <p>The failure was that
                                                  we failed to
                                                  deactivate domain due
                                                  to running tasks. <br>
                                                </p>
                                                <p>It does not seem to
                                                  be related to the
                                                  patch it was testing
                                                  and I think that the
                                                  test itself needs to
                                                  be modified to check
                                                  there are no running
                                                  tasks. <br>
                                                </p>
                                                <p>Is there perhaps a
                                                  way to query if there
                                                  are running tasks
                                                  before running the
                                                  command? can you
                                                  please take a look at
                                                  the test on OST? <br>
                                                </p>
                                                <p><b
                                                    style="font-weight:normal"
id="m_8942236351456563473m_-5014669676915136678m_3333631383734454308m_-5329876534188665956docs-internal-guid-5859b7a1-3073-5414-18b7-2d31991889e8">
                                                  </b></p>
                                                <p dir="ltr"

style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b
style="font-weight:normal"
id="m_8942236351456563473m_-5014669676915136678m_3333631383734454308m_-5329876534188665956docs-internal-guid-5859b7a1-3073-5414-18b7-2d31991889e8"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link
and headline of suspected patches:</span><b
style="font-weight:normal"
id="m_8942236351456563473m_-5014669676915136678m_3333631383734454308m_-5329876534188665956docs-internal-guid-5859b7a1-3073-5414-18b7-2d31991889e8"><b>
                                                        Not related to

error</b></b></b></p>
                                                <b
                                                  style="font-weight:normal"
id="m_8942236351456563473m_-5014669676915136678m_3333631383734454308m_-5329876534188665956docs-internal-guid-5859b7a1-3073-5414-18b7-2d31991889e8">
                                                </b>
                                                <p><b
                                                    style="font-weight:normal"
id="m_8942236351456563473m_-5014669676915136678m_3333631383734454308m_-5329876534188665956docs-internal-guid-5859b7a1-3073-5414-18b7-2d31991889e8">
                                                  </b></p>
                                                <p dir="ltr"

style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b
style="font-weight:normal"
id="m_8942236351456563473m_-5014669676915136678m_3333631383734454308m_-5329876534188665956docs-internal-guid-5859b7a1-3073-5414-18b7-2d31991889e8"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link
to Job: <a
class="m_8942236351456563473m_-5014669676915136678m_3333631383734454308m_-5329876534188665956moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
target="_blank"
moz-do-not-send="true">http://jenkins.ovirt.org/job/o<wbr...
                                                <b
                                                  style="font-weight:normal"
id="m_8942236351456563473m_-5014669676915136678m_3333631383734454308m_-5329876534188665956docs-internal-guid-5859b7a1-3073-5414-18b7-2d31991889e8">
                                                </b><b
                                                  style="font-weight:normal"
id="m_8942236351456563473m_-5014669676915136678m_3333631383734454308m_-5329876534188665956docs-internal-guid-5859b7a1-3073-5414-18b7-2d31991889e8"><br>
                                                  <p dir="ltr"

style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link
to all logs: <a
class="m_8942236351456563473m_-5014669676915136678m_3333631383734454308m_-5329876534188665956moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
target="_blank"
moz-do-not-send="true">http://jenkins.ovirt.org/job/o<wbr...
                                                  <br>
                                                  <p dir="ltr"

style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">(Relevant)
error snippet from the log: </span></p>
                                                  <p dir="ltr"

style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">&lt;error&gt;</span></p>
                                                  <p dir="ltr"

style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">
</span></p>
                                                  <p dir="ltr"

style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><b>2017-12-06
20:13:23,166-05 WARN 
[org.ovirt.engine.core.bll.sto<wbr>rage.domain.DeactivateStorageD<wbr>omainWithOvfUpdateCommand]
(default task-7) [d82880e8-1d40-4a3b-a1ba-3362f<wbr>2f130a0] Validation of action
'DeactivateStorageDomainWithOv<wbr>fUpdate' fa
iled for user admin@internal-authz. Reasons:
VAR__TYPE__STORAGE__DOMAIN,VAR<wbr>__ACTION__DEACTIVATE,ERROR_CAN<wbr>NOT_DEACTIVATE_DOMAIN_WITH_TAS<wbr>KS
2017-12-06 20:13:23,167-05 INFO 
[org.ovirt.engine.core.bll.sto<wbr>rage.domain.DeactivateStorageD<wbr>omainWithOvfUpdateCommand]
(default task-7) [d82880e8-1d40-4a3b-a1ba-3362f<wbr>2f130a0] Lock freed to object
'EngineLock:{exclusiveLocks='[<wbr>ea2fd992-8a
a4-44fe-aa43-e96754a975ba=STOR<wbr>AGE]',
sharedLocks='[5e0a0183-0e25-4f<wbr>43-b5b0-0cfb5510248e=POOL]'}'
2017-12-06 20:13:23,172-05 DEBUG
[org.ovirt.engine.core.common.<wbr>di.interceptor.DebugLoggingInt<wbr>erceptor]
(default task-7) [d82880e8-1d40-4a3b-a1ba-3362f<wbr>2f130a0] method: runAction,
params: [DeactivateStorageDomainWithOv<wbr>fUpdate, DeactivateSto
rageDomainWithOvfUpdateParamet<wbr>ers:{commandId='630c28e1-41ab-<wbr>43db-9755-a2bb870dbcb3',
user='null', commandType='Unknown'}], timeElapsed: 65ms
2017-12-06 20:13:23,176-05 ERROR
[org.ovirt.engine.api.restapi.<wbr>resource.AbstractBackendResour<wbr>ce]
(default task-7) [] Operation Failed: [Cannot deactivate Storage while there are running
tasks on this Storage.
-Please wait until tasks will finish and try again.]
</b>
</span></p>
                                                  <p dir="ltr"

style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">
</span></p>
                                                </b>&lt;error&gt;<br
class="m_8942236351456563473m_-5014669676915136678m_3333631383734454308m_-5329876534188665956Apple-interchange-newline">
                                                <p><br>
                                                </p>
                                                <p><br>
                                                </p>
                                                <p><br>
                                                </p>
                                              </div>
                                              <br>
______________________________<wbr>_________________<br>
                                              Devel mailing list<br>
                                              <a
                                                href="mailto:Devel@ovirt.org"
                                                target="_blank"

moz-do-not-send=&quot;true&quot;&gt;Devel(a)ovirt.org&lt;/a&gt;&lt;br&gt;
                                              <a

href="http://lists.ovirt.org/mailman/listinfo/devel"
                                                rel="noreferrer"
                                                target="_blank"

moz-do-not-send="true">http://lists.ovirt.org/mailman<wbr...
                                            </blockquote>
                                          </div>
                                          <br>
                                        </div>
                                      </blockquote>
                                      <p><br>
                                      </p>
                                    </div>
                                  </div>
                                </div>
                                <br>

______________________________<wbr>_________________<br>
                                Devel mailing list<br>
                                <a href="mailto:Devel@ovirt.org"
                                  target="_blank"
moz-do-not-send=&quot;true&quot;&gt;Devel(a)ovirt.org&lt;/a&gt;&lt;br&gt;
                                <a

href="http://lists.ovirt.org/mailman/listinfo/devel"
                                  rel="noreferrer" target="_blank"

moz-do-not-send="true">http://lists.ovirt.org/mailman<wbr...
                              </blockquote>
                            </div>
                            <br>
                          </div>
                        </div>
                        <br>

______________________________<wbr>_________________<br>
                        Devel mailing list<br>
                        <a href="mailto:Devel@ovirt.org"
target="_blank"

moz-do-not-send=&quot;true&quot;&gt;Devel(a)ovirt.org&lt;/a&gt;&lt;br&gt;
                        <a
                          href="http://lists.ovirt.org/mailman/listinfo/devel"
                          rel="noreferrer" target="_blank"

moz-do-not-send="true">http://lists.ovirt.org/mailman<wbr...
                      </blockquote>
                    </div>
                    <br>
                    <br clear="all">
                    <div><br>
                    </div>
                  </div>
                </div>
                <span class="HOEnZb"><font
color="#888888">-- <br>
                    <div class="m_8942236351456563473gmail_signature"
                      data-smartmail="gmail_signature">
                      <div dir="ltr">Regards,
                        <div>Eyal Shenitzky</div>
                      </div>
                    </div>
                  </font></span></div>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <p><br>
    </p>
  </body>
</html>

--------------78AFAE881715BC23A10C5F16--

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [ovirt-devel] [ OST Failure Report ] [ oVirtMaster ] [ 07-11-2017 ] [007_sd_reattach.deactivate_storage_domain ]