<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 3, 2017 at 7:20 PM, Simone Tiraboschi <span dir="ltr">&lt;<a href="mailto:stirabos@redhat.com" target="_blank">stirabos@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="gmail-">On Fri, Feb 3, 2017 at 5:22 PM, Ralf Schenk <span dir="ltr">&lt;<a href="mailto:rs@databay.de" target="_blank">rs@databay.de</a>&gt;</span> wrote:<br></span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF">
    <p>Hello,</p>
    <p>of course:</p><span class="gmail-">
    <p>[root@microcloud27 mnt]# sanlock client status<br>
      daemon 8a93c9ea-e242-408c-a63d-a9356b<wbr>b22df5.microcloud<br>
      p -1 helper<br>
      p -1 listener<br>
      p -1 status<br>
    </p>
    <p>sanlock.log attached. (Beginning 2017-01-27 where everything was
      fine)</p></span></div></blockquote>Thanks, the issue is here:<div><pre class="gmail-m_132298329271851776gmail-aLF-aPX-K0-aPE gmail-m_132298329271851776gmail-aLF-aPX-aLK-ayr-auR">2017-02-02 19:01:22+0100 4848 [1048]: s36 lockspace 7c8deaa8-be02-4aaf-b9b4-<wbr>ddc8da99ad96:3:/rhev/data-<wbr>center/mnt/glusterSD/<wbr>glusterfs.rxmgmt.databay.de:_<wbr>engine/7c8deaa8-be02-4aaf-<wbr>b9b4-ddc8da99ad96/dom_md/ids:0
2017-02-02 19:03:42+0100 4988 [12983]: s36 delta_acquire host_id 3 busy1 3 15 13129 7ad427b1-fbb6-4cee-b9ee-<wbr>01f596fddfbb.microcloud
2017-02-02 19:03:43+0100 4989 [1048]: s36 add_lockspace fail result -262</pre></div><div>Could you please check if you have other hosts contending for the same ID (id=3 in this case).</div></div></div></div></blockquote><div><br></div><div>Another option is to manually force a sanlock renewal on that host and check what happens, something like:</div><div><span class="gmail-il" style="font-size:12.8px;background-color:rgb(255,255,255)">sanlock</span><span style="font-size:12.8px"> </span><span class="gmail-il" style="font-size:12.8px;background-color:rgb(255,255,255)">client</span><span style="font-size:12.8px"> renewal -s </span>7c8deaa8-be02-4aaf-b9b4-<wbr>ddc8da99ad96:3:/rhev/data-<wbr>center/mnt/glusterSD/<wbr>glusterfs.rxmgmt.databay.de:_<wbr>engine/7c8deaa8-be02-4aaf-<wbr>b9b4-ddc8da99ad96/dom_md/ids:0<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="gmail-h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
    <p>Bye<br>
    </p><div><div class="gmail-m_132298329271851776gmail-h5">
    <br>
    <div class="gmail-m_132298329271851776gmail-m_-1867176741777387997moz-cite-prefix">Am 03.02.2017 um 16:12 schrieb Simone
      Tiraboschi:<br>
    </div>
    <blockquote type="cite">
      <div dir="ltr">The hosted-engine storage domain is mounted for
        sure,
        <div>but the issue is here:<br>
          <div>Exception: Failed to start monitoring domain
            (sd_uuid=7c8deaa8-be02-4aaf-b9<wbr>b4-ddc8da99ad96, host_id=3):
            timeout during domain acquisition<br>
          </div>
        </div>
        <div><br>
        </div>
        <div>The point is that in VDSM logs I see just something like:</div>
        <div>
          <div>2017-02-02 21:05:22,283 INFO  (jsonrpc/1) [dispatcher]
            Run and protect: repoStats(options=None) (logUtils:49)</div>
          <div>2017-02-02 21:05:22,285 INFO  (jsonrpc/1) [dispatcher]
            Run and protect: repoStats, Return response:
            {u&#39;a7fbaaad-7043-4391-9523-3be<wbr>dcdc4fb0d&#39;: {&#39;code&#39;: 0,
            &#39;actual&#39;: True, &#39;version&#39;: 0, &#39;acquired&#39;: True, &#39;delay&#39;:
            &#39;0.000748727&#39;, &#39;lastCheck&#39;: &#39;0.1&#39;, &#39;valid&#39;: True},
            u&#39;2b2a44fc-f2bd-47cd-b7af-00be<wbr>59e30a35&#39;: {&#39;code&#39;: 0,
            &#39;actual&#39;: True, &#39;version&#39;: 0, &#39;acquired&#39;: True, &#39;delay&#39;:
            &#39;0.00082529&#39;, &#39;lastCheck&#39;: &#39;0.1&#39;, &#39;valid&#39;: True},
            u&#39;5d99af76-33b5-47d8-99da-1f32<wbr>413c7bb0&#39;: {&#39;code&#39;: 0,
            &#39;actual&#39;: True, &#39;version&#39;: 4, &#39;acquired&#39;: True, &#39;delay&#39;:
            &#39;0.000349356&#39;, &#39;lastCheck&#39;: &#39;5.3&#39;, &#39;valid&#39;: True},
            u&#39;7c8deaa8-be02-4aaf-b9b4-ddc8<wbr>da99ad96&#39;: {&#39;code&#39;: 0,
            &#39;actual&#39;: True, &#39;version&#39;: 4, &#39;acquired&#39;: False, &#39;delay&#39;:
            &#39;0.000377052&#39;, &#39;lastCheck&#39;: &#39;0.6&#39;, &#39;valid&#39;: True}}
            (logUtils:52)</div>
        </div>
        <div><br>
        </div>
        <div>Where the other storage domains have &#39;acquired&#39;: True whil
          it&#39;s always &#39;acquired&#39;: False for the hosted-engine storage
          domain.</div>
        <div><br>
        </div>
        <div>Could you please share your /var/log/sanlock.log from the
          same host and the output of </div>
        <div> sanlock client status</div>
        <div>?</div>
        <div><br>
        </div>
        <div><br>
        </div>
        <div><br>
        </div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Fri, Feb 3, 2017 at 3:52 PM, Ralf
          Schenk <span dir="ltr">&lt;<a href="mailto:rs@databay.de" target="_blank">rs@databay.de</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
            <div bgcolor="#FFFFFF">
              <p>Hello,</p>
              <p>I also put host in Maintenance and restarted vdsm while
                ovirt-ha-agent is running. I can mount the gluster
                Volume &quot;engine&quot; manually in the host.<br>
              </p>
              <p>I get this repeatedly in /var/log/vdsm.log:</p>
              <p><tt>2017-02-03 15:29:28,891 INFO  (MainThread) [vds]
                  Exiting (vdsm:167)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:30,974 INFO  (MainThread)
                  [vds] (PID: 11456) I am the actual vdsm
                  4.19.4-1.el7.centos microcloud27
                  (3.10.0-514.6.1.el7.x86_64) (vdsm:145)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:30,974 INFO  (MainThread)
                  [vds] VDSM will run with cpu affinity: frozenset([1])
                  (vdsm:251)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,013 INFO  (MainThread)
                  [storage.check] Starting check service (check:91)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,017 INFO  (MainThread)
                  [storage.Dispatcher] Starting StorageDispatcher...
                  (dispatcher:47)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,017 INFO  (check/loop)
                  [storage.asyncevent] Starting &lt;EventLoop
                  running=True closed=False at 0x37480464&gt;
                  (asyncevent:122)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,156 INFO  (MainThread)
                  [dispatcher] Run and protect:
                  registerDomainStateChangeCallb<wbr>ack(callbackFunc=&lt;<a href="http://functools.pa" target="_blank">functools.pa</a><wbr>rtial
                  object at 0x2881fc8&gt;) (logUtils:49)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,156 INFO  (MainThread)
                  [dispatcher] Run and protect:
                  registerDomainStateChangeCallb<wbr>ack, Return
                  response: None (logUtils:52)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,160 INFO  (MainThread)
                  [MOM] Preparing MOM interface (momIF:49)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,161 INFO  (MainThread)
                  [MOM] Using named unix socket
                  /var/run/vdsm/mom-vdsm.sock (momIF:58)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,162 INFO  (MainThread)
                  [root] Unregistering all secrets (secret:91)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,164 INFO  (MainThread)
                  [vds] Setting channels&#39; timeout to 30 seconds.
                  (vmchannels:223)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,165 INFO  (MainThread)
                  [vds.MultiProtocolAcceptor] Listening at :::54321
                  (protocoldetector:185)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,354 INFO  (vmrecovery)
                  [vds] recovery: completed in 0s (clientIF:495)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,371 INFO  (BindingXMLRPC)
                  [vds] XMLRPC server running (bindingxmlrpc:63)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,471 INFO  (periodic/1)
                  [dispatcher] Run and protect: repoStats(options=None)
                  (logUtils:49)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,472 INFO  (periodic/1)
                  [dispatcher] Run and protect: repoStats, Return
                  response: {} (logUtils:52)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,472 WARN  (periodic/1)
                  [MOM] MOM not available. (momIF:116)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,473 WARN  (periodic/1)
                  [MOM] MOM not available, KSM stats will be missing.
                  (momIF:79)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:31,474 ERROR (periodic/1)
                  [root] failed to retrieve Hosted Engine HA info
                  (api:252)</tt><span><tt><br>
                  </tt><tt>Traceback (most recent call last):</tt><tt><br>
                  </tt></span><tt>  File &quot;/usr/lib/python2.7/site-packa<wbr>ges/vdsm/host/api.py&quot;,
                  line 231, in _getHaInfo</tt><tt><br>
                </tt><tt>    stats = instance.get_all_stats()</tt><tt><br>
                </tt><tt>  File
                  &quot;/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/cli<wbr>ent/client.py&quot;,
                  line 103, in get_all_stats</tt><tt><br>
                </tt><tt>    self._configure_broker_conn(br<wbr>oker)</tt><tt><br>
                </tt><tt>  File
                  &quot;/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/cli<wbr>ent/client.py&quot;,
                  line 180, in _configure_broker_conn</tt><tt><br>
                </tt><tt>    dom_type=dom_type)</tt><tt><br>
                </tt><tt>  File
                  &quot;/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/lib<wbr>/brokerlink.py&quot;,
                  line 177, in set_storage_domain</tt><tt><br>
                </tt><tt>    .format(sd_type, options, e))</tt><tt><br>
                </tt><tt>RequestError: Failed to set storage domain
                  FilesystemBackend, options {&#39;dom_type&#39;: &#39;glusterfs&#39;,
                  &#39;sd_uuid&#39;: &#39;7c8deaa8-be02-4aaf-b9b4-ddc8d<wbr>a99ad96&#39;}:
                  Request failed: &lt;class &#39;ovirt_hos</tt><tt><br>
                </tt><tt>ted_engine_ha.lib.storage_back<wbr>ends.BackendFailureException&#39;&gt;</tt><tt><br>
                </tt><tt>2017-02-03 15:29:35,920 INFO  (Reactor thread)
                  [ProtocolDetector.AcceptorImpl<wbr>] Accepted
                  connection from ::1:49506 (protocoldetector:72)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:35,929 INFO  (Reactor thread)
                  [ProtocolDetector.Detector] Detected protocol stomp
                  from ::1:49506 (protocoldetector:127)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:35,930 INFO  (Reactor thread)
                  [Broker.StompAdapter] Processing CONNECT request
                  (stompreactor:102)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:35,930 INFO  (JsonRpc
                  (StompReactor)) [Broker.StompAdapter] Subscribe
                  command received (stompreactor:129)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:36,067 INFO  (jsonrpc/0)
                  [jsonrpc.JsonRpcServer] RPC call Host.ping succeeded
                  in 0.00 seconds (__init__:515)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:36,071 INFO  (jsonrpc/1)
                  [throttled] Current getAllVmStats: {}
                  (throttledlog:105)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:36,071 INFO  (jsonrpc/1)
                  [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats
                  succeeded in 0.00 seconds (__init__:515)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:46,435 INFO  (periodic/0)
                  [dispatcher] Run and protect: repoStats(options=None)
                  (logUtils:49)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:46,435 INFO  (periodic/0)
                  [dispatcher] Run and protect: repoStats, Return
                  response: {} (logUtils:52)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:46,439 ERROR (periodic/0)
                  [root] failed to retrieve Hosted Engine HA info
                  (api:252)</tt><span><tt><br>
                  </tt><tt>Traceback (most recent call last):</tt><tt><br>
                  </tt></span><tt>  File &quot;/usr/lib/python2.7/site-packa<wbr>ges/vdsm/host/api.py&quot;,
                  line 231, in _getHaInfo</tt><tt><br>
                </tt><tt>    stats = instance.get_all_stats()</tt><tt><br>
                </tt><tt>  File
                  &quot;/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/cli<wbr>ent/client.py&quot;,
                  line 103, in get_all_stats</tt><tt><br>
                </tt><tt>    self._configure_broker_conn(br<wbr>oker)</tt><tt><br>
                </tt><tt>  File
                  &quot;/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/cli<wbr>ent/client.py&quot;,
                  line 180, in _configure_broker_conn</tt><tt><br>
                </tt><tt>    dom_type=dom_type)</tt><tt><br>
                </tt><tt>  File
                  &quot;/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/lib<wbr>/brokerlink.py&quot;,
                  line 177, in set_storage_domain</tt><tt><br>
                </tt><tt>    .format(sd_type, options, e))</tt><tt><br>
                </tt><tt>RequestError: Failed to set storage domain
                  FilesystemBackend, options {&#39;dom_type&#39;: &#39;glusterfs&#39;,
                  &#39;sd_uuid&#39;: &#39;7c8deaa8-be02-4aaf-b9b4-ddc8d<wbr>a99ad96&#39;}:
                  Request failed: &lt;class &#39;ovirt_hos</tt><tt><br>
                </tt><tt>ted_engine_ha.lib.storage_back<wbr>ends.BackendFailureException&#39;&gt;</tt><tt><br>
                </tt><tt>2017-02-03 15:29:51,095 INFO  (jsonrpc/2)
                  [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats
                  succeeded in 0.00 seconds (__init__:515)</tt><tt><br>
                </tt><tt>2017-02-03 15:29:51,219 INFO  (jsonrpc/3)
                  [jsonrpc.JsonRpcServer] RPC call Host.setKsmTune
                  succeeded in 0.00 seconds (__init__:515)</tt><tt><br>
                </tt><tt>2017-02-03 15:30:01,444 INFO  (periodic/1)
                  [dispatcher] Run and protect: repoStats(options=None)
                  (logUtils:49)</tt><tt><br>
                </tt><tt>2017-02-03 15:30:01,444 INFO  (periodic/1)
                  [dispatcher] Run and protect: repoStats, Return
                  response: {} (logUtils:52)</tt><tt><br>
                </tt><tt>2017-02-03 15:30:01,448 ERROR (periodic/1)
                  [root] failed to retrieve Hosted Engine HA info
                  (api:252)</tt><br>
                <br>
              </p>
              <span>
                <p><br>
                </p>
                <br>
                <div class="gmail-m_132298329271851776gmail-m_-1867176741777387997m_2083194111166540231moz-cite-prefix">Am
                  03.02.2017 um 13:39 schrieb Simone Tiraboschi:<br>
                </div>
              </span>
              <div>
                <div class="gmail-m_132298329271851776gmail-m_-1867176741777387997h5">
                  <blockquote type="cite">
                    <div dir="ltr">I see there an ERROR
                      on stopMonitoringDomain but I cannot see the
                      correspondent  startMonitoringDomain; could you
                      please look for it?</div>
                    <div class="gmail_extra"><br>
                      <div class="gmail_quote">On Fri, Feb 3, 2017 at
                        1:16 PM, Ralf Schenk <span dir="ltr">&lt;<a href="mailto:rs@databay.de" target="_blank">rs@databay.de</a>&gt;</span>
                        wrote:<br>
                        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                          <div bgcolor="#FFFFFF">
                            <p>Hello,</p>
                            <p>attached is my vdsm.log from the host
                              with hosted-engine-ha around the
                              time-frame of agent timeout that is not
                              working anymore for engine (it works in
                              Ovirt and is active). It simply isn&#39;t
                              working for engine-ha anymore after
                              Update.</p>
                            <p>At 2017-02-02 19:25:34,248 you&#39;ll find an
                              error corresponoding to agent timeout
                              error.</p>
                            <p>Bye<br>
                            </p>
                            <div>
                              <div class="gmail-m_132298329271851776gmail-m_-1867176741777387997m_2083194111166540231h5">
                                <p><br>
                                </p>
                                <br>
                                <div class="gmail-m_132298329271851776gmail-m_-1867176741777387997m_2083194111166540231m_-5371711976759655950moz-cite-prefix">Am
                                  03.02.2017 um 11:28 schrieb Simone
                                  Tiraboschi:<br>
                                </div>
                                <blockquote type="cite">
                                  <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                    <div dir="ltr">
                                      <div class="gmail_extra">
                                        <div class="gmail_quote"><span>
                                            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                              <div bgcolor="#FFFFFF">
                                                <p>3. Three of my hosts
                                                  have the hosted engine
                                                  deployed for ha. First
                                                  all three where marked
                                                  by a crown (running
                                                  was gold and others
                                                  where silver). After
                                                  upgrading the 3 Host
                                                  deployed hosted engine
                                                  ha is not active
                                                  anymore.</p>
                                                <p>I can&#39;t get this host
                                                  back with working
                                                  ovirt-ha-agent/broker.
                                                  I already rebooted,
                                                  manually restarted the
                                                  services but It isn&#39;t
                                                  able to get cluster
                                                  state according to <br>
                                                  &quot;hosted-engine
                                                  --vm-status&quot;. The
                                                  other hosts state the
                                                  host status as
                                                  &quot;unknown stale-data&quot;</p>
                                                <p>I already shut down
                                                  all agents on all
                                                  hosts and issued a
                                                  &quot;hosted-engine
                                                  --reinitialize-lockspace&quot;
                                                  but that didn&#39;t help.<br>
                                                </p>
                                                <p>Agents stops working
                                                  after a timeout-error
                                                  according to log:</p>
                                                <p><tt>MainThread::INFO::2017-02-02
19:24:52,040::hosted_engine::8<wbr>41::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_get_domain_monitor_status)
                                                    VDSM domain monitor
                                                    status: PENDING</tt><tt><br>
                                                  </tt><tt>MainThread::INFO::2017-02-02
19:24:59,185::hosted_engine::8<wbr>41::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_get_domain_monitor_status)
                                                    VDSM domain monitor
                                                    status: PENDING</tt><tt><br>
                                                  </tt><tt>MainThread::INFO::2017-02-02
19:25:06,333::hosted_engine::8<wbr>41::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_get_domain_monitor_status)
                                                    VDSM domain monitor
                                                    status: PENDING</tt><tt><br>
                                                  </tt><tt>MainThread::INFO::2017-02-02
19:25:13,554::hosted_engine::8<wbr>41::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_get_domain_monitor_status)
                                                    VDSM domain monitor
                                                    status: PENDING</tt><tt><br>
                                                  </tt><tt>MainThread::INFO::2017-02-02
19:25:20,710::hosted_engine::8<wbr>41::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_get_domain_monitor_status)
                                                    VDSM domain monitor
                                                    status: PENDING</tt><tt><br>
                                                  </tt><tt>MainThread::INFO::2017-02-02
19:25:27,865::hosted_engine::8<wbr>41::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_get_domain_monitor_status)
                                                    VDSM domain monitor
                                                    status: PENDING</tt><tt><br>
                                                  </tt><tt>MainThread::ERROR::2017-02-02
19:25:27,866::hosted_engine::8<wbr>15::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_initialize_domain_monitor)
                                                    Failed to start
                                                    monitoring domain
                                                    (sd_uuid=7c8deaa8-be02-4aaf-b9<wbr>b4-ddc8da99ad96,
                                                    host_id=3): timeout
                                                    during domain
                                                    acquisition</tt><tt><br>
                                                  </tt><tt>MainThread::WARNING::2017-02-0<wbr>2
19:25:27,866::hosted_engine::4<wbr>69::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(start_monitoring)
                                                    Error while
                                                    monitoring engine:
                                                    Failed to start
                                                    monitoring domain
                                                    (sd_uuid=7c8deaa8-be02-4aaf-b9<wbr>b4-ddc8da99ad96,
                                                    host_id=3): timeout
                                                    during domain
                                                    acquisition</tt><tt><br>
                                                  </tt><tt>MainThread::WARNING::2017-02-0<wbr>2
19:25:27,866::hosted_engine::4<wbr>72::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(start_monitoring)
                                                    Unexpected error</tt><tt><br>
                                                  </tt><tt>Traceback
                                                    (most recent call
                                                    last):</tt><tt><br>
                                                  </tt><tt>  File
                                                    &quot;/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/age<wbr>nt/hosted_engine.py&quot;,
                                                    line 443, in
                                                    start_monitoring</tt><tt><br>
                                                  </tt><tt>   
                                                    self._initialize_domain_monito<wbr>r()</tt><tt><br>
                                                  </tt><tt>  File
                                                    &quot;/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/age<wbr>nt/hosted_engine.py&quot;,
                                                    line 816, in
                                                    _initialize_domain_monitor</tt><tt><br>
                                                  </tt><tt>    raise
                                                    Exception(msg)</tt><tt><br>
                                                  </tt><tt>Exception:
                                                    Failed to start
                                                    monitoring domain
                                                    (sd_uuid=7c8deaa8-be02-4aaf-b9<wbr>b4-ddc8da99ad96,
                                                    host_id=3): timeout
                                                    during domain
                                                    acquisition</tt><tt><br>
                                                  </tt><tt>MainThread::ERROR::2017-02-02
19:25:27,866::hosted_engine::4<wbr>85::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(start_monitoring)
                                                    Shutting down the
                                                    agent because of 3
                                                    failures in a row!</tt><tt><br>
                                                  </tt><tt>MainThread::INFO::2017-02-02
19:25:32,087::hosted_engine::8<wbr>41::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_get_domain_monitor_status)
                                                    VDSM domain monitor
                                                    status: PENDING</tt><tt><br>
                                                  </tt><tt>MainThread::INFO::2017-02-02
19:25:34,250::hosted_engine::7<wbr>69::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_stop_domain_monitor)
                                                    Failed to stop
                                                    monitoring domain
                                                    (sd_uuid=7c8deaa8-be02-4aaf-b9<wbr>b4-ddc8da99ad96):
                                                    Storage domain is
                                                    member of pool:
                                                    u&#39;domain=7c8deaa8-be02-4aaf-b9<wbr>b4-ddc8da99ad96&#39;</tt><tt><br>
                                                  </tt><tt>MainThread::INFO::2017-02-02
19:25:34,254::agent::143::ovir<wbr>t_hosted_engine_ha.agent.agent<wbr>.Agent::(run)
                                                    Agent shutting down</tt></p>
                                              </div>
                                            </blockquote>
                                          </span>
                                          <div>Simone, Martin, can you
                                            please follow up on this?</div>
                                        </div>
                                      </div>
                                    </div>
                                  </blockquote>
                                  <div><br>
                                  </div>
                                  <div>Ralph, could you please attach
                                    vdsm logs from on of your hosts for
                                    the relevant time frame?</div>
                                </blockquote>
                                <br>
                              </div>
                            </div>
                            <span>
                              <div class="gmail-m_132298329271851776gmail-m_-1867176741777387997m_2083194111166540231m_-5371711976759655950moz-signature">--
                                <br>
                                <p> </p>
                                <table border="0" cellpadding="0" cellspacing="0">
                                  <tbody>
                                    <tr>
                                      <td colspan="3"><img src="cid:part3.23D0C025.3D38CC61@databay.de" height="30" border="0" width="151"></td>
                                    </tr>
                                    <tr>
                                      <td valign="top"> <font face="Verdana, Arial,
                                          sans-serif" size="-1"><br>
                                          <b>Ralf Schenk</b><br>
                                          fon <a href="tel:+49%202405%20408370" value="+492405408370" target="_blank">+49
                                            (0) 24 05 / 40 83 70</a><br>
                                          fax <a href="tel:+49%202405%204083759" value="+4924054083759" target="_blank">+49
                                            (0) 24 05 / 40 83 759</a><br>
                                          mail <a href="mailto:rs@databay.de" target="_blank"><font color="#FF0000"><b>rs@databay.de</b></font></a><br>
                                        </font> </td>
                                      <td width="30"> </td>
                                      <td valign="top"> <font face="Verdana, Arial,
                                          sans-serif" size="-1"><br>
                                          <b>Databay AG</b><br>
                                          Jens-Otto-Krag-Straße 11<br>
                                          D-52146 Würselen<br>
                                          <a href="http://www.databay.de" target="_blank"><font color="#FF0000"><b>www.databay.de</b></font></a>
                                        </font> </td>
                                    </tr>
                                    <tr>
                                      <td colspan="3" valign="top"> <font face="Verdana, Arial,
                                          sans-serif" size="1"><br>
                                          Sitz/Amtsgericht Aachen •
                                          HRB:8437 • USt-IdNr.: DE
                                          210844202<br>
                                          Vorstand: Ralf Schenk,
                                          Dipl.-Ing. Jens Conze, Aresch
                                          Yavari, Dipl.-Kfm. Philipp
                                          Hermanns<br>
                                          Aufsichtsratsvorsitzender:
                                          Wilhelm Dohmen </font> </td>
                                    </tr>
                                  </tbody>
                                </table>
                                <hr color="#000000" noshade size="1" width="100%"> </div>
                            </span></div>
                        </blockquote>
                      </div>
                      <br>
                    </div>
                  </blockquote>
                  <br>
                  <div class="gmail-m_132298329271851776gmail-m_-1867176741777387997m_2083194111166540231moz-signature">-- <br>
                    <p> </p>
                    <table border="0" cellpadding="0" cellspacing="0">
                      <tbody>
                        <tr>
                          <td colspan="3"><img src="cid:part8.BEE7FDC9.08FC5110@databay.de" height="30" border="0" width="151"></td>
                        </tr>
                        <tr>
                          <td valign="top"> <font face="Verdana, Arial,
                              sans-serif" size="-1"><br>
                              <b>Ralf Schenk</b><br>
                              fon <a href="tel:+49%202405%20408370" value="+492405408370" target="_blank">+49
                                (0) 24 05 / 40 83 70</a><br>
                              fax <a href="tel:+49%202405%204083759" value="+4924054083759" target="_blank">+49
                                (0) 24 05 / 40 83 759</a><br>
                              mail <a href="mailto:rs@databay.de" target="_blank"><font color="#FF0000"><b>rs@databay.de</b></font></a><br>
                            </font> </td>
                          <td width="30"> </td>
                          <td valign="top"> <font face="Verdana, Arial,
                              sans-serif" size="-1"><br>
                              <b>Databay AG</b><br>
                              Jens-Otto-Krag-Straße 11<br>
                              D-52146 Würselen<br>
                              <a href="http://www.databay.de" target="_blank"><font color="#FF0000"><b>www.databay.de</b></font></a>
                            </font> </td>
                        </tr>
                        <tr>
                          <td colspan="3" valign="top"> <font face="Verdana, Arial, sans-serif" size="1"><br>
                              Sitz/Amtsgericht Aachen • HRB:8437 •
                              USt-IdNr.: DE 210844202<br>
                              Vorstand: Ralf Schenk, Dipl.-Ing. Jens
                              Conze, Aresch Yavari, Dipl.-Kfm. Philipp
                              Hermanns<br>
                              Aufsichtsratsvorsitzender: Wilhelm Dohmen
                            </font> </td>
                        </tr>
                      </tbody>
                    </table>
                    <hr color="#000000" noshade size="1" width="100%"> </div>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
    <div class="gmail-m_132298329271851776gmail-m_-1867176741777387997moz-signature">-- <br>
      <p>
      </p>
      <table border="0" cellpadding="0" cellspacing="0">
        <tbody>
          <tr>
            <td colspan="3"><img src="cid:part13.E21ABAF3.DBD75827@databay.de" height="30" border="0" width="151"></td>
          </tr>
          <tr>
            <td valign="top"> <font face="Verdana, Arial, sans-serif" size="-1"><br>
                <b>Ralf Schenk</b><br>
                fon <a href="tel:+49%202405%20408370" value="+492405408370" target="_blank">+49 (0) 24 05 / 40 83 70</a><br>
                fax <a href="tel:+49%202405%204083759" value="+4924054083759" target="_blank">+49 (0) 24 05 / 40 83 759</a><br>
                mail <a href="mailto:rs@databay.de" target="_blank"><font color="#FF0000"><b>rs@databay.de</b></font></a><br>
              </font> </td>
            <td width="30"> </td>
            <td valign="top"> <font face="Verdana, Arial, sans-serif" size="-1"><br>
                <b>Databay AG</b><br>
                Jens-Otto-Krag-Straße 11<br>
                D-52146 Würselen<br>
                <a href="http://www.databay.de" target="_blank"><font color="#FF0000"><b>www.databay.de</b></font></a>
              </font> </td>
          </tr>
          <tr>
            <td colspan="3" valign="top"> <font face="Verdana, Arial,
                sans-serif" size="1"><br>
                Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE
                210844202<br>
                Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch
                Yavari, Dipl.-Kfm. Philipp Hermanns<br>
                Aufsichtsratsvorsitzender: Wilhelm Dohmen </font> </td>
          </tr>
        </tbody>
      </table>
      <hr color="#000000" noshade size="1" width="100%">
    </div>
  </div></div></div>

</blockquote></div></div></div><br></div></div>
</blockquote></div><br></div></div>