Re: [ovirt-users] vm.conf on one of the node is missing

Thursday, 24 November 2016

This is a multi-part message in MIME format.
--------------08CE69774B978D28F256FCDD
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

On 11/24/2016 07:47 PM, Simone Tiraboschi wrote:
...

 On Thu, Nov 24, 2016 at 3:06 PM, knarra <knarra(a)redhat.com 
 <mailto:knarra@redhat.com>> wrote:

     On 11/24/2016 07:27 PM, Simone Tiraboschi wrote:
>
>
>     On Thu, Nov 24, 2016 at 2:39 PM, knarra <knarra(a)redhat.com
>     <mailto:knarra@redhat.com>> wrote:
>
>         On 11/24/2016 06:56 PM, Simone Tiraboschi wrote:
>>
>>
>>         On Thu, Nov 24, 2016 at 2:08 PM, knarra <knarra(a)redhat.com
>>         <mailto:knarra@redhat.com>> wrote:
>>
>>             On 11/24/2016 06:15 PM, Simone Tiraboschi wrote:
>>>
>>>
>>>             On Thu, Nov 24, 2016 at 1:26 PM, knarra
>>>             <knarra(a)redhat.com <mailto:knarra@redhat.com>>
wrote:
>>>
>>>                 Hi,
>>>
>>>                     I have three nodes with glusterfs as storage
>>>                 domain. For some reason i see that vm.conf from
>>>                 /var/run/ovirt-hosted-engine-ha is missing and due
>>>                 to this on one of my host i see that Hosted Engine
>>>                 HA : Not Active. Once i copy the file from some
>>>                 other node and restart ovirt-ha-broker and
>>>                 ovirt-ha-agent services everything works fine. But
>>>                 then this happens again. Can some please help me
>>>                 identify why this happens. Below is the log i see
>>>                 in ovirt-ha-agent.logs.
>>>
>>>
>>>                 https://paste.fedoraproject.org/489120/79990345/
>>>                 <https://paste.fedoraproject.org/489120/79990345/>
>>>
>>>
>>>             Once the engine correctly imported the hosted-engine
>>>             storage domain, a couple of OVF_STORE volumes will
>>>             appear there.
>>>             Every modification to the engine VM configuration will
>>>             be written by the engine into that OVF_STORE, so all
>>>             the ovirt-ha-agent running on the hosted-engine hosts
>>>             will be able to re-start the engine VM with a coherent
>>>             configuration.
>>>
>>>             Till the engine imports the hosted-engine storage
>>>             domain, ovirt-ha-agent will fall back to the initial
>>>             vm.conf.
>>>
>>>             In you case the OVF_STORE volume is there,
>>>             but the agent fails extracting the engine VM configuration:
>>>             MainThread::INFO::2016-11-24
>>>            
17:55:04,914::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
>>>             Extracting Engine VM OVF from the OVF_STORE
>>>             MainThread::INFO::2016-11-24
>>>            
17:55:04,919::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
>>>             OVF_STORE volume path:
>>>            
/rhev/data-center/mnt/glusterSD/10.70.36.79:_engine/27f054c3-c245-4039-b42a-c28b37043016/images/fdf49778-9a06-49c6-bf7a-a0f12425911c/8c954add-6bcf-47f8-ac2e-4c85fc3f8699
>>>             MainThread::ERROR::2016-11-24
>>>            
17:55:04,928::ovf_store::124::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
>>>             Unable to extract HEVM OVF
>>>
>>>             So it tries to rollback to the initial vm.conf, but
>>>             also that one seams to miss some values and so the
>>>             agent is failing:
>>>             MainThread::ERROR::2016-11-24
>>>            
17:55:04,974::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>>>             Error: ''Configuration value not found:
>>>             file=/var/run/ovirt-hosted-engine-ha/vm.conf,
>>>             key=memSize'' - trying to restart agent
>>>
>>>             Both of the issue seams storage related, could yuo
>>>             please share your gluster logs?
>>>
>>>
>>>                 Thanks
>>>
>>>                 kasturi
>>>
>>>
>>             Hi Simone,
>>
>>                 Below [1] is the link for the sosreports on the
>>             first two hosts. The third host has some issue. Once it
>>             is up will give the sosreport from there as well.
>>
>>
>>         And the host where you see the initial issue was the third one?
>         It is on the first host.
>
>
>     It seams that host1 is failing reading from the the hosted-engine
>     storage domain:
>
>     [2016-11-24 12:33:43.678467] W [MSGID: 114031]
>     [client-rpc-fops.c:2938:client3_3_lookup_cbk] 0-engine-client-2:
>     remote operation failed. Path: /
>     (00000000-0000-0000-0000-000000000001) [Transport endpoint is not
>     connected]
>     [2016-11-24 12:33:43.678747] E
>     [rpc-clnt.c:365:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f077eba1642]
>     (-->
>     /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f077e96775e]
>     (-->
>     /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f077e96786e]
>     (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f077e968fc4]
>     (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f077e9698a0]
>     ))))) 0-engine-client-2: forced unwinding frame type(GlusterFS
>     3.3) op(LOOKUP(27)) called at 2016-11-24 12:33:07.495178
>     (xid=0x82a1c)
>     [2016-11-24 12:33:43.678982] E
>     [rpc-clnt.c:365:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f077eba1642]
>     (-->
>     /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f077e96775e]
>     (-->
>     /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f077e96786e]
>     (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f077e968fc4]
>     (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f077e9698a0]
>     ))))) 0-engine-client-2: forced unwinding frame type(GlusterFS
>     3.3) op(LOOKUP(27)) called at 2016-11-24 12:33:08.770637
>     (xid=0x82a1d)
>     [2016-11-24 12:33:43.679001] W [MSGID: 114031]
>     [client-rpc-fops.c:2938:client3_3_lookup_cbk] 0-engine-client-2:
>     remote operation failed. Path:
>    
/27f054c3-c245-4039-b42a-c28b37043016/images/39960f40-4aae-4714-ba73-1637785fae7c/38fa3519-f21e-4671-8c69-d1497ff8a490
>     (1090c25b-9c90-434e-a133-faf9647cc992) [Transport endpoint is not
>     connected]
>     [2016-11-24 12:33:43.679303] E
>     [rpc-clnt.c:365:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f077eba1642]
>     (-->
>     /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f077e96775e]
>     (-->
>     /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f077e96786e]
>     (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f077e968fc4]
>     (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f077e9698a0]
>     ))))) 0-engine-client-2: forced unwinding frame type(GlusterFS
>     3.3) op(LOOKUP(27)) called at 2016-11-24 12:33:11.096856
>     (xid=0x82a1e)
>     [2016-11-24 12:33:43.679596] E
>     [rpc-clnt.c:365:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f077eba1642]
>     (-->
>     /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f077e96775e]
>     (-->
>     /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f077e96786e]
>     (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f077e968fc4]
>     (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f077e9698a0]
>     ))))) 0-engine-client-2: forced unwinding frame type(GF-DUMP)
>     op(NULL(2)) called at 2016-11-24 12:33:13.673743 (xid=0x82a1f)
>     [2016-11-24 12:33:43.682310] I
>     [socket.c:3401:socket_submit_request] 0-engine-client-2: not
>     connected (priv->connected = 0)
>     [2016-11-24 12:33:43.682328] W [rpc-clnt.c:1640:rpc_clnt_submit]
>     0-engine-client-2: failed to submit rpc-request (XID: 0x82a20
>     Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport
>     (engine-client-2)
>     [2016-11-24 12:33:43.682391] W [rpc-clnt.c:1640:rpc_clnt_submit]
>     0-engine-client-2: failed to submit rpc-request (XID: 0x82a21
>     Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport
>     (engine-client-2)
>     [2016-11-24 12:33:43.682441] W [rpc-clnt.c:1640:rpc_clnt_submit]
>     0-engine-client-2: failed to submit rpc-request (XID: 0x82a22
>     Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport
>     (engine-client-2)
>     [2016-11-24 12:33:43.682441] W [MSGID: 114031]
>     [client-rpc-fops.c:2938:client3_3_lookup_cbk] 0-engine-client-2:
>     remote operation failed. Path:
>     /27f054c3-c245-4039-b42a-c28b37043016
>     (a64398f5-3fa3-48fe-9d40-d3860876cc2c) [Transport endpoint is not
>     connected]
>     [2016-11-24 12:33:43.682492] W
>     [rpc-clnt-ping.c:203:rpc_clnt_ping_cbk] 0-engine-client-2: socket
>     disconnected
>     [2016-11-24 12:33:43.682536] I [MSGID: 114018]
>     [client.c:2280:client_rpc_notify] 0-engine-client-2: disconnected
>     from engine-client-2. Client process will keep trying to connect
>     to glusterd until brick's port is available
>     [2016-11-24 12:33:43.682562] W [rpc-clnt.c:1640:rpc_clnt_submit]
>     0-engine-client-2: failed to submit rpc-request (XID: 0x82a23
>     Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport
>     (engine-client-2)
>     The message "W [MSGID: 114031]
>     [client-rpc-fops.c:2938:client3_3_lookup_cbk] 0-engine-client-2:
>     remote operation failed. Path:
>     /27f054c3-c245-4039-b42a-c28b37043016
>     (a64398f5-3fa3-48fe-9d40-d3860876cc2c) [Transport endpoint is not
>     connected]" repeated 2 times between [2016-11-24 12:33:43.682441]
>     and [2016-11-24 12:33:43.682599]
>     [2016-11-24 12:33:43.688324] W [MSGID: 114031]
>     [client-rpc-fops.c:2938:client3_3_lookup_cbk] 0-engine-client-2:
>     remote operation failed. Path: (null)
>     (00000000-0000-0000-0000-000000000000) [Transport endpoint is not
>     connected]
>
>     Before that there was a lot of self-healing activities.
>
     simone, these logs indicates that first host is not able to
     connect to the brick in the third host since it was powered down. 
     It reads remote operation failed on engine_client_2 .

 Can you please share the output of
  source /etc/ovirt-hosted-engine/hosted-engine.conf
  find /rhev/data-center/ -path 
 "*/${sdUUID}/images/${conf_image_UUID}/${conf_volume_UUID}" -type f 
 -exec sh -c 'sudo -u vdsm dd if=$1 2>/dev/null | tar -xOvf - vm.conf 
  2>/dev/null' {} {} \;
 executed on your first host? output for the above command 
https://paste.fedoraproject.org/489159/97176147/

Simone, i ran the following steps and i see that vm.conf file is missing.

1) Installed three nodes with glusterfs as storage domains.

2) Configured power management on all the nodes and ran the command 
'poweroff -f' on one of the node.

3) Engine tried to fence the host and it logged the message 'Host is 
rebooting' but this never came up.

4) After some time i manually powered on the host and once the host is 
up i see that Hosted Engine HA : Not active.

5) I see the following error message in broker.log and vm.conf file is 
missing.

Thread-1::ERROR::2016-11-24 
19:32:11,651::listener::192::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)

Error handling request, data: 'set-storage-domain
  FilesystemBackend dom_type=glusterfs 
sd_uuid=27f054c3-c245-4039-b42a-c28b37043016'
Traceback (most recent call last):
   File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
line 166, in handle
     data)
   File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
line 299, in _dispatch
     .set_storage_domain(client, sd_type, **options)
   File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",

line 66, in set_storage_domain
     self._backends[client].connect()
   File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",

line 461, in connect
     self._dom_type)
   File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",

line 108, in get_domain_path
     " in {1}".format(sd_uuid, parent))
BackendFailureException: path to storage domain 
27f054c3-c245-4039-b42a-c28b37043016 not found in 
/rhev/data-center/mnt/glusterSD

Thanks
kasturi

...

>
>>             [1]
>>             http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/HC/vm_conf/
>>            
<http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/HC/vm_conf/>
>>
>>             Thanks
>>
>>             kasturi
>>
>>
>
>

--------------08CE69774B978D28F256FCDD
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">On 11/24/2016 07:47 PM, Simone
      Tiraboschi wrote:<br>
    </div>
    <blockquote
cite="mid:CAN8-ONqt1w1uF-GqQ-8Hr2cN+E61AatDRpmnU=KB_LW8b9OBig@mail.gmail.com"
      type="cite">
      <div dir="ltr"><br>
        <div class="gmail_extra"><br>
          <div class="gmail_quote">On Thu, Nov 24, 2016 at 3:06 PM,
            knarra <span dir="ltr">&lt;<a
moz-do-not-send="true"
                href="mailto:knarra@redhat.com"
target=&quot;_blank&quot;&gt;knarra(a)redhat.com&lt;/a&gt;&amp;gt;&lt;/span&gt;
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              <div bgcolor="#FFFFFF">
                <div>
                  <div class="gmail-h5">
                    <div
                      class="gmail-m_-1038946535864355601moz-cite-prefix">On
                      11/24/2016 07:27 PM, Simone Tiraboschi wrote:<br>
                    </div>
                    <blockquote type="cite">
                      <div dir="ltr"><br>
                        <div class="gmail_extra"><br>
                          <div class="gmail_quote">On Thu, Nov 24, 2016
                            at 2:39 PM, knarra <span
dir="ltr">&lt;<a
                                moz-do-not-send="true"
                                href="mailto:knarra@redhat.com"

target=&quot;_blank&quot;&gt;knarra(a)redhat.com&lt;/a&gt;&amp;gt;&lt;/span&gt;
                            wrote:<br>
                            <blockquote class="gmail_quote"
                              style="margin:0px 0px 0px
                              0.8ex;border-left:1px solid
                              rgb(204,204,204);padding-left:1ex">
                              <div bgcolor="#FFFFFF"><span

class="gmail-m_-1038946535864355601gmail-">
                                  <div
class="gmail-m_-1038946535864355601gmail-m_-2672125017527102966moz-cite-prefix">On
                                    11/24/2016 06:56 PM, Simone
                                    Tiraboschi wrote:<br>
                                  </div>
                                  <blockquote type="cite">
                                    <div dir="ltr"><br>
                                      <div class="gmail_extra"><br>
                                        <div class="gmail_quote">On Thu,
                                          Nov 24, 2016 at 2:08 PM,
                                          knarra <span
dir="ltr">&lt;<a
                                              moz-do-not-send="true"
                                              href="mailto:knarra@redhat.com"

target=&quot;_blank&quot;&gt;knarra(a)redhat.com&lt;/a&gt;&amp;gt;&lt;/span&gt;
                                          wrote:<br>
                                          <blockquote
                                            class="gmail_quote"
                                            style="margin:0px 0px 0px
                                            0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
                                            <div bgcolor="#FFFFFF">
                                              <div>
                                                <div

class="gmail-m_-1038946535864355601gmail-m_-2672125017527102966h5">
                                                  <div
class="gmail-m_-1038946535864355601gmail-m_-2672125017527102966m_-6170493833456119351moz-cite-prefix">On
                                                    11/24/2016 06:15 PM,
                                                    Simone Tiraboschi
                                                    wrote:<br>
                                                  </div>
                                                  <blockquote
                                                    type="cite">
                                                    <div
dir="ltr"><br>
                                                      <div

class="gmail_extra"><br>
                                                        <div

class="gmail_quote">On
                                                          Thu, Nov 24,
                                                          2016 at 1:26
                                                          PM, knarra <span

dir="ltr">&lt;<a
moz-do-not-send="true" href="mailto:knarra@redhat.com"
target=&quot;_blank&quot;&gt;knarra(a)redhat.com&lt;/a&gt;&amp;gt;&lt;/span&gt;
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid

rgb(204,204,204);padding-left:1ex">Hi,<br>
                                                          <br>
                                                              I have
                                                          three nodes
                                                          with glusterfs
                                                          as storage
                                                          domain. For
                                                          some reason i
                                                          see that
                                                          vm.conf from

/var/run/ovirt-hosted-engine-h<wbr>a
                                                          is missing and
                                                          due to this on
                                                          one of my host
                                                          i see that
                                                          Hosted Engine
                                                          HA : Not
                                                          Active. Once i
                                                          copy the file
                                                          from some
                                                          other node and
                                                          restart
                                                          ovirt-ha-broker
                                                          and
                                                          ovirt-ha-agent
                                                          services
                                                          everything
                                                          works fine.
                                                          But then this
                                                          happens again.
                                                          Can some
                                                          please help me
                                                          identify why
                                                          this happens.
                                                          Below is the
                                                          log i see in
                                                          ovirt-ha-agent.logs.<br>
                                                          <br>
                                                          <br>
                                                          <a

moz-do-not-send="true"
href="https://paste.fedoraproject.org/489120/79990345/"
rel="noreferrer"
target="_blank">https://paste.fedoraproject.or<wbr>g/489120/79990345/</a><br>
                                                          <br>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          Once the
                                                          engine
                                                          correctly
                                                          imported the
                                                          hosted-engine
                                                          storage
                                                          domain, a
                                                          couple of
                                                          OVF_STORE
                                                          volumes will
                                                          appear there.<br>
                                                          Every
                                                          modification
                                                          to the engine
                                                          VM
                                                          configuration
                                                          will be
                                                          written by the
                                                          engine into
                                                          that
                                                          OVF_STORE, so
                                                          all the
                                                          ovirt-ha-agent
                                                          running on the
                                                          hosted-engine
                                                          hosts will be
                                                          able to
                                                          re-start the
                                                          engine VM with
                                                          a coherent
                                                          configuration.<br>
                                                          <br>
                                                          Till the
                                                          engine imports
                                                          the
                                                          hosted-engine
                                                          storage
                                                          domain,
                                                          ovirt-ha-agent
                                                          will fall back
                                                          to the initial
                                                          vm.conf.<br>
                                                          <br>
                                                          In you case
                                                          the OVF_STORE
                                                          volume is
                                                          there,<br>
                                                          but the agent
                                                          fails
                                                          extracting the
                                                          engine VM
                                                          configuration:<br>
                                                          MainThread::<a
moz-do-not-send="true"
class="gmail-m_-1038946535864355601gmail-m_-2672125017527102966m_-6170493833456119351moz-txt-link-freetext">INFO::2016-11-24</a>
17:55:04,914::ovf_store::112::<wbr>ovirt_hosted_engine_ha.lib.ovf<wbr>.ovf_store.OVFStore::(getEngin<wbr>eVMOVF)
                                                          Extracting
                                                          Engine VM OVF
                                                          from the
                                                          OVF_STORE<br>
                                                          MainThread::<a
moz-do-not-send="true"
class="gmail-m_-1038946535864355601gmail-m_-2672125017527102966m_-6170493833456119351moz-txt-link-freetext">INFO::2016-11-24</a>
17:55:04,919::ovf_store::119::<wbr>ovirt_hosted_engine_ha.lib.ovf<wbr>.ovf_store.OVFStore::(getEngin<wbr>eVMOVF)
                                                          OVF_STORE
                                                          volume path:

/rhev/data-center/mnt/glusterS<wbr>D/10.70.36.79:_engine/27f054c3<wbr>-c245-4039-b42a-c28b37043016/i<wbr>mages/fdf49778-9a06-49c6-bf7a-<wbr>a0f12425911c/8c954add-6bcf-<wbr>47f8-ac2e-4c85fc3f8699<br>
MainThread::ERROR::2016-11-24
17:55:04,928::ovf_store::124::<wbr>ovirt_hosted_engine_ha.lib.ovf<wbr>.ovf_store.OVFStore::(getEngin<wbr>eVMOVF)
                                                          Unable to
                                                          extract HEVM
                                                          OVF<br>
                                                          <br>
                                                          So it tries to
                                                          rollback to
                                                          the initial
                                                          vm.conf, but
                                                          also that one
                                                          seams to miss
                                                          some values
                                                          and so the
                                                          agent is
                                                          failing:<br>
MainThread::ERROR::2016-11-24
17:55:04,974::agent::205::ovir<wbr>t_hosted_engine_ha.agent.agent<wbr>.Agent::(_run_agent)
                                                          Error:
                                                          ''Configuration
                                                          value not
                                                          found:

file=/var/run/ovirt-hosted-eng<wbr>ine-ha/vm.conf,
                                                          key=memSize''
                                                          - trying to
                                                          restart agent<br>
                                                          <br>
                                                          Both of the
                                                          issue seams
                                                          storage
                                                          related, could
                                                          yuo please
                                                          share your
                                                          gluster logs?<br>
                                                          <br>

                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid

rgb(204,204,204);padding-left:1ex">
                                                          <br>
                                                          Thanks<br>
                                                          <br>
                                                          kasturi<br>
                                                          <br>
                                                          </blockquote>
                                                        </div>
                                                        <br>
                                                      </div>
                                                    </div>
                                                  </blockquote>
                                                </div>
                                              </div>
                                              <p>Hi Simone,</p>
                                              <p>    Below [1] is the
                                                link for the sosreports
                                                on the first two hosts.
                                                The third host has some
                                                issue. Once it is up
                                                will give the sosreport
                                                from there as well.</p>
                                            </div>
                                          </blockquote>
                                          <div><br>
                                          </div>
                                          <div>And the host where you
                                            see the initial issue was
                                            the third one? <br>
                                          </div>
                                        </div>
                                      </div>
                                    </div>
                                  </blockquote>
                                </span> It is on the first host.<span

class="gmail-m_-1038946535864355601gmail-"><br>
                                  <blockquote type="cite">
                                    <div dir="ltr">
                                      <div class="gmail_extra">
                                        <div class="gmail_quote">
                                          <div> </div>
                                        </div>
                                      </div>
                                    </div>
                                  </blockquote>
                                </span></div>
                            </blockquote>
                            <div><br>
                            </div>
                            <div>It seams that host1 is failing reading
                              from the the hosted-engine storage domain:</div>
                            <div><br>
                            </div>
                            <div>
                              <div>[2016-11-24 12:33:43.678467] W
                                [MSGID: 114031]
[client-rpc-fops.c:2938:<wbr>client3_3_lookup_cbk]
                                0-engine-client-2: remote operation
                                failed. Path: /
                                (00000000-0000-0000-0000-<wbr>000000000001)
                                [Transport endpoint is not connected]</div>
                              <div>[2016-11-24 12:33:43.678747] E
                                [rpc-clnt.c:365:saved_frames_<wbr>unwind]
                                (--&gt;
/lib64/libglusterfs.so.0(_gf_<wbr>log_callingfn+0x192)[<wbr>0x7f077eba1642]
                                (--&gt;
/lib64/libgfrpc.so.0(saved_<wbr>frames_unwind+0x1de)[<wbr>0x7f077e96775e]
                                (--&gt;
/lib64/libgfrpc.so.0(saved_<wbr>frames_destroy+0xe)[<wbr>0x7f077e96786e]
                                (--&gt;
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x84)[<wbr>0x7f077e968fc4]
                                (--&gt;
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>notify+0x120)[0x7f077e9698a0]
                                ))))) 0-engine-client-2: forced
                                unwinding frame type(GlusterFS 3.3)
                                op(LOOKUP(27)) called at 2016-11-24
                                12:33:07.495178 (xid=0x82a1c)</div>
                              <div>[2016-11-24 12:33:43.678982] E
                                [rpc-clnt.c:365:saved_frames_<wbr>unwind]
                                (--&gt;
/lib64/libglusterfs.so.0(_gf_<wbr>log_callingfn+0x192)[<wbr>0x7f077eba1642]
                                (--&gt;
/lib64/libgfrpc.so.0(saved_<wbr>frames_unwind+0x1de)[<wbr>0x7f077e96775e]
                                (--&gt;
/lib64/libgfrpc.so.0(saved_<wbr>frames_destroy+0xe)[<wbr>0x7f077e96786e]
                                (--&gt;
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x84)[<wbr>0x7f077e968fc4]
                                (--&gt;
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>notify+0x120)[0x7f077e9698a0]
                                ))))) 0-engine-client-2: forced
                                unwinding frame type(GlusterFS 3.3)
                                op(LOOKUP(27)) called at 2016-11-24
                                12:33:08.770637 (xid=0x82a1d)</div>
                              <div>[2016-11-24 12:33:43.679001] W
                                [MSGID: 114031]
[client-rpc-fops.c:2938:<wbr>client3_3_lookup_cbk]
                                0-engine-client-2: remote operation
                                failed. Path:

/27f054c3-c245-4039-b42a-<wbr>c28b37043016/images/39960f40-<wbr>4aae-4714-ba73-1637785fae7c/<wbr>38fa3519-f21e-4671-8c69-<wbr>d1497ff8a490
                                (1090c25b-9c90-434e-a133-<wbr>faf9647cc992)
                                [Transport endpoint is not connected]</div>
                              <div>[2016-11-24 12:33:43.679303] E
                                [rpc-clnt.c:365:saved_frames_<wbr>unwind]
                                (--&gt;
/lib64/libglusterfs.so.0(_gf_<wbr>log_callingfn+0x192)[<wbr>0x7f077eba1642]
                                (--&gt;
/lib64/libgfrpc.so.0(saved_<wbr>frames_unwind+0x1de)[<wbr>0x7f077e96775e]
                                (--&gt;
/lib64/libgfrpc.so.0(saved_<wbr>frames_destroy+0xe)[<wbr>0x7f077e96786e]
                                (--&gt;
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x84)[<wbr>0x7f077e968fc4]
                                (--&gt;
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>notify+0x120)[0x7f077e9698a0]
                                ))))) 0-engine-client-2: forced
                                unwinding frame type(GlusterFS 3.3)
                                op(LOOKUP(27)) called at 2016-11-24
                                12:33:11.096856 (xid=0x82a1e)</div>
                              <div>[2016-11-24 12:33:43.679596] E
                                [rpc-clnt.c:365:saved_frames_<wbr>unwind]
                                (--&gt;
/lib64/libglusterfs.so.0(_gf_<wbr>log_callingfn+0x192)[<wbr>0x7f077eba1642]
                                (--&gt;
/lib64/libgfrpc.so.0(saved_<wbr>frames_unwind+0x1de)[<wbr>0x7f077e96775e]
                                (--&gt;
/lib64/libgfrpc.so.0(saved_<wbr>frames_destroy+0xe)[<wbr>0x7f077e96786e]
                                (--&gt;
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x84)[<wbr>0x7f077e968fc4]
                                (--&gt;
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>notify+0x120)[0x7f077e9698a0]
                                ))))) 0-engine-client-2: forced
                                unwinding frame type(GF-DUMP)
                                op(NULL(2)) called at 2016-11-24
                                12:33:13.673743 (xid=0x82a1f)</div>
                              <div>[2016-11-24 12:33:43.682310] I
                                [socket.c:3401:socket_submit_<wbr>request]
                                0-engine-client-2: not connected
                                (priv-&gt;connected = 0)</div>
                              <div>[2016-11-24 12:33:43.682328] W
                                [rpc-clnt.c:1640:rpc_clnt_<wbr>submit]
                                0-engine-client-2: failed to submit
                                rpc-request (XID: 0x82a20 Program:
                                GlusterFS 3.3, ProgVers: 330, Proc: 30)
                                to rpc-transport (engine-client-2)</div>
                              <div>[2016-11-24 12:33:43.682391] W
                                [rpc-clnt.c:1640:rpc_clnt_<wbr>submit]
                                0-engine-client-2: failed to submit
                                rpc-request (XID: 0x82a21 Program:
                                GlusterFS 3.3, ProgVers: 330, Proc: 27)
                                to rpc-transport (engine-client-2)</div>
                              <div>[2016-11-24 12:33:43.682441] W
                                [rpc-clnt.c:1640:rpc_clnt_<wbr>submit]
                                0-engine-client-2: failed to submit
                                rpc-request (XID: 0x82a22 Program:
                                GlusterFS 3.3, ProgVers: 330, Proc: 27)
                                to rpc-transport (engine-client-2)</div>
                              <div>[2016-11-24 12:33:43.682441] W
                                [MSGID: 114031]
[client-rpc-fops.c:2938:<wbr>client3_3_lookup_cbk]
                                0-engine-client-2: remote operation
                                failed. Path:
/27f054c3-c245-4039-b42a-<wbr>c28b37043016
                                (a64398f5-3fa3-48fe-9d40-<wbr>d3860876cc2c)
                                [Transport endpoint is not connected]</div>
                              <div>[2016-11-24 12:33:43.682492] W
                                [rpc-clnt-ping.c:203:rpc_clnt_<wbr>ping_cbk]
                                0-engine-client-2: socket disconnected</div>
                              <div>[2016-11-24 12:33:43.682536] I
                                [MSGID: 114018]
                                [client.c:2280:client_rpc_<wbr>notify]
                                0-engine-client-2: disconnected from
                                engine-client-2. Client process will
                                keep trying to connect to glusterd until
                                brick's port is available</div>
                              <div>[2016-11-24 12:33:43.682562] W
                                [rpc-clnt.c:1640:rpc_clnt_<wbr>submit]
                                0-engine-client-2: failed to submit
                                rpc-request (XID: 0x82a23 Program:
                                GlusterFS 3.3, ProgVers: 330, Proc: 27)
                                to rpc-transport (engine-client-2)</div>
                              <div>The message "W [MSGID: 114031]
                                [client-rpc-fops.c:2938:<wbr>client3_3_lookup_cbk]
                                0-engine-client-2: remote operation
                                failed. Path:
/27f054c3-c245-4039-b42a-<wbr>c28b37043016
                                (a64398f5-3fa3-48fe-9d40-<wbr>d3860876cc2c)
                                [Transport endpoint is not connected]"
                                repeated 2 times between [2016-11-24
                                12:33:43.682441] and [2016-11-24
                                12:33:43.682599]</div>
                              <div>[2016-11-24 12:33:43.688324] W
                                [MSGID: 114031]
[client-rpc-fops.c:2938:<wbr>client3_3_lookup_cbk]
                                0-engine-client-2: remote operation
                                failed. Path: (null)
                                (00000000-0000-0000-0000-<wbr>000000000000)
                                [Transport endpoint is not connected]</div>
                            </div>
                            <div><br>
                            </div>
                            <div>Before that there was a lot of
                              self-healing activities.</div>
                            <div><br>
                            </div>
                          </div>
                        </div>
                      </div>
                    </blockquote>
                  </div>
                </div>
                simone, these logs indicates that first host is not able
                to connect to the brick in the third host since it was
                powered down.  It reads remote operation failed on
                engine_client_2 .</div>
            </blockquote>
            <div><br>
            </div>
            <div>Can you please share the output of</div>
             source /etc/ovirt-hosted-engine/hosted-engine.conf<br>
             find /rhev/data-center/ -path
            "*/${sdUUID}/images/${conf_image_UUID}/${conf_volume_UUID}"
            -type f -exec sh -c 'sudo -u vdsm dd if=$1 2&gt;/dev/null |
            tar -xOvf - vm.conf  2&gt;/dev/null' {} {} \;
            <div>executed on your first host?</div>
          </div>
        </div>
      </div>
    </blockquote>
    output for the above command
    <a class="moz-txt-link-freetext"
href="https://paste.fedoraproject.org/489159/97176147/">http...
    <br>
    Simone, i ran the following steps and i see that vm.conf file is
    missing.<br>
    <br>
    1) Installed three nodes with glusterfs as storage domains.<br>
    <br>
    2) Configured power management on all the nodes and ran the command
    'poweroff -f' on one of the node.<br>
    <br>
    3) Engine tried to fence the host and it logged the message 'Host is
    rebooting' but this never came up.<br>
    <br>
    4) After some time i manually powered on the host and once the host
    is up i see that Hosted Engine HA : Not active.<br>
    <br>
    5) I see the following error message in broker.log and vm.conf file
    is missing.<br>
    <br>
    Thread-1::ERROR::2016-11-24
19:32:11,651::listener::192::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
    Error handling request, data: 'set-storage-domain<br>
     FilesystemBackend dom_type=glusterfs
    sd_uuid=27f054c3-c245-4039-b42a-c28b37043016'<br>
    Traceback (most recent call last):<br>
      File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
    line 166, in handle<br>
        data)<br>
      File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
    line 299, in _dispatch<br>
        .set_storage_domain(client, sd_type, **options)<br>
      File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
    line 66, in set_storage_domain<br>
        self._backends[client].connect()<br>
      File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
    line 461, in connect<br>
        self._dom_type)<br>
      File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
    line 108, in get_domain_path<br>
        " in {1}".format(sd_uuid, parent))<br>
    BackendFailureException: path to storage domain
    27f054c3-c245-4039-b42a-c28b37043016 not found in
    /rhev/data-center/mnt/glusterSD<br>
    <br>
    Thanks<br>
    kasturi<br>
    <br>
    <blockquote
cite="mid:CAN8-ONqt1w1uF-GqQ-8Hr2cN+E61AatDRpmnU=KB_LW8b9OBig@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">
            <div><br>
            </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              <div bgcolor="#FFFFFF"><span
class="gmail-"><br>
                  <blockquote type="cite">
                    <div dir="ltr">
                      <div class="gmail_extra">
                        <div class="gmail_quote">
                          <div><br>
                          </div>
                          <div> </div>
                          <blockquote class="gmail_quote"
                            style="margin:0px 0px 0px
                            0.8ex;border-left:1px solid
                            rgb(204,204,204);padding-left:1ex">
                            <div bgcolor="#FFFFFF"><span
                                class="gmail-m_-1038946535864355601gmail-">
                                <blockquote type="cite">
                                  <div dir="ltr">
                                    <div class="gmail_extra">
                                      <div class="gmail_quote">
                                        <blockquote class="gmail_quote"
                                          style="margin:0px 0px 0px
                                          0.8ex;border-left:1px solid
                                          rgb(204,204,204);padding-left:1ex">
                                          <div bgcolor="#FFFFFF">
                                            <p>[1] <a
                                                moz-do-not-send="true"
class="gmail-m_-1038946535864355601gmail-m_-2672125017527102966m_-6170493833456119351moz-txt-link-freetext"
href="http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/HC/vm_conf...

target="_blank">http://rhsqe-repo.lab.eng.blr.<wbr>red...
                                            <p>Thanks</p>
                                            <p>kasturi<br>
                                            </p>
                                          </div>
                                        </blockquote>
                                      </div>
                                      <br>
                                    </div>
                                  </div>
                                </blockquote>
                                <p><br>
                                </p>
                              </span></div>
                          </blockquote>
                        </div>
                        <br>
                      </div>
                    </div>
                  </blockquote>
                  <p><br>
                  </p>
                </span></div>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <p><br>
    </p>
  </body>
</html>

--------------08CE69774B978D28F256FCDD--

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [ovirt-users] vm.conf on one of the node is missing