<html>
  <head>
    <meta content="text/html; charset=windows-1252"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">On 04/13/2017 02:35 AM, Precht, Andrew
      wrote:<br>
    </div>
    <blockquote
cite="mid:BY1PR09MB0824328CE2138DBDA861057EF5030@BY1PR09MB0824.namprd09.prod.outlook.com"
      type="cite">
      <meta http-equiv="Content-Type" content="text/html;
        charset=windows-1252">
      <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
        dir="ltr">
        <div>Hi all,</div>
        <div>In the end, I ran this on each host node and is what
          worked:</div>
        <div>systemctl stop glusterd &amp;&amp; rm -rf
          /var/lib/glusterd/vols/* &amp;&amp; rm -rf
          /var/lib/glusterd/peers/*</div>
      </div>
    </blockquote>
    <blockquote
cite="mid:BY1PR09MB0824328CE2138DBDA861057EF5030@BY1PR09MB0824.namprd09.prod.outlook.com"
      type="cite">
      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
        dir="ltr">
        <div>Thanks so much for your help.</div>
        <div><br>
        </div>
      </div>
    </blockquote>
    YW :-)<br>
    <blockquote
cite="mid:BY1PR09MB0824328CE2138DBDA861057EF5030@BY1PR09MB0824.namprd09.prod.outlook.com"
      type="cite">
      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
        dir="ltr">
        <div>
        </div>
        <div>P.S. I work as a sys admin for the San Jose library. Part
          of my job satisfaction comes from knowing that the work I do
          here goes directly back into this community. We’r fortunate
          that you, your coworkers, and Red Hat do so much to give back.
          I have to imagine you too feel this sense of satisfaction.
          Thanks again…</div>
        <div><br>
        </div>
        <div>P.S.S. I never did hear back from <a class="moz-txt-link-abbreviated" href="mailto:users@ovirt.org">users@ovirt.org</a> mailing
          list. I did fill out the fields on this page:
          <a class="moz-txt-link-freetext" href="https://lists.ovirt.org/mailman/listinfo/users">https://lists.ovirt.org/mailman/listinfo/users</a>. Yet, everytime
          I send them an email I get: Your message to Users awaits
          moderator approval. Is there a secret handshake, I’m not aware
          of?</div>
      </div>
    </blockquote>
    Ideally you should be able to. Not sure why . May be a separate
    email thread to this user list should help you get the right answer
    ?<br>
    <blockquote
cite="mid:BY1PR09MB0824328CE2138DBDA861057EF5030@BY1PR09MB0824.namprd09.prod.outlook.com"
      type="cite">
      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
        dir="ltr">
        <div><br>
        </div>
        <div>Regards,</div>
        <div>Andrew</div>
        <br>
      </div>
      <hr style="display:inline-block;width:98%" tabindex="-1">
      <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
          color="#000000" face="Calibri, sans-serif"><b>From:</b> knarra
          <a class="moz-txt-link-rfc2396E" href="mailto:knarra@redhat.com">&lt;knarra@redhat.com&gt;</a><br>
          <b>Sent:</b> Wednesday, April 12, 2017 10:01:33 AM<br>
          <b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal
          Nisan; Allon Mureinik; Nir Soffer<br>
          <b>Cc:</b> users<br>
          <b>Subject:</b> Re: [ovirt-users] I’m having trouble deleting
          a test gluster volume</font>
        <div> </div>
      </div>
      <div>
        <div class="moz-cite-prefix">On 04/12/2017 08:45 PM, Precht,
          Andrew wrote:<br>
        </div>
        <blockquote
cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com"
          type="cite">
          <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
          <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
            dir="ltr">
            <p>Hi all,</p>
            <p><span
                id="docs-internal-guid-44c81590-62b4-32c0-d6d1-82b627328ebb"></span></p>
            <p dir="ltr"
              style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">You asked: Any errors in ovirt-engine.log file ?</span></p>
            <p dir="ltr"
              style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:
              36pt;">
              <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">Yes, In the engine.log this error is repeated about every 3 minutes:</span></p>
            <br>
            <p dir="ltr"
              style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">2017-04-12 07:16:12,554-07 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob]
 (DefaultQuartzScheduler3) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Error updating tasks from CLI: org.ovirt.engine.core.common.errors.EngineException: EngineException: Command execution failed</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">
</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">error: Error : Request timed out</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">
</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">return code: 1 (Failed with error GlusterVolumeStatusAllFailedException and code 4161)</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">
</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">error: Error : Request timed out</span></p>
          </div>
        </blockquote>
        I am not sure why this says "Request timed out".<br>
        <blockquote
cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com"
          type="cite">
          <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
            dir="ltr">
            <p dir="ltr"
              style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:
              36pt;">
              <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"></span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">1)
 gluster volume list -&gt;  Still shows the deleted volume (test1)</span></p>
          </div>
        </blockquote>
        <blockquote
cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com"
          type="cite">
          <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
            dir="ltr">
            <p dir="ltr"
              style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:
              36pt;">
              <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"></span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">2)
 gluster peer status -&gt; Shows one of the peers twice with different uuid’s:</span></p>
            <p dir="ltr"
              style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">Hostname: 192.168.10.109</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">
</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">Uuid: 42fbb7de-8e6f-4159-a601-3f858fa65f6c</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">
</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">State: Peer in Cluster (Connected)</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">
</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"></span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">Hostname:
 192.168.10.109</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">
</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">Uuid: e058babe-7f9d-49fe-a3ea-ccdc98d7e5b5</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">
</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">State: Peer in Cluster (Connected)</span></p>
          </div>
        </blockquote>
        How did this happen? Are the hostname same for two hosts ?<br>
        <blockquote
cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com"
          type="cite">
          <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
            dir="ltr">
            <p dir="ltr"
              style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"></span></p>
            <p dir="ltr"
              style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
                style="vertical-align: baseline;"><font color="#212121"
                  face="Arial"><span style="font-size: 11.5pt; white-space: pre-wrap;">I tried a gluster volume stop test1, with this
</span><span style="font-size: 15.3333px; white-space: pre-wrap;">result</span><span style="font-size: 11.5pt; white-space: pre-wrap;">:
<span>volume stop: test1: failed: Another transaction is in progress for test1. Please try again after sometime.</span></span></font></span></p>
          </div>
        </blockquote>
        <font color="#212121"><font face="Arial">can you restart
            glusterd and try to stop and delete the volume?
          </font></font><br>
        <blockquote
cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com"
          type="cite">
          <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
            dir="ltr">
            <p dir="ltr"
              style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt; white-space: pre-wrap; text-indent: 36pt;"></span></p>
            <p dir="ltr"
              style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt; white-space: pre-wrap; text-indent: 36pt;"></span></p>
            <p dir="ltr"
              style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"><span></span></span><span style="color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt; white-space: pre-wrap; text-indent: 36pt;">The
 etc-glusterfs-glusterd.vol.log shows no activity triggered by trying to remove the test1 volume from the UI</span><span style="color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt; white-space: pre-wrap; text-indent: 36pt;">.</span></p>
            <br>
            <p dir="ltr"
              style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">The ovirt-engine.log shows this repeating many times, when
<span style="color: rgb(33, 33, 33); font-family: Arial; font-size: 15.3333px; text-indent: 48px; white-space: pre-wrap;">
trying to remove the test1 volume from the UI</span>:</span></p>
            <br>
            <p dir="ltr"
              style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">2017-04-12 07:57:38,049-07 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
 (DefaultQuartzScheduler9) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[b0e1b909-9a6a-49dc-8e20-3a027218f7e1=&lt;GLUSTER, ACTION_TYPE_FAILED_GLUSTER_OPERATION_INPROGRESS&gt;]', sharedLocks='null'}'</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">
</span></p>
          </div>
        </blockquote>
        can you restart ovirt-engine service because i see that "failed
        to acquire lock".  Once ovirt-engine is restarted some one who
        is holding the lock should be release  and things should work
        fine.<br>
        <br>
        Last but not least, if none of the above works:<br>
        <br>
        Login to all your nodes in the cluster.<br>
        rm -rf /var/lib/glusterd/vols/*<br>
        rm -rf /var/lib/glusterd/peers/*<br>
        systemctl restart glusterd on all the nodes.<br>
        <br>
        Login to UI and see if any volumes / hosts are present. If yes,
        remove them.<br>
        <br>
        This should clear things for you and you can start from basic.<br>
        <br>
        <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"></span><br
          class="kix-line-break">
        <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"></span>
        <blockquote
cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com"
          type="cite">
          <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
            dir="ltr">
            <p dir="ltr"
              style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"></span></p>
            Thanks much,
            <p>Andrew</p>
          </div>
          <hr style="display:inline-block;width:98%" tabindex="-1">
          <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
              color="#000000" face="Calibri, sans-serif"><b>From:</b>
              knarra
              <a moz-do-not-send="true" class="moz-txt-link-rfc2396E"
                href="mailto:knarra@redhat.com">&lt;knarra@redhat.com&gt;</a><br>
              <b>Sent:</b> Tuesday, April 11, 2017 11:10:04 PM<br>
              <b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose;
              Tal Nisan; Allon Mureinik; Nir Soffer<br>
              <b>Cc:</b> users<br>
              <b>Subject:</b> Re: [ovirt-users] I’m having trouble
              deleting a test gluster volume</font>
            <div> </div>
          </div>
          <div>
            <div class="moz-cite-prefix">On 04/12/2017 03:35 AM, Precht,
              Andrew wrote:<br>
            </div>
            <blockquote
cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
              type="cite">
              <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
              <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
                dir="ltr">
                <p>I just <span>noticed </span>this in the Alerts tab: <span>Detected
                    deletion of volume test1 on cluster 8000-1, and
                    deleted it from engine DB.</span><br>
                </p>
              </div>
            </blockquote>
            <blockquote
cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
              type="cite">
              <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
                dir="ltr">
                <p><span>Yet, It still shows in the web UI?</span></p>
              </div>
            </blockquote>
            Any errors in ovirt-engine.log file ? if the volume is
            deleted from db ideally it should be deleted from UI too. 
            Can you go to gluster nodes and check for the following:<br>
            <br>
            1) gluster volume list -&gt; should not return anything
            since you have deleted the volumes.<br>
            <br>
            2) gluster peer status -&gt; on all the nodes should show
            that all the peers are in connected state.<br>
            <br>
            can you tail -f /var/log/ovirt-engine/ovirt-engine.log and
            gluster log and capture the error messages when you try
            deleting the volume from UI?<br>
            <br>
            Log what you have pasted in the previous mail only gives
            info and i could not get any details from that on why volume
            delete is failing<br>
            <br>
            <blockquote
cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
              type="cite">
              <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
                dir="ltr">
              </div>
              <hr style="display:inline-block;width:98%" tabindex="-1">
              <div id="divRplyFwdMsg" dir="ltr"><font
                  style="font-size:11pt" color="#000000" face="Calibri,
                  sans-serif"><b>From:</b> Precht, Andrew<br>
                  <b>Sent:</b> Tuesday, April 11, 2017 2:39:31 PM<br>
                  <b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal
                  Nisan; Allon Mureinik; Nir Soffer<br>
                  <b>Cc:</b> users<br>
                  <b>Subject:</b> Re: [ovirt-users] I’m having trouble
                  deleting a test gluster volume</font>
                <div> </div>
              </div>
              <div>
                <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
                <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
                  dir="ltr">
                  <div>The plot thickens…</div>
                  <div>I put all hosts in the cluster into maintenance
                    mode, with the Stop Gluster service checkbox
                    checked. I then deleted the
                    /var/lib/glusterd/vols/test1 directory on all hosts.
                    I then took the host that the test1 volume was on
                    out of maintenance mode. Then I tried to remove the
                    test1 volume from within the web UI. With no luck, I
                    got the message: Could not delete Gluster Volume
                    test1 on cluster 8000-1.</div>
                  <div><br>
                  </div>
                  <div>I went back and checked all host for the test1
                    directory, it is not on any host. Yet I still can’t
                    remove it…</div>
                  <div><br>
                  </div>
                  <div>Any suggestions?</div>
                  <br>
                </div>
                <hr style="display:inline-block;width:98%" tabindex="-1">
                <div id="divRplyFwdMsg" dir="ltr"><font
                    style="font-size:11pt" color="#000000"
                    face="Calibri, sans-serif"><b>From:</b> Precht,
                    Andrew<br>
                    <b>Sent:</b> Tuesday, April 11, 2017 1:15:22 PM<br>
                    <b>To:</b> knarra; Sandro Bonazzola; Sahina Bose;
                    Tal Nisan; Allon Mureinik; Nir Soffer<br>
                    <b>Cc:</b> users<br>
                    <b>Subject:</b> Re: [ovirt-users] I’m having trouble
                    deleting a test gluster volume</font>
                  <div> </div>
                </div>
                <div>
                  <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
                  <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
                    dir="ltr">
                    <div>Here is an update…</div>
                    <div><br>
                    </div>
                    <div>I checked the
                      /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
                      on the node that had the trouble volume (test1). I
                      didn’t see any errors. So, I ran a tail -f on the
                      log as I tried to remove the volume using the web
                      UI. here is what was appended:</div>
                    <div><br>
                    </div>
                    <div>[2017-04-11 19:48:40.756360] I [MSGID: 106487]
[glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd:
                      Received cli list req</div>
                    <div>[2017-04-11 19:48:42.238840] I [MSGID: 106488]
[glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management:
                      Received get vol req</div>
                    <div>The message "I [MSGID: 106487]
                      [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends]
                      0-glusterd: Received cli list req" repeated 6
                      times between [2017-04-11 19:48:40.756360] and
                      [2017-04-11 19:49:32.596536]</div>
                    <div>The message "I [MSGID: 106488]
                      [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume]
                      0-management: Received get vol req" repeated 20
                      times between [2017-04-11 19:48:42.238840] and
                      [2017-04-11 19:49:34.082179]</div>
                    <div>[2017-04-11 19:51:41.556077] I [MSGID: 106487]
[glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd:
                      Received cli list req</div>
                    <div><br>
                    </div>
                    <div>I’m seeing that the timestamps on these log
                      entries do not match the time on the node.</div>
                    <div><br>
                    </div>
                    <div>The next steps</div>
                    <div>I stopped the glusterd service on the node with
                      volume test1</div>
                    <div>I deleted it with:  rm -rf
                      /var/lib/glusterd/vols/test1</div>
                    <div>I started the glusterd service.</div>
                    <div><br>
                    </div>
                    <div>After starting the gluster service back up, the
                      directory /var/lib/glusterd/vols/test1 reappears.</div>
                    <div>I’m guessing syncing with the other nodes?</div>
                    <div>Is this because I have the Volume Option: auth
                      allow *</div>
                    <div>Do I need to remove the directory
                      /var/lib/glusterd/vols/test1 on all nodes in the
                      cluster individually?</div>
                    <div><br>
                    </div>
                    <div>thanks</div>
                    <br>
                  </div>
                  <hr style="display:inline-block;width:98%"
                    tabindex="-1">
                  <div id="divRplyFwdMsg" dir="ltr"><font
                      style="font-size:11pt" color="#000000"
                      face="Calibri, sans-serif"><b>From:</b> knarra
                      <a moz-do-not-send="true"
                        class="moz-txt-link-rfc2396E"
                        href="mailto:knarra@redhat.com">
                        &lt;knarra@redhat.com&gt;</a><br>
                      <b>Sent:</b> Tuesday, April 11, 2017 11:51:18 AM<br>
                      <b>To:</b> Precht, Andrew; Sandro Bonazzola;
                      Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br>
                      <b>Cc:</b> users<br>
                      <b>Subject:</b> Re: [ovirt-users] I’m having
                      trouble deleting a test gluster volume</font>
                    <div> </div>
                  </div>
                  <div>
                    <div class="moz-cite-prefix">On 04/11/2017 11:28 PM,
                      Precht, Andrew wrote:<br>
                    </div>
                    <blockquote
cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
                      type="cite">
                      <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
                      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
                        dir="ltr">
                        <div>Hi all,</div>
                        <div>The node is oVirt Node 4.1.1 with
                          glusterfs-3.8.10-1.el7. </div>
                        <div>On the node I can not find
                          /var/log/glusterfs/glusterd.log However, there
                          is a /var/log/glusterfs/glustershd.log
                          <br>
                        </div>
                      </div>
                    </blockquote>
                    can you check if
                    /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
                    exists? if yes, can you check if there is any error
                    present in that file ?<span style="color: rgb(84,
                      84, 84); font-family: arial, sans-serif;
                      font-size: small; font-style: normal;
                      font-variant-ligatures: normal; font-variant-caps:
                      normal; font-weight: normal; letter-spacing:
                      normal; orphans: 2; text-align: left; text-indent:
                      0px; text-transform: none; white-space: normal;
                      widows: 2; word-spacing: 0px;
                      -webkit-text-stroke-width: 0px; background-color:
                      rgb(255, 255, 255); display: inline !important;
                      float: none;"></span>
                    <blockquote
cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
                      type="cite">
                      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
                        dir="ltr">
                        <div><br>
                        </div>
                        <div>What happens if I follow the four steps
                          outlined here to remove the volume from the
                          node
                          <u>BUT</u>, I do have another volume present
                          in the cluster. It too is a test volume.
                          Neither one has any data on them. So, data
                          loss is not an issue.</div>
                      </div>
                    </blockquote>
                    Running those four steps will remove the volume from
                    your cluster . If the volumes what you have are test
                    volumes you could just follow the steps outlined to
                    delete them (since you are not able to delete from
                    UI) and bring back the cluster into a normal state.<br>
                    <blockquote
cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
                      type="cite">
                      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
                        dir="ltr">
                        <br>
                      </div>
                      <hr style="display:inline-block;width:98%"
                        tabindex="-1">
                      <div id="divRplyFwdMsg" dir="ltr"><font
                          style="font-size:11pt" color="#000000"
                          face="Calibri, sans-serif"><b>From:</b> knarra
                          <a moz-do-not-send="true"
                            class="moz-txt-link-rfc2396E"
                            href="mailto:knarra@redhat.com">
                            &lt;knarra@redhat.com&gt;</a><br>
                          <b>Sent:</b> Tuesday, April 11, 2017 10:32:27
                          AM<br>
                          <b>To:</b> Sandro Bonazzola; Precht, Andrew;
                          Sahina Bose; Tal Nisan; Allon Mureinik; Nir
                          Soffer<br>
                          <b>Cc:</b> users<br>
                          <b>Subject:</b> Re: [ovirt-users] I’m having
                          trouble deleting a test gluster volume</font>
                        <div> </div>
                      </div>
                      <div>
                        <div class="moz-cite-prefix">On 04/11/2017 10:44
                          PM, Sandro Bonazzola wrote:<br>
                        </div>
                        <blockquote
cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com"
                          type="cite">
                          <div dir="auto">Adding some people</div>
                          <div class="gmail_extra"><br>
                            <div class="gmail_quote">Il 11/Apr/2017
                              19:06, "Precht, Andrew" &lt;<a
                                moz-do-not-send="true"
                                href="mailto:Andrew.Precht@sjlibrary.org">Andrew.Precht@sjlibrary.org</a>&gt;
                              ha scritto:<br type="attribution">
                              <blockquote class="gmail_quote"
                                style="margin:0 0 0 .8ex;border-left:1px
                                #ccc solid;padding-left:1ex">
                                <div dir="ltr">
                                  <div
                                    id="m_7567123270890452842divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
                                    dir="ltr">
                                    <div>Hi Ovirt users,</div>
                                    <div>I’m a newbie to oVirt and I’m
                                      having trouble deleting a test
                                      gluster volume. The nodes are
                                      4.1.1 and the engine is 4.1.0</div>
                                    <div><br>
                                    </div>
                                    <div>When I try to remove the test
                                      volume, I click Remove, the dialog
                                      box prompting to confirm the
                                      deletion pops up and after I click
                                      OK, the dialog box changes to show
                                      a little spinning wheel and then
                                      it disappears. In the end the
                                      volume is still there.</div>
                                  </div>
                                </div>
                              </blockquote>
                            </div>
                          </div>
                        </blockquote>
                        with the latest version of glusterfs &amp; ovirt
                        we do not see any issue with deleting a volume.
                        Can you please check
                        /var/log/glusterfs/glusterd.log file if there is
                        any error present?
                        <br>
                        <br>
                        <br>
                        <blockquote
cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com"
                          type="cite">
                          <div class="gmail_extra">
                            <div class="gmail_quote">
                              <blockquote class="gmail_quote"
                                style="margin:0 0 0 .8ex;border-left:1px
                                #ccc solid;padding-left:1ex">
                                <div dir="ltr">
                                  <div
                                    id="m_7567123270890452842divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
                                    dir="ltr">
                                    The test volume was distributed with
                                    two host members. One of the hosts I
                                    was able to remove from the volume
                                    by removing the host form the
                                    cluster. When I try to remove the
                                    remaining host in the volume, even
                                    with the “Force Remove” box ticked,
                                    I get this response: Cannot remove
                                    Host. Server having Gluster volume.
                                    <div><br>
                                    </div>
                                    <div>What to try next?</div>
                                  </div>
                                </div>
                              </blockquote>
                            </div>
                          </div>
                        </blockquote>
                        since you have already removed the volume from
                        one host in the cluster and you still see it on
                        another host you can do the following to remove
                        the volume from another host.<br>
                        <br>
                        1) Login to the host where the volume is
                        present.<br>
                        2) cd to /var/lib/glusterd/vols<br>
                        3) rm -rf &lt;vol_name&gt;<br>
                        4) Restart glusterd on that  host.<br>
                        <br>
                        And before doing the above make sure that you do
                        not have any other volume present in the
                        cluster.<br>
                        <br>
                        Above steps should not be run on a production
                        system as you might loose the volume and data.
                        <br>
                        <br>
                        Now removing the host from UI should succed.<br>
                        <br>
                        <blockquote
cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com"
                          type="cite">
                          <div class="gmail_extra">
                            <div class="gmail_quote">
                              <blockquote class="gmail_quote"
                                style="margin:0 0 0 .8ex;border-left:1px
                                #ccc solid;padding-left:1ex">
                                <div dir="ltr">
                                  <div
                                    id="m_7567123270890452842divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
                                    dir="ltr">
                                    <div><br>
                                    </div>
                                    <div>P.S. I’ve tried to join this
                                      user group several times in the
                                      past, with no response.</div>
                                    <div>Is it possible for me to join
                                      this group?</div>
                                    <div><br>
                                    </div>
                                    <div>Regards,</div>
                                    <div>Andrew</div>
                                    <br>
                                  </div>
                                </div>
                              </blockquote>
                            </div>
                          </div>
                          <br>
                          <fieldset class="mimeAttachmentHeader"></fieldset>
                          <br>
                          <pre wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
                        </blockquote>
                        <p><br>
                        </p>
                      </div>
                    </blockquote>
                    <p><br>
                    </p>
                  </div>
                </div>
              </div>
            </blockquote>
            <p><br>
            </p>
          </div>
        </blockquote>
        <p><br>
        </p>
      </div>
    </blockquote>
    <p><br>
    </p>
  </body>
</html>