Re: [ovirt-users] I’m having trouble deleting a test gluster volume

12 Apr 2017


      This is a multi-part message in MIME format.
--------------338202019CEB535D6EE10912
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit

On 04/12/2017 01:45 AM, Precht, Andrew wrote:
...
Here is an update
I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the 
node that had the trouble volume (test1). I didnt see any errors. So, 
I ran a tail -f on the log as I tried to remove the volume using the 
web UI. here is what was appended:
[2017-04-11 19:48:40.756360] I [MSGID: 106487] 
[glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 
0-glusterd: Received cli list req
[2017-04-11 19:48:42.238840] I [MSGID: 106488] 
[glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 
0-management: Received get vol req
The message "I [MSGID: 106487] 
[glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 
0-glusterd: Received cli list req" repeated 6 times between 
[2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536]
The message "I [MSGID: 106488] 
[glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 
0-management: Received get vol req" repeated 20 times between 
[2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179]
[2017-04-11 19:51:41.556077] I [MSGID: 106487] 
[glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 
0-glusterd: Received cli list req
Im seeing that the timestamps on these log entries do not match the 
time on the node.
gluster logs are in UTC format. That is the reason you might be seeing a 
different timestamp on your node and in the gluster logs.
The next steps
I stopped the glusterd service on the node with volume test1
I deleted it with:  rm -rf /var/lib/glusterd/vols/test1
I started the glusterd service.
After starting the gluster service back up, the directory 
/var/lib/glusterd/vols/test1 reappears.
Im guessing syncing with the other nodes?
yes, since you deleted it only one one node.
Is this because I have the Volume Option: auth allow *
Do I need to remove the directory /var/lib/glusterd/vols/test1 on all 
nodes in the cluster individually?
you need to remove the file /var/lib/glusterd/vols/test1 on all nodes 
and restart glusterd service on all the nodes in the cluster.
thanks
...
Hi all,
The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7.
On the node I can not find /var/log/glusterfs/glusterd.log However, 
there is a /var/log/glusterfs/glustershd.log
can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log 
exists? if yes, can you check if there is any error present in that file ?
What happens if I follow the four steps outlined here to remove the 
volume from the node _BUT_, I do have another volume present in the 
cluster. It too is a test volume. Neither one has any data on them. 
So, data loss is not an issue.
Running those four steps will remove the volume from your cluster . If
------------------------------------------------------------------------
*From:* knarra <knarra@redhat.com>
*Sent:* Tuesday, April 11, 2017 11:51:18 AM
*To:* Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon 
Mureinik; Nir Soffer
*Cc:* users
*Subject:* Re: [ovirt-users] Im having trouble deleting a test 
gluster volume
On 04/11/2017 11:28 PM, Precht, Andrew wrote:
the volumes what you have are test volumes you could just follow the 
steps outlined to delete them (since you are not able to delete from 
UI) and bring back the cluster into a normal state.
...
------------------------------------------------------------------------
*From:* knarra <knarra@redhat.com>
*Sent:* Tuesday, April 11, 2017 10:32:27 AM
*To:* Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon 
Mureinik; Nir Soffer
*Cc:* users
*Subject:* Re: [ovirt-users] Im having trouble deleting a test 
gluster volume
On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:
...
Adding some people
Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht@sjlibrary.org 
<mailto:Andrew.Precht@sjlibrary.org>> ha scritto:
Hi Ovirt users,
    Im a newbie to oVirt and Im having trouble deleting a test
    gluster volume. The nodes are 4.1.1 and the engine is 4.1.0
When I try to remove the test volume, I click Remove, the dialog
    box prompting to confirm the deletion pops up and after I click
    OK, the dialog box changes to show a little spinning wheel and
    then it disappears. In the end the volume is still there.
with the latest version of glusterfs & ovirt we do not see any issue 
with deleting a volume. Can you please check 
/var/log/glusterfs/glusterd.log file if there is any error present?
...
The test volume was distributed with two host members. One of
    the hosts I was able to remove from the volume by removing the
    host form the cluster. When I try to remove the remaining host
    in the volume, even with the Force Remove box ticked, I get
    this response: Cannot remove Host. Server having Gluster volume.
What to try next?
since you have already removed the volume from one host in the 
cluster and you still see it on another host you can do the following 
to remove the volume from another host.
1) Login to the host where the volume is present.
2) cd to /var/lib/glusterd/vols
3) rm -rf <vol_name>
4) Restart glusterd on that  host.
And before doing the above make sure that you do not have any other 
volume present in the cluster.
Above steps should not be run on a production system as you might 
loose the volume and data.
Now removing the host from UI should succed.
...
P.S. Ive tried to join this user group several times in the
    past, with no response.
    Is it possible for me to join this group?
Regards,
    Andrew
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--------------338202019CEB535D6EE10912
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=windows-1252"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">On 04/12/2017 01:45 AM, Precht, Andrew
      wrote:<br>
    </div>
    <blockquote
cite="mid:BY1PR09MB0824E15BF3EEC19B5B776EB5F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
      type="cite">
      <meta http-equiv="Content-Type" content="text/html;
        charset=windows-1252">
      <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
        dir="ltr">
        <div>Here is an update</div>
        <div><br>
        </div>
        <div>I checked the
          /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the node
          that had the trouble volume (test1). I didnt see any errors.
          So, I ran a tail -f on the log as I tried to remove the volume
          using the web UI. here is what was appended:</div>
        <div><br>
        </div>
        <div>[2017-04-11 19:48:40.756360] I [MSGID: 106487]
          [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends]
          0-glusterd: Received cli list req</div>
        <div>[2017-04-11 19:48:42.238840] I [MSGID: 106488]
          [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume]
          0-management: Received get vol req</div>
        <div>The message "I [MSGID: 106487]
          [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends]
          0-glusterd: Received cli list req" repeated 6 times between
          [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536]</div>
        <div>The message "I [MSGID: 106488]
          [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume]
          0-management: Received get vol req" repeated 20 times between
          [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179]</div>
        <div>[2017-04-11 19:51:41.556077] I [MSGID: 106487]
          [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends]
          0-glusterd: Received cli list req</div>
        <div><br>
        </div>
        <div>Im seeing that the timestamps on these log entries do not
          match the time on the node.</div>
      </div>
    </blockquote>
    gluster logs are in UTC format. That is the reason you might be
    seeing a different timestamp on your node and in the gluster logs.<br>
    <blockquote
cite="mid:BY1PR09MB0824E15BF3EEC19B5B776EB5F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
      type="cite">
      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
        dir="ltr">
        <div><br>
        </div>
        <div>The next steps</div>
        <div>I stopped the glusterd service on the node with volume
          test1</div>
        <div>I deleted it with:  rm -rf /var/lib/glusterd/vols/test1</div>
        <div>I started the glusterd service.</div>
        <div><br>
        </div>
        <div>After starting the gluster service back up, the directory
          /var/lib/glusterd/vols/test1 reappears.</div>
        <div>Im guessing syncing with the other nodes?</div>
      </div>
    </blockquote>
    yes, since you deleted it only one one node.<br>
    <blockquote
cite="mid:BY1PR09MB0824E15BF3EEC19B5B776EB5F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
      type="cite">
      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
        dir="ltr">
        <div>Is this because I have the Volume Option: auth allow *</div>
        <div>Do I need to remove the directory
          /var/lib/glusterd/vols/test1 on all nodes in the cluster
          individually?</div>
      </div>
    </blockquote>
    you need to remove the file /var/lib/glusterd/vols/test1 on all
    nodes and restart glusterd service on all the nodes in the cluster.<br>
    <blockquote
cite="mid:BY1PR09MB0824E15BF3EEC19B5B776EB5F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
      type="cite">
      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
        dir="ltr">
        <div><br>
        </div>
        <div>thanks</div>
        <br>
      </div>
      <hr style="display:inline-block;width:98%" tabindex="-1">
      <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
          color="#000000" face="Calibri, sans-serif"><b>From:</b> knarra
          <a class="moz-txt-link-rfc2396E" href="mailto:knarra@redhat.com"><knarra@redhat.com></a><br>
          <b>Sent:</b> Tuesday, April 11, 2017 11:51:18 AM<br>
          <b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal
          Nisan; Allon Mureinik; Nir Soffer<br>
          <b>Cc:</b> users<br>
          <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting
          a test gluster volume</font>
        <div> </div>
      </div>
      <div>
        <div class="moz-cite-prefix">On 04/11/2017 11:28 PM, Precht,
          Andrew wrote:<br>
        </div>
        <blockquote
cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
          type="cite">
          <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
          <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
            dir="ltr">
            <div>Hi all,</div>
            <div>The node is oVirt Node 4.1.1 with
              glusterfs-3.8.10-1.el7. </div>
            <div>On the node I can not find
              /var/log/glusterfs/glusterd.log However, there is a
              /var/log/glusterfs/glustershd.log
              <br>
            </div>
          </div>
        </blockquote>
        can you check if
        /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? if
        yes, can you check if there is any error present in that file ?<span
          style="color: rgb(84, 84, 84); font-family: arial, sans-serif;
          font-size: small; font-style: normal; font-variant-ligatures:
          normal; font-variant-caps: normal; font-weight: normal;
          letter-spacing: normal; orphans: 2; text-align: left;
          text-indent: 0px; text-transform: none; white-space: normal;
          widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;
          background-color: rgb(255, 255, 255); display: inline
          !important; float: none;"></span>
        <blockquote
cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
          type="cite">
          <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
            dir="ltr">
            <div><br>
            </div>
            <div>What happens if I follow the four steps outlined here
              to remove the volume from the node
              <u>BUT</u>, I do have another volume present in the
              cluster. It too is a test volume. Neither one has any data
              on them. So, data loss is not an issue.</div>
          </div>
        </blockquote>
        Running those four steps will remove the volume from your
        cluster . If the volumes what you have are test volumes you
        could just follow the steps outlined to delete them (since you
        are not able to delete from UI) and bring back the cluster into
        a normal state.<br>
        <blockquote
cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
          type="cite">
          <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
            dir="ltr">
            <br>
          </div>
          <hr style="display:inline-block;width:98%" tabindex="-1">
          <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
              color="#000000" face="Calibri, sans-serif"><b>From:</b>
              knarra
              <a moz-do-not-send="true" class="moz-txt-link-rfc2396E"
                href="mailto:knarra@redhat.com"><knarra@redhat.com></a><br>
              <b>Sent:</b> Tuesday, April 11, 2017 10:32:27 AM<br>
              <b>To:</b> Sandro Bonazzola; Precht, Andrew; Sahina Bose;
              Tal Nisan; Allon Mureinik; Nir Soffer<br>
              <b>Cc:</b> users<br>
              <b>Subject:</b> Re: [ovirt-users] Im having trouble
              deleting a test gluster volume</font>
            <div> </div>
          </div>
          <div>
            <div class="moz-cite-prefix">On 04/11/2017 10:44 PM, Sandro
              Bonazzola wrote:<br>
            </div>
            <blockquote
cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com"
              type="cite">
              <div dir="auto">Adding some people</div>
              <div class="gmail_extra"><br>
                <div class="gmail_quote">Il 11/Apr/2017 19:06, "Precht,
                  Andrew" <<a moz-do-not-send="true"
                    href="mailto:Andrew.Precht@sjlibrary.org">Andrew.Precht@sjlibrary.org</a>>
                  ha scritto:<br type="attribution">
                  <blockquote class="gmail_quote" style="margin:0 0 0
                    .8ex;border-left:1px #ccc solid;padding-left:1ex">
                    <div dir="ltr">
                      <div
                        id="m_7567123270890452842divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
                        dir="ltr">
                        <div>Hi Ovirt users,</div>
                        <div>Im a newbie to oVirt and Im having
                          trouble deleting a test gluster volume. The
                          nodes are 4.1.1 and the engine is 4.1.0</div>
                        <div><br>
                        </div>
                        <div>When I try to remove the test volume, I
                          click Remove, the dialog box prompting to
                          confirm the deletion pops up and after I click
                          OK, the dialog box changes to show a little
                          spinning wheel and then it disappears. In the
                          end the volume is still there.</div>
                      </div>
                    </div>
                  </blockquote>
                </div>
              </div>
            </blockquote>
            with the latest version of glusterfs & ovirt we do not
            see any issue with deleting a volume. Can you please check
            /var/log/glusterfs/glusterd.log file if there is any error
            present?
            <br>
            <br>
            <br>
            <blockquote
cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com"
              type="cite">
              <div class="gmail_extra">
                <div class="gmail_quote">
                  <blockquote class="gmail_quote" style="margin:0 0 0
                    .8ex;border-left:1px #ccc solid;padding-left:1ex">
                    <div dir="ltr">
                      <div
                        id="m_7567123270890452842divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
                        dir="ltr">
                        The test volume was distributed with two host
                        members. One of the hosts I was able to remove
                        from the volume by removing the host form the
                        cluster. When I try to remove the remaining host
                        in the volume, even with the Force Remove box
                        ticked, I get this response: Cannot remove Host.
                        Server having Gluster volume.
                        <div><br>
                        </div>
                        <div>What to try next?</div>
                      </div>
                    </div>
                  </blockquote>
                </div>
              </div>
            </blockquote>
            since you have already removed the volume from one host in
            the cluster and you still see it on another host you can do
            the following to remove the volume from another host.<br>
            <br>
            1) Login to the host where the volume is present.<br>
            2) cd to /var/lib/glusterd/vols<br>
            3) rm -rf <vol_name><br>
            4) Restart glusterd on that  host.<br>
            <br>
            And before doing the above make sure that you do not have
            any other volume present in the cluster.<br>
            <br>
            Above steps should not be run on a production system as you
            might loose the volume and data.
            <br>
            <br>
            Now removing the host from UI should succed.<br>
            <br>
            <blockquote
cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com"
              type="cite">
              <div class="gmail_extra">
                <div class="gmail_quote">
                  <blockquote class="gmail_quote" style="margin:0 0 0
                    .8ex;border-left:1px #ccc solid;padding-left:1ex">
                    <div dir="ltr">
                      <div
                        id="m_7567123270890452842divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
                        dir="ltr">
                        <div><br>
                        </div>
                        <div>P.S. Ive tried to join this user group
                          several times in the past, with no response.</div>
                        <div>Is it possible for me to join this group?</div>
                        <div><br>
                        </div>
                        <div>Regards,</div>
                        <div>Andrew</div>
                        <br>
                      </div>
                    </div>
                  </blockquote>
                </div>
              </div>
              <br>
              <fieldset class="mimeAttachmentHeader"></fieldset>
              <br>
              <pre wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
            </blockquote>
            <p><br>
            </p>
          </div>
        </blockquote>
        <p><br>
        </p>
      </div>
    </blockquote>
    <p><br>
    </p>
  </body>
</html>

--------------338202019CEB535D6EE10912--