Re: [ovirt-users] I’m having trouble deleting a test gluster volume

Wednesday, 12 April 2017

This is a multi-part message in MIME format.
--------------47436FFB8B4C5BD03AEEA978
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit

On 04/12/2017 08:45 PM, Precht, Andrew wrote:
...

 Hi all,

 You asked: Any errors in ovirt-engine.log file ?

 Yes, In the engine.log this error is repeated about every 3 minutes:

 2017-04-12 07:16:12,554-07 ERROR 
 [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob] 
 (DefaultQuartzScheduler3) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Error 
 updating tasks from CLI: 
 org.ovirt.engine.core.common.errors.EngineException: EngineException: 
 Command execution failederror: Error : Request timed outreturn code: 1 
 (Failed with error GlusterVolumeStatusAllFailedException and code 
 4161)error: Error : Request timed out
 I am not sure why this says "Request timed out".
...

 1) gluster volume list ->  Still shows the deleted volume (test1)

 2) gluster peer status -> Shows one of the peers twice with different 
 uuids:

 Hostname: 192.168.10.109Uuid: 
 42fbb7de-8e6f-4159-a601-3f858fa65f6cState: Peer in Cluster 
 (Connected)Hostname: 192.168.10.109Uuid: 
 e058babe-7f9d-49fe-a3ea-ccdc98d7e5b5State: Peer in Cluster (Connected)
 How did this happen? Are the hostname same for two hosts ?
...

 I tried a gluster volume stop test1, with this result: volume stop: 
 test1: failed: Another transaction is in progress for test1. Please 
 try again after sometime.
 can you restart glusterd and try to stop and delete the volume?
...

 The etc-glusterfs-glusterd.vol.log shows no activity triggered by 
 trying to remove the test1 volume from the UI.

 The ovirt-engine.log shows this repeating many times, when trying to 
 remove the test1 volume from the UI:

 2017-04-12 07:57:38,049-07 INFO 
  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] 
 (DefaultQuartzScheduler9) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] 
 Failed to acquire lock and wait lock 
 'EngineLock:{exclusiveLocks='[b0e1b909-9a6a-49dc-8e20-3a027218f7e1=<GLUSTER, 
 ACTION_TYPE_FAILED_GLUSTER_OPERATION_INPROGRESS>]',
sharedLocks='null'}'
 can you restart ovirt-engine service because i see that "failed to 
acquire lock".  Once ovirt-engine is restarted some one who is holding 
the lock should be release  and things should work fine.

Last but not least, if none of the above works:

Login to all your nodes in the cluster.
rm -rf /var/lib/glusterd/vols/*
rm -rf /var/lib/glusterd/peers/*
systemctl restart glusterd on all the nodes.

Login to UI and see if any volumes / hosts are present. If yes, remove them.

This should clear things for you and you can start from basic.

...
 Thanks much,

 Andrew

 ------------------------------------------------------------------------
 *From:* knarra <knarra(a)redhat.com&gt;
 *Sent:* Tuesday, April 11, 2017 11:10:04 PM
 *To:* Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon 
 Mureinik; Nir Soffer
 *Cc:* users
 *Subject:* Re: [ovirt-users] Im having trouble deleting a test 
 gluster volume
 On 04/12/2017 03:35 AM, Precht, Andrew wrote:
>
> I just noticed this in the Alerts tab: Detected deletion of volume 
> test1 on cluster 8000-1, and deleted it from engine DB.
>
> Yet, It still shows in the web UI?
>
 Any errors in ovirt-engine.log file ? if the volume is deleted from db 
 ideally it should be deleted from UI too.  Can you go to gluster nodes 
 and check for the following:

 1) gluster volume list -> should not return anything since you have 
 deleted the volumes.

 2) gluster peer status -> on all the nodes should show that all the 
 peers are in connected state.

 can you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster log 
 and capture the error messages when you try deleting the volume from UI?

 Log what you have pasted in the previous mail only gives info and i 
 could not get any details from that on why volume delete is failing

> ------------------------------------------------------------------------
> *From:* Precht, Andrew
> *Sent:* Tuesday, April 11, 2017 2:39:31 PM
> *To:* knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon 
> Mureinik; Nir Soffer
> *Cc:* users
> *Subject:* Re: [ovirt-users] Im having trouble deleting a test 
> gluster volume
> The plot thickens
> I put all hosts in the cluster into maintenance mode, with the Stop 
> Gluster service checkbox checked. I then deleted the 
> /var/lib/glusterd/vols/test1 directory on all hosts. I then took the 
> host that the test1 volume was on out of maintenance mode. Then I 
> tried to remove the test1 volume from within the web UI. With no 
> luck, I got the message: Could not delete Gluster Volume test1 on 
> cluster 8000-1.
>
> I went back and checked all host for the test1 directory, it is not 
> on any host. Yet I still cant remove it
>
> Any suggestions?
>
> ------------------------------------------------------------------------
> *From:* Precht, Andrew
> *Sent:* Tuesday, April 11, 2017 1:15:22 PM
> *To:* knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon 
> Mureinik; Nir Soffer
> *Cc:* users
> *Subject:* Re: [ovirt-users] Im having trouble deleting a test 
> gluster volume
> Here is an update
>
> I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on 
> the node that had the trouble volume (test1). I didnt see any 
> errors. So, I ran a tail -f on the log as I tried to remove the 
> volume using the web UI. here is what was appended:
>
> [2017-04-11 19:48:40.756360] I [MSGID: 106487] 
> [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 
> 0-glusterd: Received cli list req
> [2017-04-11 19:48:42.238840] I [MSGID: 106488] 
> [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 
> 0-management: Received get vol req
> The message "I [MSGID: 106487] 
> [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 
> 0-glusterd: Received cli list req" repeated 6 times between 
> [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536]
> The message "I [MSGID: 106488] 
> [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 
> 0-management: Received get vol req" repeated 20 times between 
> [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179]
> [2017-04-11 19:51:41.556077] I [MSGID: 106487] 
> [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 
> 0-glusterd: Received cli list req
>
> Im seeing that the timestamps on these log entries do not match the 
> time on the node.
>
> The next steps
> I stopped the glusterd service on the node with volume test1
> I deleted it with:  rm -rf /var/lib/glusterd/vols/test1
> I started the glusterd service.
>
> After starting the gluster service back up, the directory 
> /var/lib/glusterd/vols/test1 reappears.
> Im guessing syncing with the other nodes?
> Is this because I have the Volume Option: auth allow *
> Do I need to remove the directory /var/lib/glusterd/vols/test1 on all 
> nodes in the cluster individually?
>
> thanks
>
> ------------------------------------------------------------------------
> *From:* knarra <knarra(a)redhat.com&gt;
> *Sent:* Tuesday, April 11, 2017 11:51:18 AM
> *To:* Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon 
> Mureinik; Nir Soffer
> *Cc:* users
> *Subject:* Re: [ovirt-users] Im having trouble deleting a test 
> gluster volume
> On 04/11/2017 11:28 PM, Precht, Andrew wrote:
>> Hi all,
>> The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7.
>> On the node I can not find /var/log/glusterfs/glusterd.log However, 
>> there is a /var/log/glusterfs/glustershd.log
> can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log 
> exists? if yes, can you check if there is any error present in that 
> file ?
>>
>> What happens if I follow the four steps outlined here to remove the 
>> volume from the node _BUT_, I do have another volume present in the 
>> cluster. It too is a test volume. Neither one has any data on them. 
>> So, data loss is not an issue.
> Running those four steps will remove the volume from your cluster . 
> If the volumes what you have are test volumes you could just follow 
> the steps outlined to delete them (since you are not able to delete 
> from UI) and bring back the cluster into a normal state.
>>
>> ------------------------------------------------------------------------
>> *From:* knarra <knarra(a)redhat.com&gt;
>> *Sent:* Tuesday, April 11, 2017 10:32:27 AM
>> *To:* Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; 
>> Allon Mureinik; Nir Soffer
>> *Cc:* users
>> *Subject:* Re: [ovirt-users] Im having trouble deleting a test 
>> gluster volume
>> On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:
>>> Adding some people
>>>
>>> Il 11/Apr/2017 19:06, "Precht, Andrew"
<Andrew.Precht(a)sjlibrary.org 
>>> <mailto:Andrew.Precht@sjlibrary.org>> ha scritto:
>>>
>>>     Hi Ovirt users,
>>>     Im a newbie to oVirt and Im having trouble deleting a test
>>>     gluster volume. The nodes are 4.1.1 and the engine is 4.1.0
>>>
>>>     When I try to remove the test volume, I click Remove, the
>>>     dialog box prompting to confirm the deletion pops up and after
>>>     I click OK, the dialog box changes to show a little spinning
>>>     wheel and then it disappears. In the end the volume is still there.
>>>
>> with the latest version of glusterfs & ovirt we do not see any issue 
>> with deleting a volume. Can you please check 
>> /var/log/glusterfs/glusterd.log file if there is any error present?
>>
>>
>>>     The test volume was distributed with two host members. One of
>>>     the hosts I was able to remove from the volume by removing the
>>>     host form the cluster. When I try to remove the remaining host
>>>     in the volume, even with the Force Remove box ticked, I get
>>>     this response: Cannot remove Host. Server having Gluster volume.
>>>
>>>     What to try next?
>>>
>> since you have already removed the volume from one host in the 
>> cluster and you still see it on another host you can do the 
>> following to remove the volume from another host.
>>
>> 1) Login to the host where the volume is present.
>> 2) cd to /var/lib/glusterd/vols
>> 3) rm -rf <vol_name>
>> 4) Restart glusterd on that  host.
>>
>> And before doing the above make sure that you do not have any other 
>> volume present in the cluster.
>>
>> Above steps should not be run on a production system as you might 
>> loose the volume and data.
>>
>> Now removing the host from UI should succed.
>>
>>>
>>>     P.S. Ive tried to join this user group several times in the
>>>     past, with no response.
>>>     Is it possible for me to join this group?
>>>
>>>     Regards,
>>>     Andrew
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>

--------------47436FFB8B4C5BD03AEEA978
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=windows-1252"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">On 04/12/2017 08:45 PM, Precht, Andrew
      wrote:<br>
    </div>
    <blockquote
cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com"
      type="cite">
      <meta http-equiv="Content-Type" content="text/html;
        charset=windows-1252">
      <style type="text/css" style="display:none;"><!-- P
{margin-top:0;margin-bottom:0;} --></style>
      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
        dir="ltr">
        <p>Hi all,</p>
        <p><span

id="docs-internal-guid-44c81590-62b4-32c0-d6d1-82b627328ebb"></span></p>
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align:
baseline; white-space: pre-wrap;">You asked: Any errors in ovirt-engine.log file
?</span></p>
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:
          36pt;">
          <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33,
33); vertical-align: baseline; white-space: pre-wrap;">Yes, In the engine.log this
error is repeated about every 3 minutes:</span></p>
        <br>
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align:
baseline; white-space: pre-wrap;">2017-04-12 07:16:12,554-07 ERROR
[org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob]
 (DefaultQuartzScheduler3) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Error updating tasks
from CLI: org.ovirt.engine.core.common.errors.EngineException: EngineException: Command
execution failed</span><span style="font-size: 11.5pt; font-family: Arial;
color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">

</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33,
33, 33); vertical-align: baseline; white-space: pre-wrap;">error: Error : Request
timed out</span><span style="font-size: 11.5pt; font-family: Arial; color:
rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">

</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33,
33, 33); vertical-align: baseline; white-space: pre-wrap;">return code: 1 (Failed
with error GlusterVolumeStatusAllFailedException and code 4161)</span><span
style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align:
baseline; white-space: pre-wrap;">

</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33,
33, 33); vertical-align: baseline; white-space: pre-wrap;">error: Error : Request
timed out</span></p>
      </div>
    </blockquote>
    I am not sure why this says "Request timed out".<br>
    <blockquote
cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com"
      type="cite">
      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
        dir="ltr">
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:
          36pt;">
          <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33,
33); vertical-align: baseline; white-space: pre-wrap;">

</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33,
33, 33); vertical-align: baseline; white-space: pre-wrap;">1) gluster volume list
-&gt;  Still shows the deleted volume (test1)</span></p>
      </div>
    </blockquote>
    <blockquote
cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com"
      type="cite">
      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
        dir="ltr">
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:
          36pt;">
          <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33,
33); vertical-align: baseline; white-space: pre-wrap;">

</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33,
33, 33); vertical-align: baseline; white-space: pre-wrap;">2) gluster peer status
-&gt; Shows one of the peers twice with different uuids:</span></p>
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align:
baseline; white-space: pre-wrap;">Hostname: 192.168.10.109</span><span
style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align:
baseline; white-space: pre-wrap;">

</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33,
33, 33); vertical-align: baseline; white-space: pre-wrap;">Uuid:
42fbb7de-8e6f-4159-a601-3f858fa65f6c</span><span style="font-size: 11.5pt;
font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space:
pre-wrap;">

</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33,
33, 33); vertical-align: baseline; white-space: pre-wrap;">State: Peer in Cluster
(Connected)</span><span style="font-size: 11.5pt; font-family: Arial; color:
rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">

</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33,
33, 33); vertical-align: baseline; white-space: pre-wrap;">

</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33,
33, 33); vertical-align: baseline; white-space: pre-wrap;">Hostname:
192.168.10.109</span><span style="font-size: 11.5pt; font-family: Arial;
color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">

</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33,
33, 33); vertical-align: baseline; white-space: pre-wrap;">Uuid:
e058babe-7f9d-49fe-a3ea-ccdc98d7e5b5</span><span style="font-size: 11.5pt;
font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space:
pre-wrap;">

</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33,
33, 33); vertical-align: baseline; white-space: pre-wrap;">State: Peer in Cluster
(Connected)</span></p>
      </div>
    </blockquote>
    How did this happen? Are the hostname same for two hosts ?<br>
    <blockquote
cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com"
      type="cite">
      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
        dir="ltr">
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align:
baseline; white-space: pre-wrap;">

</span></p>
        <p dir="ltr"

style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
            style="vertical-align: baseline;"><font
color="#212121"
              face="Arial"><span style="font-size: 11.5pt;
white-space: pre-wrap;">I tried a gluster volume stop test1, with this
</span><span style="font-size: 15.3333px; white-space:
pre-wrap;">result</span><span style="font-size: 11.5pt; white-space:
pre-wrap;">:
<span>volume stop: test1: failed: Another transaction is in progress for test1.
Please try again after
sometime.</span></span></font></span></p>
      </div>
    </blockquote>
    <font color="#212121"><font face="Arial">can you
restart glusterd
        and try to stop and delete the volume? </font></font><br>
    <blockquote
cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com"
      type="cite">
      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
        dir="ltr">
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt; white-space:
pre-wrap; text-indent: 36pt;">

</span></p>
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt; white-space:
pre-wrap; text-indent: 36pt;">

</span></p>
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align:
baseline; white-space: pre-wrap;"><span></span></span><span
style="color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt; white-space:
pre-wrap; text-indent: 36pt;">The
 etc-glusterfs-glusterd.vol.log shows no activity triggered by trying to remove the test1
volume from the UI</span><span style="color: rgb(33, 33, 33); font-family:
Arial; font-size: 11.5pt; white-space: pre-wrap; text-indent:
36pt;">.</span></p>
        <br>
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align:
baseline; white-space: pre-wrap;">The ovirt-engine.log shows this repeating many
times, when
<span style="color: rgb(33, 33, 33); font-family: Arial; font-size: 15.3333px;
text-indent: 48px; white-space: pre-wrap;">
trying to remove the test1 volume from the UI</span>:</span></p>
        <br>
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align:
baseline; white-space: pre-wrap;">2017-04-12 07:57:38,049-07 INFO
 [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
 (DefaultQuartzScheduler9) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Failed to acquire lock
and wait lock
'EngineLock:{exclusiveLocks='[b0e1b909-9a6a-49dc-8e20-3a027218f7e1=&lt;GLUSTER,
ACTION_TYPE_FAILED_GLUSTER_OPERATION_INPROGRESS&gt;]',
sharedLocks='null'}'</span><span style="font-size: 11.5pt;
font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space:
pre-wrap;">
</span></p>
      </div>
    </blockquote>
    can you restart ovirt-engine service because i see that "failed to
    acquire lock".  Once ovirt-engine is restarted some one who is
    holding the lock should be release  and things should work fine.<br>
    <br>
    Last but not least, if none of the above works:<br>
    <br>
    Login to all your nodes in the cluster.<br>
    rm -rf /var/lib/glusterd/vols/*<br>
    rm -rf /var/lib/glusterd/peers/*<br>
    systemctl restart glusterd on all the nodes.<br>
    <br>
    Login to UI and see if any volumes / hosts are present. If yes,
    remove them.<br>
    <br>
    This should clear things for you and you can start from basic.<br>
    <br>
    <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33);
vertical-align: baseline; white-space: pre-wrap;"></span><br
      class="kix-line-break">
    <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33);
vertical-align: baseline; white-space: pre-wrap;"></span>
    <blockquote
cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com"
      type="cite">
      <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
        dir="ltr">
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align:
baseline; white-space: pre-wrap;">
</span></p>
        Thanks much,
        <p>Andrew</p>
      </div>
      <hr style="display:inline-block;width:98%" tabindex="-1">
      <div id="divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt"
          color="#000000" face="Calibri,
sans-serif"><b>From:</b> knarra
          <a class="moz-txt-link-rfc2396E"
href="mailto:knarra@redhat.com">&lt;knarra@redhat.com&gt;</a><br>
          <b>Sent:</b> Tuesday, April 11, 2017 11:10:04 PM<br>
          <b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal
          Nisan; Allon Mureinik; Nir Soffer<br>
          <b>Cc:</b> users<br>
          <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting
          a test gluster volume</font>
        <div> </div>
      </div>
      <div>
        <div class="moz-cite-prefix">On 04/12/2017 03:35 AM, Precht,
          Andrew wrote:<br>
        </div>
        <blockquote
cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
          type="cite">
          <style type="text/css" style="display:none;"><!-- P
{margin-top:0;margin-bottom:0;} --></style>
          <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
            dir="ltr">
            <p>I just <span>noticed </span>this in the Alerts
tab: <span>Detected
                deletion of volume test1 on cluster 8000-1, and deleted
                it from engine DB.</span><br>
            </p>
          </div>
        </blockquote>
        <blockquote
cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
          type="cite">
          <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
            dir="ltr">
            <p><span>Yet, It still shows in the web
UI?</span></p>
          </div>
        </blockquote>
        Any errors in ovirt-engine.log file ? if the volume is deleted
        from db ideally it should be deleted from UI too.  Can you go to
        gluster nodes and check for the following:<br>
        <br>
        1) gluster volume list -&gt; should not return anything since
        you have deleted the volumes.<br>
        <br>
        2) gluster peer status -&gt; on all the nodes should show that
        all the peers are in connected state.<br>
        <br>
        can you tail -f /var/log/ovirt-engine/ovirt-engine.log and
        gluster log and capture the error messages when you try deleting
        the volume from UI?<br>
        <br>
        Log what you have pasted in the previous mail only gives info
        and i could not get any details from that on why volume delete
        is failing<br>
        <br>
        <blockquote
cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
          type="cite">
          <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
            dir="ltr">
          </div>
          <hr style="display:inline-block;width:98%"
tabindex="-1">
          <div id="divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt"
              color="#000000" face="Calibri,
sans-serif"><b>From:</b>
              Precht, Andrew<br>
              <b>Sent:</b> Tuesday, April 11, 2017 2:39:31 PM<br>
              <b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal
              Nisan; Allon Mureinik; Nir Soffer<br>
              <b>Cc:</b> users<br>
              <b>Subject:</b> Re: [ovirt-users] Im having trouble
              deleting a test gluster volume</font>
            <div> </div>
          </div>
          <div>
            <style type="text/css" style="display:none;"><!--
P {margin-top:0;margin-bottom:0;} --></style>
            <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
              dir="ltr">
              <div>The plot thickens</div>
              <div>I put all hosts in the cluster into maintenance mode,
                with the Stop Gluster service checkbox checked. I then
                deleted the /var/lib/glusterd/vols/test1 directory on
                all hosts. I then took the host that the test1 volume
                was on out of maintenance mode. Then I tried to remove
                the test1 volume from within the web UI. With no luck, I
                got the message: Could not delete Gluster Volume test1
                on cluster 8000-1.</div>
              <div><br>
              </div>
              <div>I went back and checked all host for the test1
                directory, it is not on any host. Yet I still cant
                remove it</div>
              <div><br>
              </div>
              <div>Any suggestions?</div>
              <br>
            </div>
            <hr style="display:inline-block;width:98%"
tabindex="-1">
            <div id="divRplyFwdMsg" dir="ltr"><font
                style="font-size:11pt" color="#000000"
face="Calibri,
                sans-serif"><b>From:</b> Precht, Andrew<br>
                <b>Sent:</b> Tuesday, April 11, 2017 1:15:22 PM<br>
                <b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal
                Nisan; Allon Mureinik; Nir Soffer<br>
                <b>Cc:</b> users<br>
                <b>Subject:</b> Re: [ovirt-users] Im having trouble
                deleting a test gluster volume</font>
              <div> </div>
            </div>
            <div>
              <style type="text/css"
style="display:none;"><!-- P {margin-top:0;margin-bottom:0;}
--></style>
              <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
                dir="ltr">
                <div>Here is an update</div>
                <div><br>
                </div>
                <div>I checked the
                  /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on
                  the node that had the trouble volume (test1). I didnt
                  see any errors. So, I ran a tail -f on the log as I
                  tried to remove the volume using the web UI. here is
                  what was appended:</div>
                <div><br>
                </div>
                <div>[2017-04-11 19:48:40.756360] I [MSGID: 106487]
                  [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends]
                  0-glusterd: Received cli list req</div>
                <div>[2017-04-11 19:48:42.238840] I [MSGID: 106488]
                  [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume]
                  0-management: Received get vol req</div>
                <div>The message "I [MSGID: 106487]
                  [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends]
                  0-glusterd: Received cli list req" repeated 6 times
                  between [2017-04-11 19:48:40.756360] and [2017-04-11
                  19:49:32.596536]</div>
                <div>The message "I [MSGID: 106488]
                  [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume]
                  0-management: Received get vol req" repeated 20 times
                  between [2017-04-11 19:48:42.238840] and [2017-04-11
                  19:49:34.082179]</div>
                <div>[2017-04-11 19:51:41.556077] I [MSGID: 106487]
                  [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends]
                  0-glusterd: Received cli list req</div>
                <div><br>
                </div>
                <div>Im seeing that the timestamps on these log entries
                  do not match the time on the node.</div>
                <div><br>
                </div>
                <div>The next steps</div>
                <div>I stopped the glusterd service on the node with
                  volume test1</div>
                <div>I deleted it with:  rm -rf
                  /var/lib/glusterd/vols/test1</div>
                <div>I started the glusterd service.</div>
                <div><br>
                </div>
                <div>After starting the gluster service back up, the
                  directory /var/lib/glusterd/vols/test1 reappears.</div>
                <div>Im guessing syncing with the other nodes?</div>
                <div>Is this because I have the Volume Option: auth
                  allow *</div>
                <div>Do I need to remove the directory
                  /var/lib/glusterd/vols/test1 on all nodes in the
                  cluster individually?</div>
                <div><br>
                </div>
                <div>thanks</div>
                <br>
              </div>
              <hr style="display:inline-block;width:98%"
tabindex="-1">
              <div id="divRplyFwdMsg" dir="ltr"><font
                  style="font-size:11pt" color="#000000"
face="Calibri,
                  sans-serif"><b>From:</b> knarra
                  <a moz-do-not-send="true"
                    class="moz-txt-link-rfc2396E"

href="mailto:knarra@redhat.com">&lt;knarra@redhat.com&gt;</a><br>
                  <b>Sent:</b> Tuesday, April 11, 2017 11:51:18 AM<br>
                  <b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina
                  Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br>
                  <b>Cc:</b> users<br>
                  <b>Subject:</b> Re: [ovirt-users] Im having trouble
                  deleting a test gluster volume</font>
                <div> </div>
              </div>
              <div>
                <div class="moz-cite-prefix">On 04/11/2017 11:28 PM,
                  Precht, Andrew wrote:<br>
                </div>
                <blockquote
cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
                  type="cite">
                  <style type="text/css"
style="display:none;"><!-- P {margin-top:0;margin-bottom:0;}
--></style>
                  <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
                    dir="ltr">
                    <div>Hi all,</div>
                    <div>The node is oVirt Node 4.1.1 with
                      glusterfs-3.8.10-1.el7. </div>
                    <div>On the node I can not find
                      /var/log/glusterfs/glusterd.log However, there is
                      a /var/log/glusterfs/glustershd.log
                      <br>
                    </div>
                  </div>
                </blockquote>
                can you check if
                /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
                exists? if yes, can you check if there is any error
                present in that file ?<span style="color: rgb(84, 84,
                  84); font-family: arial, sans-serif; font-size: small;
                  font-style: normal; font-variant-ligatures: normal;
                  font-variant-caps: normal; font-weight: normal;
                  letter-spacing: normal; orphans: 2; text-align: left;
                  text-indent: 0px; text-transform: none; white-space:
                  normal; widows: 2; word-spacing: 0px;
                  -webkit-text-stroke-width: 0px; background-color:
                  rgb(255, 255, 255); display: inline !important; float:
                  none;"></span>
                <blockquote
cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
                  type="cite">
                  <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
                    dir="ltr">
                    <div><br>
                    </div>
                    <div>What happens if I follow the four steps
                      outlined here to remove the volume from the node
                      <u>BUT</u>, I do have another volume present in
                      the cluster. It too is a test volume. Neither one
                      has any data on them. So, data loss is not an
                      issue.</div>
                  </div>
                </blockquote>
                Running those four steps will remove the volume from
                your cluster . If the volumes what you have are test
                volumes you could just follow the steps outlined to
                delete them (since you are not able to delete from UI)
                and bring back the cluster into a normal state.<br>
                <blockquote
cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
                  type="cite">
                  <div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
                    dir="ltr">
                    <br>
                  </div>
                  <hr style="display:inline-block;width:98%"
                    tabindex="-1">
                  <div id="divRplyFwdMsg" dir="ltr"><font
                      style="font-size:11pt" color="#000000"
                      face="Calibri, sans-serif"><b>From:</b>
knarra
                      <a moz-do-not-send="true"
                        class="moz-txt-link-rfc2396E"
                        href="mailto:knarra@redhat.com">
                        &amp;lt;knarra(a)redhat.com&amp;gt;&lt;/a&gt;&lt;br&gt;
                      <b>Sent:</b> Tuesday, April 11, 2017 10:32:27
AM<br>
                      <b>To:</b> Sandro Bonazzola; Precht, Andrew;
                      Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br>
                      <b>Cc:</b> users<br>
                      <b>Subject:</b> Re: [ovirt-users] Im having
                      trouble deleting a test gluster volume</font>
                    <div> </div>
                  </div>
                  <div>
                    <div class="moz-cite-prefix">On 04/11/2017 10:44 PM,
                      Sandro Bonazzola wrote:<br>
                    </div>
                    <blockquote
cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com"
                      type="cite">
                      <div dir="auto">Adding some people</div>
                      <div class="gmail_extra"><br>
                        <div class="gmail_quote">Il 11/Apr/2017 19:06,
                          "Precht, Andrew" &lt;<a
moz-do-not-send="true"

href="mailto:Andrew.Precht@sjlibrary.org">Andrew.Precht@sjlibrary.org</a>&gt;
                          ha scritto:<br type="attribution">
                          <blockquote class="gmail_quote"
                            style="margin:0 0 0 .8ex;border-left:1px
                            #ccc solid;padding-left:1ex">
                            <div dir="ltr">
                              <div
                                id="m_7567123270890452842divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
                                dir="ltr">
                                <div>Hi Ovirt users,</div>
                                <div>Im a newbie to oVirt and Im
                                  having trouble deleting a test gluster
                                  volume. The nodes are 4.1.1 and the
                                  engine is 4.1.0</div>
                                <div><br>
                                </div>
                                <div>When I try to remove the test
                                  volume, I click Remove, the dialog box
                                  prompting to confirm the deletion pops
                                  up and after I click OK, the dialog
                                  box changes to show a little spinning
                                  wheel and then it disappears. In the
                                  end the volume is still there.</div>
                              </div>
                            </div>
                          </blockquote>
                        </div>
                      </div>
                    </blockquote>
                    with the latest version of glusterfs &amp; ovirt we
                    do not see any issue with deleting a volume. Can you
                    please check /var/log/glusterfs/glusterd.log file if
                    there is any error present?
                    <br>
                    <br>
                    <br>
                    <blockquote
cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com"
                      type="cite">
                      <div class="gmail_extra">
                        <div class="gmail_quote">
                          <blockquote class="gmail_quote"
                            style="margin:0 0 0 .8ex;border-left:1px
                            #ccc solid;padding-left:1ex">
                            <div dir="ltr">
                              <div
                                id="m_7567123270890452842divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
                                dir="ltr">
                                The test volume was distributed with two
                                host members. One of the hosts I was
                                able to remove from the volume by
                                removing the host form the cluster. When
                                I try to remove the remaining host in
                                the volume, even with the Force Remove
                                box ticked, I get this response: Cannot
                                remove Host. Server having Gluster
                                volume.
                                <div><br>
                                </div>
                                <div>What to try next?</div>
                              </div>
                            </div>
                          </blockquote>
                        </div>
                      </div>
                    </blockquote>
                    since you have already removed the volume from one
                    host in the cluster and you still see it on another
                    host you can do the following to remove the volume
                    from another host.<br>
                    <br>
                    1) Login to the host where the volume is present.<br>
                    2) cd to /var/lib/glusterd/vols<br>
                    3) rm -rf &lt;vol_name&gt;<br>
                    4) Restart glusterd on that  host.<br>
                    <br>
                    And before doing the above make sure that you do not
                    have any other volume present in the cluster.<br>
                    <br>
                    Above steps should not be run on a production system
                    as you might loose the volume and data.
                    <br>
                    <br>
                    Now removing the host from UI should succed.<br>
                    <br>
                    <blockquote
cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com"
                      type="cite">
                      <div class="gmail_extra">
                        <div class="gmail_quote">
                          <blockquote class="gmail_quote"
                            style="margin:0 0 0 .8ex;border-left:1px
                            #ccc solid;padding-left:1ex">
                            <div dir="ltr">
                              <div
                                id="m_7567123270890452842divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
                                dir="ltr">
                                <div><br>
                                </div>
                                <div>P.S. Ive tried to join this user
                                  group several times in the past, with
                                  no response.</div>
                                <div>Is it possible for me to join this
                                  group?</div>
                                <div><br>
                                </div>
                                <div>Regards,</div>
                                <div>Andrew</div>
                                <br>
                              </div>
                            </div>
                          </blockquote>
                        </div>
                      </div>
                      <br>
                      <fieldset
class="mimeAttachmentHeader"></fieldset>
                      <br>
                      <pre
wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
                    </blockquote>
                    <p><br>
                    </p>
                  </div>
                </blockquote>
                <p><br>
                </p>
              </div>
            </div>
          </div>
        </blockquote>
        <p><br>
        </p>
      </div>
    </blockquote>
    <p><br>
    </p>
  </body>
</html>

--------------47436FFB8B4C5BD03AEEA978--

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [ovirt-users] I’m having trouble deleting a test gluster volume