Re: [ovirt-users] I’m having trouble deleting a test gluster volume

--_000_BY1PR09MB0824A056B7185A98CDF374C7F5030BY1PR09MB0824namp_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

Hi all,

You asked: Any errors in ovirt-engine.log file ?

Yes, In the engine.log this error is repeated about every 3 minutes:


2017-04-12 07:16:12,554-07 ERROR [org.ovirt.engine.core.bll.gluster.Gluster=
TasksSyncJob] (DefaultQuartzScheduler3) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7=
b8] Error updating tasks from CLI: org.ovirt.engine.core.common.errors.Engi=
neException: EngineException: Command execution failed
error: Error : Request timed out
return code: 1 (Failed with error GlusterVolumeStatusAllFailedException and=
 code 4161)
error: Error : Request timed out

1) gluster volume list ->  Still shows the deleted volume (test1)

2) gluster peer status -> Shows one of the peers twice with different uuid=
=92s:

Hostname: 192.168.10.109
Uuid: 42fbb7de-8e6f-4159-a601-3f858fa65f6c
State: Peer in Cluster (Connected)

Hostname: 192.168.10.109
Uuid: e058babe-7f9d-49fe-a3ea-ccdc98d7e5b5
State: Peer in Cluster (Connected)


I tried a gluster volume stop test1, with this result: volume stop: test1: =
failed: Another transaction is in progress for test1. Please try again afte=
r sometime.



The etc-glusterfs-glusterd.vol.log shows no activity triggered by trying to=
 remove the test1 volume from the UI.


The ovirt-engine.log shows this repeating many times, when trying to remove=
 the test1 volume from the UI:


2017-04-12 07:57:38,049-07 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLo=
ckManager] (DefaultQuartzScheduler9) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8]=
 Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks=3D'[b0e1b=
909-9a6a-49dc-8e20-3a027218f7e1=3D<GLUSTER, ACTION_TYPE_FAILED_GLUSTER_OPER=
ATION_INPROGRESS>]', sharedLocks=3D'null'}'


Thanks much,

Andrew

________________________________
From: knarra <knarra@redhat.com>
Sent: Tuesday, April 11, 2017 11:10:04 PM
To: Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureini=
k; Nir Soffer
Cc: users
Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol=
ume

On 04/12/2017 03:35 AM, Precht, Andrew wrote:

I just noticed this in the Alerts tab: Detected deletion of volume test1 on=
 cluster 8000-1, and deleted it from engine DB.

Yet, It still shows in the web UI?

Any errors in ovirt-engine.log file ? if the volume is deleted from db idea=
lly it should be deleted from UI too.  Can you go to gluster nodes and chec=
k for the following:

1) gluster volume list -> should not return anything since you have deleted=
 the volumes.

2) gluster peer status -> on all the nodes should show that all the peers a=
re in connected state.

can you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster log and =
capture the error messages when you try deleting the volume from UI?

Log what you have pasted in the previous mail only gives info and i could n=
ot get any details from that on why volume delete is failing

________________________________
From: Precht, Andrew
Sent: Tuesday, April 11, 2017 2:39:31 PM
To: knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir S=
offer
Cc: users
Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol=
ume

The plot thickens=85
I put all hosts in the cluster into maintenance mode, with the Stop Gluster=
 service checkbox checked. I then deleted the /var/lib/glusterd/vols/test1 =
directory on all hosts. I then took the host that the test1 volume was on o=
ut of maintenance mode. Then I tried to remove the test1 volume from within=
 the web UI. With no luck, I got the message: Could not delete Gluster Volu=
me test1 on cluster 8000-1.

I went back and checked all host for the test1 directory, it is not on any =
host. Yet I still can=92t remove it=85

Any suggestions?

________________________________
From: Precht, Andrew
Sent: Tuesday, April 11, 2017 1:15:22 PM
To: knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir S=
offer
Cc: users
Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol=
ume

Here is an update=85

I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the node=
 that had the trouble volume (test1). I didn=92t see any errors. So, I ran =
a tail -f on the log as I tried to remove the volume using the web UI. here=
 is what was appended:

[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:1474:__g=
lusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:1537:__g=
lusterd_handle_cli_get_volume] 0-management: Received get vol req
The message "I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_c=
li_list_friends] 0-glusterd: Received cli list req" repeated 6 times betwee=
n [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536]
The message "I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_c=
li_get_volume] 0-management: Received get vol req" repeated 20 times betwee=
n [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179]
[2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:1474:__g=
lusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

I=92m seeing that the timestamps on these log entries do not match the time=
 on the node.

The next steps
I stopped the glusterd service on the node with volume test1
I deleted it with:  rm -rf /var/lib/glusterd/vols/test1
I started the glusterd service.

After starting the gluster service back up, the directory /var/lib/glusterd=
/vols/test1 reappears.
I=92m guessing syncing with the other nodes?
Is this because I have the Volume Option: auth allow *
Do I need to remove the directory /var/lib/glusterd/vols/test1 on all nodes=
 in the cluster individually?

thanks

________________________________
From: knarra <knarra@redhat.com><mailto:knarra@redhat.com>
Sent: Tuesday, April 11, 2017 11:51:18 AM
To: Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureini=
k; Nir Soffer
Cc: users
Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol=
ume

On 04/11/2017 11:28 PM, Precht, Andrew wrote:
Hi all,
The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7.
On the node I can not find /var/log/glusterfs/glusterd.log However, there i=
s a /var/log/glusterfs/glustershd.log
can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? =
if yes, can you check if there is any error present in that file ?

What happens if I follow the four steps outlined here to remove the volume =
from the node BUT, I do have another volume present in the cluster. It too =
is a test volume. Neither one has any data on them. So, data loss is not an=
 issue.
Running those four steps will remove the volume from your cluster . If the =
volumes what you have are test volumes you could just follow the steps outl=
ined to delete them (since you are not able to delete from UI) and bring ba=
ck the cluster into a normal state.

________________________________
From: knarra <knarra@redhat.com><mailto:knarra@redhat.com>
Sent: Tuesday, April 11, 2017 10:32:27 AM
To: Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureini=
k; Nir Soffer
Cc: users
Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol=
ume

On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:
Adding some people

Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht@sjlibrary.org<mailto:=
Andrew.Precht@sjlibrary.org>> ha scritto:
Hi Ovirt users,
I=92m a newbie to oVirt and I=92m having trouble deleting a test gluster vo=
lume. The nodes are 4.1.1 and the engine is 4.1.0

When I try to remove the test volume, I click Remove, the dialog box prompt=
ing to confirm the deletion pops up and after I click OK, the dialog box ch=
anges to show a little spinning wheel and then it disappears. In the end th=
e volume is still there.
with the latest version of glusterfs & ovirt we do not see any issue with d=
eleting a volume. Can you please check /var/log/glusterfs/glusterd.log file=
 if there is any error present?


The test volume was distributed with two host members. One of the hosts I w=
as able to remove from the volume by removing the host form the cluster. Wh=
en I try to remove the remaining host in the volume, even with the =93Force=
 Remove=94 box ticked, I get this response: Cannot remove Host. Server havi=
ng Gluster volume.

What to try next?
since you have already removed the volume from one host in the cluster and =
you still see it on another host you can do the following to remove the vol=
ume from another host.

1) Login to the host where the volume is present.
2) cd to /var/lib/glusterd/vols
3) rm -rf <vol_name>
4) Restart glusterd on that  host.

And before doing the above make sure that you do not have any other volume =
present in the cluster.

Above steps should not be run on a production system as you might loose the=
 volume and data.

Now removing the host from UI should succed.


P.S. I=92ve tried to join this user group several times in the past, with n=
o response.
Is it possible for me to join this group?

Regards,
Andrew




_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users





--_000_BY1PR09MB0824A056B7185A98CDF374C7F5030BY1PR09MB0824namp_
Content-Type: text/html; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1=
252">
</head>
<body bgcolor=3D"#FFFFFF" text=3D"#000000">
<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style>
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<p>Hi all,</p>
<p><span id=3D"docs-internal-guid-44c81590-62b4-32c0-d6d1-82b627328ebb"></p=
>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;">You asked: Any errors=
 in ovirt-engine.log file ?</span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;t=
ext-indent: 36pt;">
<span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33=
); vertical-align: baseline; white-space: pre-wrap;">Yes, In the engine.log=
 this error is repeated about every 3 minutes:</span></p>
<br>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;">2017-04-12 07:16:12,5=
54-07 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob]
 (DefaultQuartzScheduler3) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Error upd=
ating tasks from CLI: org.ovirt.engine.core.common.errors.EngineException: =
EngineException: Command execution failed</span><span style=3D"font-size: 1=
1.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline=
; white-space: pre-wrap;"><br class=3D"kix-line-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
 33, 33); vertical-align: baseline; white-space: pre-wrap;">error: Error : =
Request timed out</span><span style=3D"font-size: 11.5pt; font-family: Aria=
l; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;=
"><br class=3D"kix-line-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
 33, 33); vertical-align: baseline; white-space: pre-wrap;">return code: 1 =
(Failed with error GlusterVolumeStatusAllFailedException and code 4161)</sp=
an><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33,=
 33); vertical-align: baseline; white-space: pre-wrap;"><br class=3D"kix-li=
ne-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
 33, 33); vertical-align: baseline; white-space: pre-wrap;">error: Error : =
Request timed out</span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;t=
ext-indent: 36pt;">
<span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33=
); vertical-align: baseline; white-space: pre-wrap;"><br class=3D"kix-line-=
break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
 33, 33); vertical-align: baseline; white-space: pre-wrap;">1) gluster volu=
me list ->  Still shows the deleted volume (test1)</span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;t=
ext-indent: 36pt;">
<span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33=
); vertical-align: baseline; white-space: pre-wrap;"><br class=3D"kix-line-=
break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
 33, 33); vertical-align: baseline; white-space: pre-wrap;">2) gluster peer=
 status -> Shows one of the peers twice with different uuid=92s:</span><=
/p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;">Hostname: 192.168.10.=
109</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(=
33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"><br class=3D=
"kix-line-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
 33, 33); vertical-align: baseline; white-space: pre-wrap;">Uuid: 42fbb7de-=
8e6f-4159-a601-3f858fa65f6c</span><span style=3D"font-size: 11.5pt; font-fa=
mily: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space:=
 pre-wrap;"><br class=3D"kix-line-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
 33, 33); vertical-align: baseline; white-space: pre-wrap;">State: Peer in =
Cluster (Connected)</span><span style=3D"font-size: 11.5pt; font-family: Ar=
ial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wra=
p;"><br class=3D"kix-line-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
 33, 33); vertical-align: baseline; white-space: pre-wrap;"><br class=3D"ki=
x-line-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
 33, 33); vertical-align: baseline; white-space: pre-wrap;">Hostname: 192.1=
68.10.109</span><span style=3D"font-size: 11.5pt; font-family: Arial; color=
: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"><br cl=
ass=3D"kix-line-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
 33, 33); vertical-align: baseline; white-space: pre-wrap;">Uuid: e058babe-=
7f9d-49fe-a3ea-ccdc98d7e5b5</span><span style=3D"font-size: 11.5pt; font-fa=
mily: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space:=
 pre-wrap;"><br class=3D"kix-line-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
 33, 33); vertical-align: baseline; white-space: pre-wrap;">State: Peer in =
Cluster (Connected)</span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;"><br>
</span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"vertical-align: baseline;"><font color=3D"#212121" face=3D"=
Arial"><span style=3D"font-size: 11.5pt; white-space: pre-wrap;">I tried a =
gluster volume stop test1, with this
</span><span style=3D"font-size: 15.3333px; white-space: pre-wrap;">result<=
/span><span style=3D"font-size: 11.5pt; white-space: pre-wrap;">:
<span>volume stop: test1: failed: Another transaction is in progress for te=
st1. Please try again after sometime.</span></span></font></span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5=
pt; white-space: pre-wrap; text-indent: 36pt;"><br>
</span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5=
pt; white-space: pre-wrap; text-indent: 36pt;"><br>
</span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;"><span></span></span><=
span style=3D"color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt=
; white-space: pre-wrap; text-indent: 36pt;">The
 etc-glusterfs-glusterd.vol.log shows no activity triggered by trying to re=
move the test1 volume from the UI</span><span style=3D"color: rgb(33, 33, 3=
3); font-family: Arial; font-size: 11.5pt; white-space: pre-wrap; text-inde=
nt: 36pt;">.</span></p>
<br>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;">The ovirt-engine.log =
shows this repeating many times, when
<span style=3D"color: rgb(33, 33, 33); font-family: Arial; font-size: 15.33=
33px; text-indent: 48px; white-space: pre-wrap;">
trying to remove the test1 volume from the UI</span>:</span></p>
<br>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;">2017-04-12 07:57:38,0=
49-07 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
 (DefaultQuartzScheduler9) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Failed to=
 acquire lock and wait lock 'EngineLock:{exclusiveLocks=3D'[b0e1b909-9a6a-4=
9dc-8e20-3a027218f7e1=3D<GLUSTER, ACTION_TYPE_FAILED_GLUSTER_OPERATION_I=
NPROGRESS>]', sharedLocks=3D'null'}'</span><span style=3D"font-size: 11.=
5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; =
white-space: pre-wrap;"><br class=3D"kix-line-break">
<br class=3D"kix-line-break">
</span></p>
</span>Thanks much,
<p></p>
<p>Andrew</p>
</div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" st=
yle=3D"font-size:11pt" color=3D"#000000"><b>From:</b> knarra <knarra@red=
hat.com><br>
<b>Sent:</b> Tuesday, April 11, 2017 11:10:04 PM<br>
<b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon =
Mureinik; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus=
ter volume</font>
<div> </div>
</div>
<div>
<div class=3D"moz-cite-prefix">On 04/12/2017 03:35 AM, Precht, Andrew wrote=
:<br>
</div>
<blockquote cite=3D"mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style>
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<p>I just <span>noticed </span>this in the Alerts tab: <span>Dete=
cted deletion of volume test1 on cluster 8000-1, and deleted it from engine=
 DB.</span><br>
</p>
</div>
</blockquote>
<blockquote cite=3D"mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<p><span>Yet, It still shows in the web UI?</span></p>
</div>
</blockquote>
Any errors in ovirt-engine.log file ? if the volume is deleted from db idea=
lly it should be deleted from UI too.  Can you go to gluster nodes and=
 check for the following:<br>
<br>
1) gluster volume list -> should not return anything since you have dele=
ted the volumes.<br>
<br>
2) gluster peer status -> on all the nodes should show that all the peer=
s are in connected state.<br>
<br>
can you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster log and =
capture the error messages when you try deleting the volume from UI?<br>
<br>
Log what you have pasted in the previous mail only gives info and i could n=
ot get any details from that on why volume delete is failing<br>
<br>
<blockquote cite=3D"mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
</div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color=
=3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> Precht, Andrew<br>
<b>Sent:</b> Tuesday, April 11, 2017 2:39:31 PM<br>
<b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik=
; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus=
ter volume</font>
<div> </div>
</div>
<div><style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;=
margin-bottom:0;} --></style>
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<div>The plot thickens=85</div>
<div>I put all hosts in the cluster into maintenance mode, with the Stop Gl=
uster service checkbox checked. I then deleted the /var/lib/glusterd/vols/t=
est1 directory on all hosts. I then took the host that the test1 volume was=
 on out of maintenance mode. Then
 I tried to remove the test1 volume from within the web UI. With no luck, I=
 got the message: Could not delete Gluster Volume test1 on cluster 8000-1.<=
/div>
<div><br>
</div>
<div>I went back and checked all host for the test1 directory, it is not on=
 any host. Yet I still can=92t remove it=85</div>
<div><br>
</div>
<div>Any suggestions?</div>
<br>
</div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color=
=3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> Precht, Andrew<br>
<b>Sent:</b> Tuesday, April 11, 2017 1:15:22 PM<br>
<b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik=
; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus=
ter volume</font>
<div> </div>
</div>
<div><style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;=
margin-bottom:0;} --></style>
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<div>Here is an update=85</div>
<div><br>
</div>
<div>I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the=
 node that had the trouble volume (test1). I didn=92t see any errors. So, I=
 ran a tail -f on the log as I tried to remove the volume using the web UI.=
 here is what was appended:</div>
<div><br>
</div>
<div>[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:147=
4:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</di=
v>
<div>[2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:153=
7:__glusterd_handle_cli_get_volume] 0-management: Received get vol req</div=
>
<div>The message "I [MSGID: 106487] [glusterd-handler.c:1474:__gluster=
d_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated=
 6 times between [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.5965=
36]</div>
<div>The message "I [MSGID: 106488] [glusterd-handler.c:1537:__gluster=
d_handle_cli_get_volume] 0-management: Received get vol req" repeated =
20 times between [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.0821=
79]</div>
<div>[2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:147=
4:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</di=
v>
<div><br>
</div>
<div>I=92m seeing that the timestamps on these log entries do not match the=
 time on the node.</div>
<div><br>
</div>
<div>The next steps</div>
<div>I stopped the glusterd service on the node with volume test1</div>
<div>I deleted it with:  rm -rf /var/lib/glusterd/vols/test1</div>
<div>I started the glusterd service.</div>
<div><br>
</div>
<div>After starting the gluster service back up, the directory /var/lib/glu=
sterd/vols/test1 reappears.</div>
<div>I=92m guessing syncing with the other nodes?</div>
<div>Is this because I have the Volume Option: auth allow *</div>
<div>Do I need to remove the directory /var/lib/glusterd/vols/test1 on all =
nodes in the cluster individually?</div>
<div><br>
</div>
<div>thanks</div>
<br>
</div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color=
=3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> knarra
<a class=3D"moz-txt-link-rfc2396E" href=3D"mailto:knarra@redhat.com"><kn=
arra@redhat.com></a><br>
<b>Sent:</b> Tuesday, April 11, 2017 11:51:18 AM<br>
<b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon =
Mureinik; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus=
ter volume</font>
<div> </div>
</div>
<div>
<div class=3D"moz-cite-prefix">On 04/11/2017 11:28 PM, Precht, Andrew wrote=
:<br>
</div>
<blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style>
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<div>Hi all,</div>
<div>The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. </div>
<div>On the node I can not find /var/log/glusterfs/glusterd.log However, th=
ere is a /var/log/glusterfs/glustershd.log
<br>
</div>
</div>
</blockquote>
can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? =
if yes, can you check if there is any error present in that file ?<span sty=
le=3D"color: rgb(84, 84, 84); font-family:
              arial, sans-serif; font-size: small; font-style: normal;
              font-variant-ligatures: normal; font-variant-caps: normal;
              font-weight: normal; letter-spacing: normal; orphans: 2;
              text-align: left; text-indent: 0px; text-transform: none;
              white-space: normal; widows: 2; word-spacing: 0px;
              -webkit-text-stroke-width: 0px; background-color: rgb(255,
              255, 255); display: inline !important; float: none;"></span>
<blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<div><br>
</div>
<div>What happens if I follow the four steps outlined here to remove the vo=
lume from the node
<u>BUT</u>, I do have another volume present in the cluster. It too is a te=
st volume. Neither one has any data on them. So, data loss is not an issue.=
</div>
</div>
</blockquote>
Running those four steps will remove the volume from your cluster . If the =
volumes what you have are test volumes you could just follow the steps outl=
ined to delete them (since you are not able to delete from UI) and bring ba=
ck the cluster into a normal state.<br>
<blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<br>
</div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color=
=3D"#000000" face=3D"Calibri,
                  sans-serif"><b>From:</b> knarra
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-rfc2396E" href=3D"mailto:=
knarra@redhat.com">
<knarra@redhat.com></a><br>
<b>Sent:</b> Tuesday, April 11, 2017 10:32:27 AM<br>
<b>To:</b> Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon =
Mureinik; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus=
ter volume</font>
<div> </div>
</div>
<div>
<div class=3D"moz-cite-prefix">On 04/11/2017 10:44 PM, Sandro Bonazzola wro=
te:<br>
</div>
<blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA=
bkkQ@mail.gmail.com" type=3D"cite">
<div dir=3D"auto">Adding some people</div>
<div class=3D"gmail_extra"><br>
<div class=3D"gmail_quote">Il 11/Apr/2017 19:06, "Precht, Andrew"=
 <<a moz-do-not-send=3D"true" href=3D"mailto:Andrew.Precht@sjlibrary.org=
">Andrew.Precht@sjlibrary.org</a>> ha scritto:<br type=3D"attribution">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0
                        0 .8ex;border-left:1px #ccc
                        solid;padding-left:1ex">
<div dir=3D"ltr">
<div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12=
pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr=
">
<div>Hi Ovirt users,</div>
<div>I=92m a newbie to oVirt and I=92m having trouble deleting a test glust=
er volume. The nodes are 4.1.1 and the engine is 4.1.0</div>
<div><br>
</div>
<div>When I try to remove the test volume, I click Remove, the dialog box p=
rompting to confirm the deletion pops up and after I click OK, the dialog b=
ox changes to show a little spinning wheel and then it disappears. In the e=
nd the volume is still there.</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
with the latest version of glusterfs & ovirt we do not see any issue wi=
th deleting a volume. Can you please check /var/log/glusterfs/glusterd.log =
file if there is any error present?
<br>
<br>
<br>
<blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA=
bkkQ@mail.gmail.com" type=3D"cite">
<div class=3D"gmail_extra">
<div class=3D"gmail_quote">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0
                        0 .8ex;border-left:1px #ccc
                        solid;padding-left:1ex">
<div dir=3D"ltr">
<div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12=
pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr=
">
The test volume was distributed with two host members. One of the hosts I w=
as able to remove from the volume by removing the host form the cluster. Wh=
en I try to remove the remaining host in the volume, even with the =93Force=
 Remove=94 box ticked, I get this response:
 Cannot remove Host. Server having Gluster volume.
<div><br>
</div>
<div>What to try next?</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
since you have already removed the volume from one host in the cluster and =
you still see it on another host you can do the following to remove the vol=
ume from another host.<br>
<br>
1) Login to the host where the volume is present.<br>
2) cd to /var/lib/glusterd/vols<br>
3) rm -rf <vol_name><br>
4) Restart glusterd on that  host.<br>
<br>
And before doing the above make sure that you do not have any other volume =
present in the cluster.<br>
<br>
Above steps should not be run on a production system as you might loose the=
 volume and data.
<br>
<br>
Now removing the host from UI should succed.<br>
<br>
<blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA=
bkkQ@mail.gmail.com" type=3D"cite">
<div class=3D"gmail_extra">
<div class=3D"gmail_quote">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0
                        0 .8ex;border-left:1px #ccc
                        solid;padding-left:1ex">
<div dir=3D"ltr">
<div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12=
pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr=
">
<div><br>
</div>
<div>P.S. I=92ve tried to join this user group several times in the past, w=
ith no response.</div>
<div>Is it possible for me to join this group?</div>
<div><br>
</div>
<div>Regards,</div>
<div>Andrew</div>
<br>
</div>
</div>
</blockquote>
</div>
</div>
<br>
<fieldset class=3D"mimeAttachmentHeader"></fieldset> <br>
<pre wrap=3D"">_______________________________________________
Users mailing list
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"mail=
to:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext" href=3D"http://=
lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/list=
info/users</a>
</pre>
</blockquote>
<p><br>
</p>
</div>
</blockquote>
<p><br>
</p>
</div>
</div>
</div>
</blockquote>
<p><br>
</p>
</div>
</body>
</html>

--_000_BY1PR09MB0824A056B7185A98CDF374C7F5030BY1PR09MB0824namp_--