2:10 a.m.
This is a multi-part message in MIME format.
--------------FBCB929546CA281A03A4314D
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit
On 04/12/2017 03:35 AM, Precht, Andrew wrote:
I just noticed this in the Alerts tab: Detected deletion of volume
test1 on cluster 8000-1, and deleted it from engine DB.
Yet, It still shows in the web UI?
Any errors in ovirt-engine.log file ? if the volume is deleted from db
ideally it should be deleted from UI too. Can you go to gluster nodes
and check for the following:
1) gluster volume list -> should not return anything since you have
deleted the volumes.
2) gluster peer status -> on all the nodes should show that all the
peers are in connected state.
can you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster log
and capture the error messages when you try deleting the volume from UI?
Log what you have pasted in the previous mail only gives info and i
could not get any details from that on why volume delete is failing
------------------------------------------------------------------------
*From:* Precht, Andrew
*Sent:* Tuesday, April 11, 2017 2:39:31 PM
*To:* knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon
Mureinik; Nir Soffer
*Cc:* users
*Subject:* Re: [ovirt-users] Im having trouble deleting a test
gluster volume
The plot thickens
I put all hosts in the cluster into maintenance mode, with the Stop
Gluster service checkbox checked. I then deleted the
/var/lib/glusterd/vols/test1 directory on all hosts. I then took the
host that the test1 volume was on out of maintenance mode. Then I
tried to remove the test1 volume from within the web UI. With no luck,
I got the message: Could not delete Gluster Volume test1 on cluster
8000-1.
I went back and checked all host for the test1 directory, it is not on
any host. Yet I still cant remove it
Any suggestions?
------------------------------------------------------------------------
*From:* Precht, Andrew
*Sent:* Tuesday, April 11, 2017 1:15:22 PM
*To:* knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon
Mureinik; Nir Soffer
*Cc:* users
*Subject:* Re: [ovirt-users] Im having trouble deleting a test
gluster volume
Here is an update
I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the
node that had the trouble volume (test1). I didnt see any errors. So,
I ran a tail -f on the log as I tried to remove the volume using the
web UI. here is what was appended:
[2017-04-11 19:48:40.756360] I [MSGID: 106487]
[glusterd-handler.c:1474:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req
[2017-04-11 19:48:42.238840] I [MSGID: 106488]
[glusterd-handler.c:1537:__glusterd_handle_cli_get_volume]
0-management: Received get vol req
The message "I [MSGID: 106487]
[glusterd-handler.c:1474:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req" repeated 6 times between
[2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536]
The message "I [MSGID: 106488]
[glusterd-handler.c:1537:__glusterd_handle_cli_get_volume]
0-management: Received get vol req" repeated 20 times between
[2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179]
[2017-04-11 19:51:41.556077] I [MSGID: 106487]
[glusterd-handler.c:1474:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req
Im seeing that the timestamps on these log entries do not match the
time on the node.
The next steps
I stopped the glusterd service on the node with volume test1
I deleted it with: rm -rf /var/lib/glusterd/vols/test1
I started the glusterd service.
After starting the gluster service back up, the directory
/var/lib/glusterd/vols/test1 reappears.
Im guessing syncing with the other nodes?
Is this because I have the Volume Option: auth allow *
Do I need to remove the directory /var/lib/glusterd/vols/test1 on all
nodes in the cluster individually?
thanks
------------------------------------------------------------------------
*From:* knarra <knarra(a)redhat.com>
*Sent:* Tuesday, April 11, 2017 11:51:18 AM
*To:* Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon
Mureinik; Nir Soffer
*Cc:* users
*Subject:* Re: [ovirt-users] Im having trouble deleting a test
gluster volume
On 04/11/2017 11:28 PM, Precht, Andrew wrote:
> Hi all,
> The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7.
> On the node I can not find /var/log/glusterfs/glusterd.log However,
> there is a /var/log/glusterfs/glustershd.log
can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
exists? if yes, can you check if there is any error present in that file ?
>
> What happens if I follow the four steps outlined here to remove the
> volume from the node _BUT_, I do have another volume present in the
> cluster. It too is a test volume. Neither one has any data on them.
> So, data loss is not an issue.
Running those four steps will remove the volume from your cluster . If
the volumes what you have are test volumes you could just follow the
steps outlined to delete them (since you are not able to delete from
UI) and bring back the cluster into a normal state.
>
> ------------------------------------------------------------------------
> *From:* knarra <knarra(a)redhat.com>
> *Sent:* Tuesday, April 11, 2017 10:32:27 AM
> *To:* Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon
> Mureinik; Nir Soffer
> *Cc:* users
> *Subject:* Re: [ovirt-users] Im having trouble deleting a test
> gluster volume
> On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:
>> Adding some people
>>
>> Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht(a)sjlibrary.org
>> <mailto:Andrew.Precht@sjlibrary.org>> ha scritto:
>>
>> Hi Ovirt users,
>> Im a newbie to oVirt and Im having trouble deleting a test
>> gluster volume. The nodes are 4.1.1 and the engine is 4.1.0
>>
>> When I try to remove the test volume, I click Remove, the dialog
>> box prompting to confirm the deletion pops up and after I click
>> OK, the dialog box changes to show a little spinning wheel and
>> then it disappears. In the end the volume is still there.
>>
> with the latest version of glusterfs & ovirt we do not see any issue
> with deleting a volume. Can you please check
> /var/log/glusterfs/glusterd.log file if there is any error present?
>
>
>> The test volume was distributed with two host members. One of
>> the hosts I was able to remove from the volume by removing the
>> host form the cluster. When I try to remove the remaining host
>> in the volume, even with the Force Remove box ticked, I get
>> this response: Cannot remove Host. Server having Gluster volume.
>>
>> What to try next?
>>
> since you have already removed the volume from one host in the
> cluster and you still see it on another host you can do the following
> to remove the volume from another host.
>
> 1) Login to the host where the volume is present.
> 2) cd to /var/lib/glusterd/vols
> 3) rm -rf <vol_name>
> 4) Restart glusterd on that host.
>
> And before doing the above make sure that you do not have any other
> volume present in the cluster.
>
> Above steps should not be run on a production system as you might
> loose the volume and data.
>
> Now removing the host from UI should succed.
>
>>
>> P.S. Ive tried to join this user group several times in the
>> past, with no response.
>> Is it possible for me to join this group?
>>
>> Regards,
>> Andrew
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
>
--------------FBCB929546CA281A03A4314D
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 04/12/2017 03:35 AM, Precht, Andrew
wrote:<br>
</div>
<blockquote
cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;"><!-- P
{margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
dir="ltr">
<p>I just <span>noticed </span>this in the Alerts
tab: <span>Detected
deletion of volume test1 on cluster 8000-1, and deleted it
from engine DB.</span><br>
</p>
</div>
</blockquote>
<blockquote
cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
type="cite">
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
dir="ltr">
<p><span>Yet, It still shows in the web UI?</span></p>
</div>
</blockquote>
Any errors in ovirt-engine.log file ? if the volume is deleted from
db ideally it should be deleted from UI too. Can you go to gluster
nodes and check for the following:<br>
<br>
1) gluster volume list -> should not return anything since you
have deleted the volumes.<br>
<br>
2) gluster peer status -> on all the nodes should show that all
the peers are in connected state.<br>
<br>
can you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster
log and capture the error messages when you try deleting the volume
from UI?<br>
<br>
Log what you have pasted in the previous mail only gives info and i
could not get any details from that on why volume delete is failing<br>
<br>
<blockquote
cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
type="cite">
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
dir="ltr">
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt"
color="#000000" face="Calibri,
sans-serif"><b>From:</b>
Precht, Andrew<br>
<b>Sent:</b> Tuesday, April 11, 2017 2:39:31 PM<br>
<b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan;
Allon Mureinik; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] Im having trouble deleting
a test gluster volume</font>
<div> </div>
</div>
<div>
<style type="text/css" style="display:none;"><!-- P
{margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
dir="ltr">
<div>The plot thickens
</div>
<div>I put all hosts in the cluster into maintenance mode,
with the Stop Gluster service checkbox checked. I then
deleted the /var/lib/glusterd/vols/test1 directory on all
hosts. I then took the host that the test1 volume was on out
of maintenance mode. Then I tried to remove the test1 volume
from within the web UI. With no luck, I got the message:
Could not delete Gluster Volume test1 on cluster 8000-1.</div>
<div><br>
</div>
<div>I went back and checked all host for the test1 directory,
it is not on any host. Yet I still cant remove it
</div>
<div><br>
</div>
<div>Any suggestions?</div>
<br>
</div>
<hr style="display:inline-block;width:98%"
tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt"
color="#000000" face="Calibri,
sans-serif"><b>From:</b>
Precht, Andrew<br>
<b>Sent:</b> Tuesday, April 11, 2017 1:15:22 PM<br>
<b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan;
Allon Mureinik; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] Im having trouble
deleting a test gluster volume</font>
<div> </div>
</div>
<div>
<style type="text/css" style="display:none;"><!-- P
{margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
dir="ltr">
<div>Here is an update
</div>
<div><br>
</div>
<div>I checked the
/var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the
node that had the trouble volume (test1). I didnt see any
errors. So, I ran a tail -f on the log as I tried to
remove the volume using the web UI. here is what was
appended:</div>
<div><br>
</div>
<div>[2017-04-11 19:48:40.756360] I [MSGID: 106487]
[glusterd-handler.c:1474:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req</div>
<div>[2017-04-11 19:48:42.238840] I [MSGID: 106488]
[glusterd-handler.c:1537:__glusterd_handle_cli_get_volume]
0-management: Received get vol req</div>
<div>The message "I [MSGID: 106487]
[glusterd-handler.c:1474:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req" repeated 6 times
between [2017-04-11 19:48:40.756360] and [2017-04-11
19:49:32.596536]</div>
<div>The message "I [MSGID: 106488]
[glusterd-handler.c:1537:__glusterd_handle_cli_get_volume]
0-management: Received get vol req" repeated 20 times
between [2017-04-11 19:48:42.238840] and [2017-04-11
19:49:34.082179]</div>
<div>[2017-04-11 19:51:41.556077] I [MSGID: 106487]
[glusterd-handler.c:1474:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req</div>
<div><br>
</div>
<div>Im seeing that the timestamps on these log entries do
not match the time on the node.</div>
<div><br>
</div>
<div>The next steps</div>
<div>I stopped the glusterd service on the node with volume
test1</div>
<div>I deleted it with: rm -rf
/var/lib/glusterd/vols/test1</div>
<div>I started the glusterd service.</div>
<div><br>
</div>
<div>After starting the gluster service back up, the
directory /var/lib/glusterd/vols/test1 reappears.</div>
<div>Im guessing syncing with the other nodes?</div>
<div>Is this because I have the Volume Option: auth allow *</div>
<div>Do I need to remove the directory
/var/lib/glusterd/vols/test1 on all nodes in the cluster
individually?</div>
<div><br>
</div>
<div>thanks</div>
<br>
</div>
<hr style="display:inline-block;width:98%"
tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt"
color="#000000" face="Calibri,
sans-serif"><b>From:</b>
knarra <a class="moz-txt-link-rfc2396E"
href="mailto:knarra@redhat.com"><knarra@redhat.com></a><br>
<b>Sent:</b> Tuesday, April 11, 2017 11:51:18 AM<br>
<b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose;
Tal Nisan; Allon Mureinik; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] Im having trouble
deleting a test gluster volume</font>
<div> </div>
</div>
<div>
<div class="moz-cite-prefix">On 04/11/2017 11:28 PM, Precht,
Andrew wrote:<br>
</div>
<blockquote
cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
type="cite">
<style type="text/css"
style="display:none;"><!-- P {margin-top:0;margin-bottom:0;}
--></style>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
dir="ltr">
<div>Hi all,</div>
<div>The node is oVirt Node 4.1.1 with
glusterfs-3.8.10-1.el7. </div>
<div>On the node I can not find
/var/log/glusterfs/glusterd.log However, there is a
/var/log/glusterfs/glustershd.log
<br>
</div>
</div>
</blockquote>
can you check if
/var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? if
yes, can you check if there is any error present in that
file ?<span style="color: rgb(84, 84, 84); font-family:
arial, sans-serif; font-size: small; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; orphans: 2;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); display: inline !important; float: none;"></span>
<blockquote
cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
type="cite">
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
dir="ltr">
<div><br>
</div>
<div>What happens if I follow the four steps outlined
here to remove the volume from the node
<u>BUT</u>, I do have another volume present in the
cluster. It too is a test volume. Neither one has any
data on them. So, data loss is not an issue.</div>
</div>
</blockquote>
Running those four steps will remove the volume from your
cluster . If the volumes what you have are test volumes you
could just follow the steps outlined to delete them (since
you are not able to delete from UI) and bring back the
cluster into a normal state.<br>
<blockquote
cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com"
type="cite">
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
dir="ltr">
<br>
</div>
<hr style="display:inline-block;width:98%"
tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt" color="#000000"
face="Calibri,
sans-serif"><b>From:</b> knarra
<a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:knarra@redhat.com"><knarra@redhat.com></a><br>
<b>Sent:</b> Tuesday, April 11, 2017 10:32:27 AM<br>
<b>To:</b> Sandro Bonazzola; Precht, Andrew; Sahina
Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] Im having trouble
deleting a test gluster volume</font>
<div> </div>
</div>
<div>
<div class="moz-cite-prefix">On 04/11/2017 10:44 PM,
Sandro Bonazzola wrote:<br>
</div>
<blockquote
cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com"
type="cite">
<div dir="auto">Adding some people</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">Il 11/Apr/2017 19:06,
"Precht, Andrew" <<a
moz-do-not-send="true"
href="mailto:Andrew.Precht@sjlibrary.org">Andrew.Precht@sjlibrary.org</a>>
ha scritto:<br type="attribution">
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">
<div
id="m_7567123270890452842divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
dir="ltr">
<div>Hi Ovirt users,</div>
<div>Im a newbie to oVirt and Im having
trouble deleting a test gluster volume.
The nodes are 4.1.1 and the engine is
4.1.0</div>
<div><br>
</div>
<div>When I try to remove the test volume, I
click Remove, the dialog box prompting to
confirm the deletion pops up and after I
click OK, the dialog box changes to show a
little spinning wheel and then it
disappears. In the end the volume is still
there.</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
with the latest version of glusterfs & ovirt we do
not see any issue with deleting a volume. Can you please
check /var/log/glusterfs/glusterd.log file if there is
any error present?
<br>
<br>
<br>
<blockquote
cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com"
type="cite">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">
<div
id="m_7567123270890452842divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
dir="ltr">
The test volume was distributed with two
host members. One of the hosts I was able to
remove from the volume by removing the host
form the cluster. When I try to remove the
remaining host in the volume, even with the
Force Remove box ticked, I get this
response: Cannot remove Host. Server having
Gluster volume.
<div><br>
</div>
<div>What to try next?</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
since you have already removed the volume from one host
in the cluster and you still see it on another host you
can do the following to remove the volume from another
host.<br>
<br>
1) Login to the host where the volume is present.<br>
2) cd to /var/lib/glusterd/vols<br>
3) rm -rf <vol_name><br>
4) Restart glusterd on that host.<br>
<br>
And before doing the above make sure that you do not
have any other volume present in the cluster.<br>
<br>
Above steps should not be run on a production system as
you might loose the volume and data.
<br>
<br>
Now removing the host from UI should succed.<br>
<br>
<blockquote
cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com"
type="cite">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">
<div
id="m_7567123270890452842divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif"
dir="ltr">
<div><br>
</div>
<div>P.S. Ive tried to join this user group
several times in the past, with no
response.</div>
<div>Is it possible for me to join this
group?</div>
<div><br>
</div>
<div>Regards,</div>
<div>Andrew</div>
<br>
</div>
</div>
</blockquote>
</div>
</div>
<br>
<fieldset
class="mimeAttachmentHeader"></fieldset>
<br>
<pre
wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<p><br>
</p>
</div>
</blockquote>
<p><br>
</p>
</div>
</div>
</div>
</blockquote>
<p><br>
</p>
</body>
</html>
--------------FBCB929546CA281A03A4314D--