Re: [ovirt-users] I’m having trouble deleting a test gluster volume

--_000_BY1PR09MB0824A056B7185A98CDF374C7F5030BY1PR09MB0824namp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi all, You asked: Any errors in ovirt-engine.log file ? Yes, In the engine.log this error is repeated about every 3 minutes: 2017-04-12 07:16:12,554-07 ERROR [org.ovirt.engine.core.bll.gluster.Gluster= TasksSyncJob] (DefaultQuartzScheduler3) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7= b8] Error updating tasks from CLI: org.ovirt.engine.core.common.errors.Engi= neException: EngineException: Command execution failed error: Error : Request timed out return code: 1 (Failed with error GlusterVolumeStatusAllFailedException and= code 4161) error: Error : Request timed out 1) gluster volume list -> Still shows the deleted volume (test1) 2) gluster peer status -> Shows one of the peers twice with different uuid= =92s: Hostname: 192.168.10.109 Uuid: 42fbb7de-8e6f-4159-a601-3f858fa65f6c State: Peer in Cluster (Connected) Hostname: 192.168.10.109 Uuid: e058babe-7f9d-49fe-a3ea-ccdc98d7e5b5 State: Peer in Cluster (Connected) I tried a gluster volume stop test1, with this result: volume stop: test1: = failed: Another transaction is in progress for test1. Please try again afte= r sometime. The etc-glusterfs-glusterd.vol.log shows no activity triggered by trying to= remove the test1 volume from the UI. The ovirt-engine.log shows this repeating many times, when trying to remove= the test1 volume from the UI: 2017-04-12 07:57:38,049-07 INFO [org.ovirt.engine.core.bll.lock.InMemoryLo= ckManager] (DefaultQuartzScheduler9) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8]= Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks=3D'[b0e1b= 909-9a6a-49dc-8e20-3a027218f7e1=3D<GLUSTER, ACTION_TYPE_FAILED_GLUSTER_OPER= ATION_INPROGRESS>]', sharedLocks=3D'null'}' Thanks much, Andrew ________________________________ From: knarra <knarra(a)redhat.comSent: Tuesday, April 11, 2017 11:10:04 PM To: Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureini= k; Nir Soffer Cc: users Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol= ume On 04/12/2017 03:35 AM, Precht, Andrew wrote: I just noticed this in the Alerts tab: Detected deletion of volume test1 on= cluster 8000-1, and deleted it from engine DB. Yet, It still shows in the web UI? Any errors in ovirt-engine.log file ? if the volume is deleted from db idea= lly it should be deleted from UI too. Can you go to gluster nodes and chec= k for the following: 1) gluster volume list -> should not return anything since you have deleted= the volumes. 2) gluster peer status -> on all the nodes should show that all the peers a= re in connected state. can you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster log and = capture the error messages when you try deleting the volume from UI? Log what you have pasted in the previous mail only gives info and i could n= ot get any details from that on why volume delete is failing ________________________________ From: Precht, Andrew Sent: Tuesday, April 11, 2017 2:39:31 PM To: knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir S= offer Cc: users Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol= ume The plot thickens=85 I put all hosts in the cluster into maintenance mode, with the Stop Gluster= service checkbox checked. I then deleted the /var/lib/glusterd/vols/test1 = directory on all hosts. I then took the host that the test1 volume was on o= ut of maintenance mode. Then I tried to remove the test1 volume from within= the web UI. With no luck, I got the message: Could not delete Gluster Volu= me test1 on cluster 8000-1. I went back and checked all host for the test1 directory, it is not on any = host. Yet I still can=92t remove it=85 Any suggestions? ________________________________ From: Precht, Andrew Sent: Tuesday, April 11, 2017 1:15:22 PM To: knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir S= offer Cc: users Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol= ume Here is an update=85 I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the node= that had the trouble volume (test1). I didn=92t see any errors. So, I ran = a tail -f on the log as I tried to remove the volume using the web UI. here= is what was appended: [2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:1474:__g= lusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:1537:__g= lusterd_handle_cli_get_volume] 0-management: Received get vol req The message "I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_c= li_list_friends] 0-glusterd: Received cli list req" repeated 6 times betwee= n [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536] The message "I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_c= li_get_volume] 0-management: Received get vol req" repeated 20 times betwee= n [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179] [2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:1474:__g= lusterd_handle_cli_list_friends] 0-glusterd: Received cli list req I=92m seeing that the timestamps on these log entries do not match the time= on the node. The next steps I stopped the glusterd service on the node with volume test1 I deleted it with: rm -rf /var/lib/glusterd/vols/test1 I started the glusterd service. After starting the gluster service back up, the directory /var/lib/glusterd= /vols/test1 reappears. I=92m guessing syncing with the other nodes? Is this because I have the Volume Option: auth allow * Do I need to remove the directory /var/lib/glusterd/vols/test1 on all nodes= in the cluster individually? thanks ________________________________ From: knarra <knarra@redhat.com><mailto:knarra@redhat.comSent: Tuesday, April 11, 2017 11:51:18 AM To: Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureini= k; Nir Soffer Cc: users Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol= ume On 04/11/2017 11:28 PM, Precht, Andrew wrote: Hi all, The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. On the node I can not find /var/log/glusterfs/glusterd.log However, there i= s a /var/log/glusterfs/glustershd.log can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? = if yes, can you check if there is any error present in that file ? What happens if I follow the four steps outlined here to remove the volume = from the node BUT, I do have another volume present in the cluster. It too = is a test volume. Neither one has any data on them. So, data loss is not an= issue. Running those four steps will remove the volume from your cluster . If the = volumes what you have are test volumes you could just follow the steps outl= ined to delete them (since you are not able to delete from UI) and bring ba= ck the cluster into a normal state. ________________________________ From: knarra <knarra@redhat.com><mailto:knarra@redhat.comSent: Tuesday, April 11, 2017 10:32:27 AM To: Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureini= k; Nir Soffer Cc: users Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol= ume On 04/11/2017 10:44 PM, Sandro Bonazzola wrote: Adding some people Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht@sjlibrary.org<mailto:= Andrew.Precht(a)sjlibrary.org&gt;&gt; ha scritto: Hi Ovirt users, I=92m a newbie to oVirt and I=92m having trouble deleting a test gluster vo= lume. The nodes are 4.1.1 and the engine is 4.1.0 When I try to remove the test volume, I click Remove, the dialog box prompt= ing to confirm the deletion pops up and after I click OK, the dialog box ch= anges to show a little spinning wheel and then it disappears. In the end th= e volume is still there. with the latest version of glusterfs & ovirt we do not see any issue with d= eleting a volume. Can you please check /var/log/glusterfs/glusterd.log file= if there is any error present? The test volume was distributed with two host members. One of the hosts I w= as able to remove from the volume by removing the host form the cluster. Wh= en I try to remove the remaining host in the volume, even with the =93Force= Remove=94 box ticked, I get this response: Cannot remove Host. Server havi= ng Gluster volume. What to try next? since you have already removed the volume from one host in the cluster and = you still see it on another host you can do the following to remove the vol= ume from another host. 1) Login to the host where the volume is present. 2) cd to /var/lib/glusterd/vols 3) rm -rf <vol_name4) Restart glusterd on that host. And before doing the above make sure that you do not have any other volume = present in the cluster. Above steps should not be run on a production system as you might loose the= volume and data. Now removing the host from UI should succed. P.S. I=92ve tried to join this user group several times in the past, with n= o response. Is it possible for me to join this group? Regards, Andrew _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users --_000_BY1PR09MB0824A056B7185A98CDF374C7F5030BY1PR09MB0824namp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html<head<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"</head<body bgcolor=3D"#FFFFFF" text=3D"#000000"<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi= n-bottom:0;} --></style<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"<p>Hi all,</p<p><span id=3D"docs-internal-guid-44c81590-62b4-32c0-d6d1-82b627328ebb"></p= <p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"= 3); vertical-align: baseline; white-space: pre-wrap;">You asked: Any errors= in ovirt-engine.log file ?</span></p<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;t= ext-indent: 36pt;"<span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33= ); vertical-align: baseline; white-space: pre-wrap;">Yes, In the engine.log= this error is repeated about every 3 minutes:</span></p<br<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"= 3); vertical-align: baseline; white-space: pre-wrap;">2017-04-12 07:16:12,5= 54-07 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob] (DefaultQuartzScheduler3) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Error upd= ating tasks from CLI: org.ovirt.engine.core.common.errors.EngineException: = EngineException: Command execution failed</span><span style=3D"font-size: 1= 1.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline= ; white-space: pre-wrap;"><br class=3D"kix-line-break"</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,= 33, 33); vertical-align: baseline; white-space: pre-wrap;">error: Error : = Request timed out</span><span style=3D"font-size: 11.5pt; font-family: Aria= l; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;= "><br class=3D"kix-line-break"</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,= 33, 33); vertical-align: baseline; white-space: pre-wrap;">return code: 1 = (Failed with error GlusterVolumeStatusAllFailedException and code 4161)</sp= an><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33,= 33); vertical-align: baseline; white-space: pre-wrap;"><br class=3D"kix-li= ne-break"</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,= 33, 33); vertical-align: baseline; white-space: pre-wrap;">error: Error : = Request timed out</span></p<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;t= ext-indent: 36pt;"<span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33= ); vertical-align: baseline; white-space: pre-wrap;"><br class=3D"kix-line-= break"</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,= 33, 33); vertical-align: baseline; white-space: pre-wrap;">1) gluster volu= me list -&gt; &nbsp;Still shows the deleted volume (test1)</span></p<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;t= ext-indent: 36pt;"<span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33= ); vertical-align: baseline; white-space: pre-wrap;"><br class=3D"kix-line-= break"</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,= 33, 33); vertical-align: baseline; white-space: pre-wrap;">2) gluster peer= status -&gt; Shows one of the peers twice with different uuid=92s:</span><= /p<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"= 3); vertical-align: baseline; white-space: pre-wrap;">Hostname: 192.168.10.= 109</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(= 33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"><br class=3D= "kix-line-break"</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,= 33, 33); vertical-align: baseline; white-space: pre-wrap;">Uuid: 42fbb7de-= 8e6f-4159-a601-3f858fa65f6c</span><span style=3D"font-size: 11.5pt; font-fa= mily: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space:= pre-wrap;"><br class=3D"kix-line-break"</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,= 33, 33); vertical-align: baseline; white-space: pre-wrap;">State: Peer in = Cluster (Connected)</span><span style=3D"font-size: 11.5pt; font-family: Ar= ial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wra= p;"><br class=3D"kix-line-break"</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,= 33, 33); vertical-align: baseline; white-space: pre-wrap;"><br class=3D"ki= x-line-break"</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,= 33, 33); vertical-align: baseline; white-space: pre-wrap;">Hostname: 192.1= 68.10.109</span><span style=3D"font-size: 11.5pt; font-family: Arial; color= : rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"><br cl= ass=3D"kix-line-break"</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,= 33, 33); vertical-align: baseline; white-space: pre-wrap;">Uuid: e058babe-= 7f9d-49fe-a3ea-ccdc98d7e5b5</span><span style=3D"font-size: 11.5pt; font-fa= mily: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space:= pre-wrap;"><br class=3D"kix-line-break"</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,= 33, 33); vertical-align: baseline; white-space: pre-wrap;">State: Peer in = Cluster (Connected)</span></p<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"= 3); vertical-align: baseline; white-space: pre-wrap;"><br</span></p<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"= Arial"><span style=3D"font-size: 11.5pt; white-space: pre-wrap;">I tried a = gluster volume stop test1, with this </span><span style=3D"font-size: 15.3333px; white-space: pre-wrap;">result<= /span><span style=3D"font-size: 11.5pt; white-space: pre-wrap;">: <span>volume stop: test1: failed: Another transaction is in progress for te= st1. Please try again after sometime.</span></span></font></span></p<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"= pt; white-space: pre-wrap; text-indent: 36pt;"><br</span></p<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"= pt; white-space: pre-wrap; text-indent: 36pt;"><br</span></p<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"= 3); vertical-align: baseline; white-space: pre-wrap;"><span></span></span><= span style=3D"color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt= ; white-space: pre-wrap; text-indent: 36pt;">The etc-glusterfs-glusterd.vol.log shows no activity triggered by trying to re= move the test1 volume from the UI</span><span style=3D"color: rgb(33, 33, 3= 3); font-family: Arial; font-size: 11.5pt; white-space: pre-wrap; text-inde= nt: 36pt;">.</span></p<br<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"= 3); vertical-align: baseline; white-space: pre-wrap;">The ovirt-engine.log = shows this repeating many times, when <span style=3D"color: rgb(33, 33, 33); font-family: Arial; font-size: 15.33= 33px; text-indent: 48px; white-space: pre-wrap;"trying to remove the test1 volume from the UI</span>:</span></p<br<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"= 3); vertical-align: baseline; white-space: pre-wrap;">2017-04-12 07:57:38,0= 49-07 INFO &nbsp;[org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler9) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Failed to= acquire lock and wait lock 'EngineLock:{exclusiveLocks=3D'[b0e1b909-9a6a-4= 9dc-8e20-3a027218f7e1=3D&lt;GLUSTER, ACTION_TYPE_FAILED_GLUSTER_OPERATION_I= NPROGRESS&gt;]', sharedLocks=3D'null'}'</span><span style=3D"font-size: 11.= 5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; = white-space: pre-wrap;"><br class=3D"kix-line-break"<br class=3D"kix-line-break"</span></p</span>Thanks much, <p></p<p>Andrew</p</div<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" st= yle=3D"font-size:11pt" color=3D"#000000"><b>From:</b> knarra &lt;knarra@red= hat.com&gt;<br<b>Sent:</b> Tuesday, April 11, 2017 11:10:04 PM<br<b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon = Mureinik; Nir Soffer<br<b>Cc:</b> users<br<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus= ter volume</font<div>&nbsp;</div</div<div<div class=3D"moz-cite-prefix">On 04/12/2017 03:35 AM, Precht, Andrew wrote= :<br</div<blockquote cite=3D"mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB082= 4.namprd09.prod.outlook.com" type=3D"cite"<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi= n-bottom:0;} --></style<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"<p>I just <span>noticed&nbsp;</span>this in the Alerts tab:&nbsp;<span>Dete= cted deletion of volume test1 on cluster 8000-1, and deleted it from engine= DB.</span><br</p</div</blockquote<blockquote cite=3D"mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB082= 4.namprd09.prod.outlook.com" type=3D"cite"<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"<p><span>Yet, It still shows in the web UI?</span></p</div</blockquoteAny errors in ovirt-engine.log file ? if the volume is deleted from db idea= lly it should be deleted from UI too.&nbsp; Can you go to gluster nodes and= check for the following:<br<br1) gluster volume list -&gt; should not return anything since you have dele= ted the volumes.<br<br2) gluster peer status -&gt; on all the nodes should show that all the peer= s are in connected state.<br<brcan you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster log and = capture the error messages when you try deleting the volume from UI?<br<brLog what you have pasted in the previous mail only gives info and i could n= ot get any details from that on why volume delete is failing<br<br<blockquote cite=3D"mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB082= 4.namprd09.prod.outlook.com" type=3D"cite"<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"</div<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color= =3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> Precht, Andrew<br<b>Sent:</b> Tuesday, April 11, 2017 2:39:31 PM<br<b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik= ; Nir Soffer<br<b>Cc:</b> users<br<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus= ter volume</font<div>&nbsp;</div</div<div><style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;= margin-bottom:0;} --></style<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"<div>The plot thickens=85</div<div>I put all hosts in the cluster into maintenance mode, with the Stop Gl= uster service checkbox checked. I then deleted the /var/lib/glusterd/vols/t= est1 directory on all hosts. I then took the host that the test1 volume was= on out of maintenance mode. Then I tried to remove the test1 volume from within the web UI. With no luck, I= got the message: Could not delete Gluster Volume test1 on cluster 8000-1.<= /div<div><br</div<div>I went back and checked all host for the test1 directory, it is not on= any host. Yet I still can=92t remove it=85</div<div><br</div<div>Any suggestions?</div<br</div<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color= =3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> Precht, Andrew<br<b>Sent:</b> Tuesday, April 11, 2017 1:15:22 PM<br<b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik= ; Nir Soffer<br<b>Cc:</b> users<br<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus= ter volume</font<div>&nbsp;</div</div<div><style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;= margin-bottom:0;} --></style<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"<div>Here is an update=85</div<div><br</div<div>I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the= node that had the trouble volume (test1). I didn=92t see any errors. So, I= ran a tail -f on the log as I tried to remove the volume using the web UI.= here is what was appended:</div<div><br</div<div>[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:147= 4:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</di= v<div>[2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:153= 7:__glusterd_handle_cli_get_volume] 0-management: Received get vol req</div= <div>The message &quot;I [MSGID: 106487] [glusterd-handler.c:1474:__gluster= d_handle_cli_list_friends] 0-glusterd: Received cli list req&quot; repeated= 6 times between [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.5965= 36]</div<div>The message &quot;I [MSGID: 106488] [glusterd-handler.c:1537:__gluster= d_handle_cli_get_volume] 0-management: Received get vol req&quot; repeated = 20 times between [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.0821= 79]</div<div>[2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:147= 4:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</di= v<div><br</div<div>I=92m seeing that the timestamps on these log entries do not match the= time on the node.</div<div><br</div<div>The next steps</div<div>I stopped the glusterd service on the node with volume test1</div<div>I deleted it with: &nbsp;rm -rf /var/lib/glusterd/vols/test1</div<div>I started the glusterd service.</div<div><br</div<div>After starting the gluster service back up, the directory /var/lib/glu= sterd/vols/test1 reappears.</div<div>I=92m guessing syncing with the other nodes?</div<div>Is this because I have the Volume Option: auth allow *</div<div>Do I need to remove the directory /var/lib/glusterd/vols/test1 on all = nodes in the cluster individually?</div<div><br</div<div>thanks</div<br</div<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color= =3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> knarra <a class=3D"moz-txt-link-rfc2396E" href=3D"mailto:knarra@redhat.com">&lt;kn= arra(a)redhat.com&amp;gt;&lt;/a&gt;&lt;br<b>Sent:</b> Tuesday, April 11, 2017 11:51:18 AM<br<b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon = Mureinik; Nir Soffer<br<b>Cc:</b> users<br<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus= ter volume</font<div>&nbsp;</div</div<div<div class=3D"moz-cite-prefix">On 04/11/2017 11:28 PM, Precht, Andrew wrote= :<br</div<blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082= 4.namprd09.prod.outlook.com" type=3D"cite"<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi= n-bottom:0;} --></style<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"<div>Hi all,</div<div>The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7.&nbsp;</div<div>On the node I can not find /var/log/glusterfs/glusterd.log However, th= ere is a /var/log/glusterfs/glustershd.log <br</div</div</blockquotecan you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? = if yes, can you check if there is any error present in that file ?<span sty= le=3D"color: rgb(84, 84, 84); font-family: arial, sans-serif; font-size: small; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); display: inline !important; float: none;"></span<blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082= 4.namprd09.prod.outlook.com" type=3D"cite"<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"<div><br</div<div>What happens if I follow the four steps outlined here to remove the vo= lume from the node <u>BUT</u>, I do have another volume present in the cluster. It too is a te= st volume. Neither one has any data on them. So, data loss is not an issue.= </div</div</blockquoteRunning those four steps will remove the volume from your cluster . If the = volumes what you have are test volumes you could just follow the steps outl= ined to delete them (since you are not able to delete from UI) and bring ba= ck the cluster into a normal state.<br<blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082= 4.namprd09.prod.outlook.com" type=3D"cite"<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"<br</div<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color= =3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> knarra <a moz-do-not-send=3D"true" class=3D"moz-txt-link-rfc2396E" href=3D"mailto:= knarra(a)redhat.com&quot;&amp;lt;knarra(a)redhat.com&amp;gt;&lt;/a&gt;&lt;br<b>Sent:</b> Tuesday, April 11, 2017 10:32:27 AM<br<b>To:</b> Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon = Mureinik; Nir Soffer<br<b>Cc:</b> users<br<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus= ter volume</font<div>&nbsp;</div</div<div<div class=3D"moz-cite-prefix">On 04/11/2017 10:44 PM, Sandro Bonazzola wro= te:<br</div<blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr&#43;EAAHgY_oA= bkkQ(a)mail.gmail.com&quot; type=3D"cite"<div dir=3D"auto">Adding some people</div<div class=3D"gmail_extra"><br<div class=3D"gmail_quote">Il 11/Apr/2017 19:06, &quot;Precht, Andrew&quot;= &lt;<a moz-do-not-send=3D"true" href=3D"mailto:Andrew.Precht@sjlibrary.org= &quot;&gt;Andrew.Precht(a)sjlibrary.org&lt;/a&gt;&amp;gt; ha scritto:<br type=3D"attribution"<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"<div dir=3D"ltr"<div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12= pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr= "<div>Hi Ovirt users,</div<div>I=92m a newbie to oVirt and I=92m having trouble deleting a test glust= er volume. The nodes are 4.1.1 and the engine is 4.1.0</div<div><br</div<div>When I try to remove the test volume, I click Remove, the dialog box p= rompting to confirm the deletion pops up and after I click OK, the dialog b= ox changes to show a little spinning wheel and then it disappears. In the e= nd the volume is still there.</div</div</div</blockquote</div</div</blockquotewith the latest version of glusterfs &amp; ovirt we do not see any issue wi= th deleting a volume. Can you please check /var/log/glusterfs/glusterd.log = file if there is any error present? <br<br<br<blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr&#43;EAAHgY_oA= bkkQ(a)mail.gmail.com&quot; type=3D"cite"<div class=3D"gmail_extra"<div class=3D"gmail_quote"<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"<div dir=3D"ltr"<div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12= pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr= "The test volume was distributed with two host members. One of the hosts I w= as able to remove from the volume by removing the host form the cluster. Wh= en I try to remove the remaining host in the volume, even with the =93Force= Remove=94 box ticked, I get this response: Cannot remove Host. Server having Gluster volume. <div><br</div<div>What to try next?</div</div</div</blockquote</div</div</blockquotesince you have already removed the volume from one host in the cluster and = you still see it on another host you can do the following to remove the vol= ume from another host.<br<br1) Login to the host where the volume is present.<br2) cd to /var/lib/glusterd/vols<br3) rm -rf &lt;vol_name&gt;<br4) Restart glusterd on that&nbsp; host.<br<brAnd before doing the above make sure that you do not have any other volume = present in the cluster.<br<brAbove steps should not be run on a production system as you might loose the= volume and data. <br<brNow removing the host from UI should succed.<br<br<blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr&#43;EAAHgY_oA= bkkQ(a)mail.gmail.com&quot; type=3D"cite"<div class=3D"gmail_extra"<div class=3D"gmail_quote"<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"<div dir=3D"ltr"<div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12= pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr= "<div><br</div<div>P.S. I=92ve tried to join this user group several times in the past, w= ith no response.</div<div>Is it possible for me to join this group?</div<div><br</div<div>Regards,</div<div>Andrew</div<br</div</div</blockquote</div</div<br<fieldset class=3D"mimeAttachmentHeader"></fieldset> <br<pre wrap=3D"">_______________________________________________ Users mailing list <a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"mail= to:Users@ovirt.org">Users@ovirt.org</a<a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext" href=3D"http://= lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/ma... info/users</a</pre</blockquote<p><br</p</div</blockquote<p><br</p</div</div</div</blockquote<p><br</p</div</body</html --_000_BY1PR09MB0824A056B7185A98CDF374C7F5030BY1PR09MB0824namp_--