I’m having trouble deleting a test gluster volume
--_000_BY1PR09MB0824B7766560130DB97C50A0F5000BY1PR09MB0824namp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi Ovirt users, I=92m a newbie to oVirt and I=92m having trouble deleting a test gluster vo= lume. The nodes are 4.1.1 and the engine is 4.1.0 When I try to remove the test volume, I click Remove, the dialog box prompt= ing to confirm the deletion pops up and after I click OK, the dialog box ch= anges to show a little spinning wheel and then it disappears. In the end th= e volume is still there. The test volume was distributed with two host members. One of the hosts I w= as able to remove from the volume by removing the host form the cluster. Wh= en I try to remove the remaining host in the volume, even with the =93Force= Remove=94 box ticked, I get this response: Cannot remove Host. Server havi= ng Gluster volume. What to try next? P.S. I=92ve tried to join this user group several times in the past, with n= o response. Is it possible for me to join this group? Regards, Andrew --_000_BY1PR09MB0824B7766560130DB97C50A0F5000BY1PR09MB0824namp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> <style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi= n-bottom:0;} --></style> </head> <body dir=3D"ltr"> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <p></p> <div>Hi Ovirt users,</div> <div>I=92m a newbie to oVirt and I=92m having trouble deleting a test glust= er volume. The nodes are 4.1.1 and the engine is 4.1.0</div> <div><br> </div> <div>When I try to remove the test volume, I click Remove, the dialog box p= rompting to confirm the deletion pops up and after I click OK, the dialog b= ox changes to show a little spinning wheel and then it disappears. In the e= nd the volume is still there.</div> <div><br> </div> <div>The test volume was distributed with two host members. One of the host= s I was able to remove from the volume by removing the host form the cluste= r. When I try to remove the remaining host in the volume, even with the =93= Force Remove=94 box ticked, I get this response: Cannot remove Host. Server having Gluster volume.</div> <div><br> </div> <div>What to try next?</div> <div><br> </div> <div>P.S. I=92ve tried to join this user group several times in the past, w= ith no response.</div> <div>Is it possible for me to join this group?</div> <div><br> </div> <div>Regards,</div> <div>Andrew</div> <br> <p></p> </div> </body> </html> --_000_BY1PR09MB0824B7766560130DB97C50A0F5000BY1PR09MB0824namp_--
Adding some people Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht@sjlibrary.org> ha scritto:
Hi Ovirt users, I’m a newbie to oVirt and I’m having trouble deleting a test gluster volume. The nodes are 4.1.1 and the engine is 4.1.0
When I try to remove the test volume, I click Remove, the dialog box prompting to confirm the deletion pops up and after I click OK, the dialog box changes to show a little spinning wheel and then it disappears. In the end the volume is still there.
The test volume was distributed with two host members. One of the hosts I was able to remove from the volume by removing the host form the cluster. When I try to remove the remaining host in the volume, even with the “Force Remove” box ticked, I get this response: Cannot remove Host. Server having Gluster volume.
What to try next?
P.S. I’ve tried to join this user group several times in the past, with no response. Is it possible for me to join this group?
Regards, Andrew
This is a multi-part message in MIME format. --------------A2300E89362703AC9371C3F9 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:
Adding some people
Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht@sjlibrary.org <mailto:Andrew.Precht@sjlibrary.org>> ha scritto:
Hi Ovirt users, Im a newbie to oVirt and Im having trouble deleting a test gluster volume. The nodes are 4.1.1 and the engine is 4.1.0
When I try to remove the test volume, I click Remove, the dialog box prompting to confirm the deletion pops up and after I click OK, the dialog box changes to show a little spinning wheel and then it disappears. In the end the volume is still there.
with the latest version of glusterfs & ovirt we do not see any issue with deleting a volume. Can you please check /var/log/glusterfs/glusterd.log file if there is any error present?
The test volume was distributed with two host members. One of the hosts I was able to remove from the volume by removing the host form the cluster. When I try to remove the remaining host in the volume, even with the Force Remove box ticked, I get this response: Cannot remove Host. Server having Gluster volume.
What to try next?
since you have already removed the volume from one host in the cluster and you still see it on another host you can do the following to remove the volume from another host. 1) Login to the host where the volume is present. 2) cd to /var/lib/glusterd/vols 3) rm -rf <vol_name> 4) Restart glusterd on that host. And before doing the above make sure that you do not have any other volume present in the cluster. Above steps should not be run on a production system as you might loose the volume and data. Now removing the host from UI should succed.
P.S. Ive tried to join this user group several times in the past, with no response. Is it possible for me to join this group?
Regards, Andrew
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--------------A2300E89362703AC9371C3F9 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:<br> </div> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div dir="auto">Adding some people</div> <div class="gmail_extra"><br> <div class="gmail_quote">Il 11/Apr/2017 19:06, "Precht, Andrew" <<a moz-do-not-send="true" href="mailto:Andrew.Precht@sjlibrary.org">Andrew.Precht@sjlibrary.org</a>> ha scritto:<br type="attribution"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> <div>Hi Ovirt users,</div> <div>Im a newbie to oVirt and Im having trouble deleting a test gluster volume. The nodes are 4.1.1 and the engine is 4.1.0</div> <div><br> </div> <div>When I try to remove the test volume, I click Remove, the dialog box prompting to confirm the deletion pops up and after I click OK, the dialog box changes to show a little spinning wheel and then it disappears. In the end the volume is still there.</div> </div> </div> </blockquote> </div> </div> </blockquote> with the latest version of glusterfs & ovirt we do not see any issue with deleting a volume. Can you please check /var/log/glusterfs/glusterd.log file if there is any error present? <br> <br> <br> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">The test volume was distributed with two host members. One of the hosts I was able to remove from the volume by removing the host form the cluster. When I try to remove the remaining host in the volume, even with the Force Remove box ticked, I get this response: Cannot remove Host. Server having Gluster volume. <div><br> </div> <div>What to try next?</div> </div> </div> </blockquote> </div> </div> </blockquote> since you have already removed the volume from one host in the cluster and you still see it on another host you can do the following to remove the volume from another host.<br> <br> 1) Login to the host where the volume is present.<br> 2) cd to /var/lib/glusterd/vols<br> 3) rm -rf <vol_name><br> 4) Restart glusterd on that host.<br> <br> And before doing the above make sure that you do not have any other volume present in the cluster.<br> <br> Above steps should not be run on a production system as you might loose the volume and data. <br> <br> Now removing the host from UI should succed.<br> <br> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> <div><br> </div> <div>P.S. Ive tried to join this user group several times in the past, with no response.</div> <div>Is it possible for me to join this group?</div> <div><br> </div> <div>Regards,</div> <div>Andrew</div> <br> </div> </div> </blockquote> </div> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <p><br> </p> </body> </html> --------------A2300E89362703AC9371C3F9--
--_000_BY1PR09MB08246AC41807C7DFFE7A3308F5000BY1PR09MB0824namp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi all, The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. On the node I can not find /var/log/glusterfs/glusterd.log However, there i= s a /var/log/glusterfs/glustershd.log What happens if I follow the four steps outlined here to remove the volume = from the node BUT, I do have another volume present in the cluster. It too = is a test volume. Neither one has any data on them. So, data loss is not an= issue. ________________________________ From: knarra <knarra@redhat.com> Sent: Tuesday, April 11, 2017 10:32:27 AM To: Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureini= k; Nir Soffer Cc: users Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol= ume On 04/11/2017 10:44 PM, Sandro Bonazzola wrote: Adding some people Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht@sjlibrary.org<mailto:= Andrew.Precht@sjlibrary.org>> ha scritto: Hi Ovirt users, I=92m a newbie to oVirt and I=92m having trouble deleting a test gluster vo= lume. The nodes are 4.1.1 and the engine is 4.1.0 When I try to remove the test volume, I click Remove, the dialog box prompt= ing to confirm the deletion pops up and after I click OK, the dialog box ch= anges to show a little spinning wheel and then it disappears. In the end th= e volume is still there. with the latest version of glusterfs & ovirt we do not see any issue with d= eleting a volume. Can you please check /var/log/glusterfs/glusterd.log file= if there is any error present? The test volume was distributed with two host members. One of the hosts I w= as able to remove from the volume by removing the host form the cluster. Wh= en I try to remove the remaining host in the volume, even with the =93Force= Remove=94 box ticked, I get this response: Cannot remove Host. Server havi= ng Gluster volume. What to try next? since you have already removed the volume from one host in the cluster and = you still see it on another host you can do the following to remove the vol= ume from another host. 1) Login to the host where the volume is present. 2) cd to /var/lib/glusterd/vols 3) rm -rf <vol_name> 4) Restart glusterd on that host. And before doing the above make sure that you do not have any other volume = present in the cluster. Above steps should not be run on a production system as you might loose the= volume and data. Now removing the host from UI should succed. P.S. I=92ve tried to join this user group several times in the past, with n= o response. Is it possible for me to join this group? Regards, Andrew _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users --_000_BY1PR09MB08246AC41807C7DFFE7A3308F5000BY1PR09MB0824namp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> </head> <body bgcolor=3D"#FFFFFF" text=3D"#000000"> <style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi= n-bottom:0;} --></style> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <p></p> <div>Hi all,</div> <div>The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. </div> <div>On the node I can not find /var/log/glusterfs/glusterd.log However, th= ere is a /var/log/glusterfs/glustershd.log </div> <div><br> </div> <div>What happens if I follow the four steps outlined here to remove the vo= lume from the node <u>BUT</u>, I do have another volume present in the cluster. It too is a te= st volume. Neither one has any data on them. So, data loss is not an issue.= </div> <br> <p></p> </div> <hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"> <div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" st= yle=3D"font-size:11pt" color=3D"#000000"><b>From:</b> knarra <knarra@red= hat.com><br> <b>Sent:</b> Tuesday, April 11, 2017 10:32:27 AM<br> <b>To:</b> Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon = Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus= ter volume</font> <div> </div> </div> <div> <div class=3D"moz-cite-prefix">On 04/11/2017 10:44 PM, Sandro Bonazzola wro= te:<br> </div> <blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA= bkkQ@mail.gmail.com" type=3D"cite"> <div dir=3D"auto">Adding some people</div> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote">Il 11/Apr/2017 19:06, "Precht, Andrew"= <<a moz-do-not-send=3D"true" href=3D"mailto:Andrew.Precht@sjlibrary.org= ">Andrew.Precht@sjlibrary.org</a>> ha scritto:<br type=3D"attribution"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir=3D"ltr"> <div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12= pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr= "> <div>Hi Ovirt users,</div> <div>I=92m a newbie to oVirt and I=92m having trouble deleting a test glust= er volume. The nodes are 4.1.1 and the engine is 4.1.0</div> <div><br> </div> <div>When I try to remove the test volume, I click Remove, the dialog box p= rompting to confirm the deletion pops up and after I click OK, the dialog b= ox changes to show a little spinning wheel and then it disappears. In the e= nd the volume is still there.</div> </div> </div> </blockquote> </div> </div> </blockquote> with the latest version of glusterfs & ovirt we do not see any issue wi= th deleting a volume. Can you please check /var/log/glusterfs/glusterd.log = file if there is any error present? <br> <br> <br> <blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA= bkkQ@mail.gmail.com" type=3D"cite"> <div class=3D"gmail_extra"> <div class=3D"gmail_quote"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir=3D"ltr"> <div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12= pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr= "> The test volume was distributed with two host members. One of the hosts I w= as able to remove from the volume by removing the host form the cluster. Wh= en I try to remove the remaining host in the volume, even with the =93Force= Remove=94 box ticked, I get this response: Cannot remove Host. Server having Gluster volume. <div><br> </div> <div>What to try next?</div> </div> </div> </blockquote> </div> </div> </blockquote> since you have already removed the volume from one host in the cluster and = you still see it on another host you can do the following to remove the vol= ume from another host.<br> <br> 1) Login to the host where the volume is present.<br> 2) cd to /var/lib/glusterd/vols<br> 3) rm -rf <vol_name><br> 4) Restart glusterd on that host.<br> <br> And before doing the above make sure that you do not have any other volume = present in the cluster.<br> <br> Above steps should not be run on a production system as you might loose the= volume and data. <br> <br> Now removing the host from UI should succed.<br> <br> <blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA= bkkQ@mail.gmail.com" type=3D"cite"> <div class=3D"gmail_extra"> <div class=3D"gmail_quote"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir=3D"ltr"> <div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12= pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr= "> <div><br> </div> <div>P.S. I=92ve tried to join this user group several times in the past, w= ith no response.</div> <div>Is it possible for me to join this group?</div> <div><br> </div> <div>Regards,</div> <div>Andrew</div> <br> </div> </div> </blockquote> </div> </div> <br> <fieldset class=3D"mimeAttachmentHeader"></fieldset> <br> <pre wrap=3D"">_______________________________________________ Users mailing list <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:Users@ovirt.org">Users= @ovirt.org</a> <a class=3D"moz-txt-link-freetext" href=3D"http://lists.ovirt.org/mailman/l= istinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <p><br> </p> </div> </body> </html> --_000_BY1PR09MB08246AC41807C7DFFE7A3308F5000BY1PR09MB0824namp_--
Hi all, The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. On the node I can not find /var/log/glusterfs/glusterd.log However, there is a /var/log/glusterfs/glustershd.log can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? if yes, can you check if there is any error present in that file ?
What happens if I follow the four steps outlined here to remove the volume from the node _BUT_, I do have another volume present in the cluster. It too is a test volume. Neither one has any data on them. So, data loss is not an issue. Running those four steps will remove the volume from your cluster . If
This is a multi-part message in MIME format. --------------2A81CE619084AC4F617005FF Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit On 04/11/2017 11:28 PM, Precht, Andrew wrote: the volumes what you have are test volumes you could just follow the steps outlined to delete them (since you are not able to delete from UI) and bring back the cluster into a normal state.
------------------------------------------------------------------------ *From:* knarra <knarra@redhat.com> *Sent:* Tuesday, April 11, 2017 10:32:27 AM *To:* Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:
Adding some people
Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht@sjlibrary.org <mailto:Andrew.Precht@sjlibrary.org>> ha scritto:
Hi Ovirt users, Im a newbie to oVirt and Im having trouble deleting a test gluster volume. The nodes are 4.1.1 and the engine is 4.1.0
When I try to remove the test volume, I click Remove, the dialog box prompting to confirm the deletion pops up and after I click OK, the dialog box changes to show a little spinning wheel and then it disappears. In the end the volume is still there.
with the latest version of glusterfs & ovirt we do not see any issue with deleting a volume. Can you please check /var/log/glusterfs/glusterd.log file if there is any error present?
The test volume was distributed with two host members. One of the hosts I was able to remove from the volume by removing the host form the cluster. When I try to remove the remaining host in the volume, even with the Force Remove box ticked, I get this response: Cannot remove Host. Server having Gluster volume.
What to try next?
since you have already removed the volume from one host in the cluster and you still see it on another host you can do the following to remove the volume from another host.
1) Login to the host where the volume is present. 2) cd to /var/lib/glusterd/vols 3) rm -rf <vol_name> 4) Restart glusterd on that host.
And before doing the above make sure that you do not have any other volume present in the cluster.
Above steps should not be run on a production system as you might loose the volume and data.
Now removing the host from UI should succed.
P.S. Ive tried to join this user group several times in the past, with no response. Is it possible for me to join this group?
Regards, Andrew
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--------------2A81CE619084AC4F617005FF Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">On 04/11/2017 11:28 PM, Precht, Andrew wrote:<br> </div> <blockquote cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <meta http-equiv="Content-Type" content="text/html; charset=windows-1252"> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div>Hi all,</div> <div>The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. </div> <div>On the node I can not find /var/log/glusterfs/glusterd.log However, there is a /var/log/glusterfs/glustershd.log <br> </div> </div> </blockquote> can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? if yes, can you check if there is any error present in that file ?<span style="color: rgb(84, 84, 84); font-family: arial, sans-serif; font-size: small; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); display: inline !important; float: none;"></span> <blockquote cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div><br> </div> <div>What happens if I follow the four steps outlined here to remove the volume from the node <u>BUT</u>, I do have another volume present in the cluster. It too is a test volume. Neither one has any data on them. So, data loss is not an issue.</div> </div> </blockquote> Running those four steps will remove the volume from your cluster . If the volumes what you have are test volumes you could just follow the steps outlined to delete them (since you are not able to delete from UI) and bring back the cluster into a normal state.<br> <blockquote cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <br> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> knarra <a class="moz-txt-link-rfc2396E" href="mailto:knarra@redhat.com"><knarra@redhat.com></a><br> <b>Sent:</b> Tuesday, April 11, 2017 10:32:27 AM<br> <b>To:</b> Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <div class="moz-cite-prefix">On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:<br> </div> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div dir="auto">Adding some people</div> <div class="gmail_extra"><br> <div class="gmail_quote">Il 11/Apr/2017 19:06, "Precht, Andrew" <<a moz-do-not-send="true" href="mailto:Andrew.Precht@sjlibrary.org">Andrew.Precht@sjlibrary.org</a>> ha scritto:<br type="attribution"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> <div>Hi Ovirt users,</div> <div>Im a newbie to oVirt and Im having trouble deleting a test gluster volume. The nodes are 4.1.1 and the engine is 4.1.0</div> <div><br> </div> <div>When I try to remove the test volume, I click Remove, the dialog box prompting to confirm the deletion pops up and after I click OK, the dialog box changes to show a little spinning wheel and then it disappears. In the end the volume is still there.</div> </div> </div> </blockquote> </div> </div> </blockquote> with the latest version of glusterfs & ovirt we do not see any issue with deleting a volume. Can you please check /var/log/glusterfs/glusterd.log file if there is any error present? <br> <br> <br> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> The test volume was distributed with two host members. One of the hosts I was able to remove from the volume by removing the host form the cluster. When I try to remove the remaining host in the volume, even with the Force Remove box ticked, I get this response: Cannot remove Host. Server having Gluster volume. <div><br> </div> <div>What to try next?</div> </div> </div> </blockquote> </div> </div> </blockquote> since you have already removed the volume from one host in the cluster and you still see it on another host you can do the following to remove the volume from another host.<br> <br> 1) Login to the host where the volume is present.<br> 2) cd to /var/lib/glusterd/vols<br> 3) rm -rf <vol_name><br> 4) Restart glusterd on that host.<br> <br> And before doing the above make sure that you do not have any other volume present in the cluster.<br> <br> Above steps should not be run on a production system as you might loose the volume and data. <br> <br> Now removing the host from UI should succed.<br> <br> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> <div><br> </div> <div>P.S. Ive tried to join this user group several times in the past, with no response.</div> <div>Is it possible for me to join this group?</div> <div><br> </div> <div>Regards,</div> <div>Andrew</div> <br> </div> </div> </blockquote> </div> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <p><br> </p> </div> </blockquote> <p><br> </p> </body> </html> --------------2A81CE619084AC4F617005FF--
--_000_BY1PR09MB0824E15BF3EEC19B5B776EB5F5000BY1PR09MB0824namp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Here is an update=85 I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the node= that had the trouble volume (test1). I didn=92t see any errors. So, I ran = a tail -f on the log as I tried to remove the volume using the web UI. here= is what was appended: [2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:1474:__g= lusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:1537:__g= lusterd_handle_cli_get_volume] 0-management: Received get vol req The message "I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_c= li_list_friends] 0-glusterd: Received cli list req" repeated 6 times betwee= n [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536] The message "I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_c= li_get_volume] 0-management: Received get vol req" repeated 20 times betwee= n [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179] [2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:1474:__g= lusterd_handle_cli_list_friends] 0-glusterd: Received cli list req I=92m seeing that the timestamps on these log entries do not match the time= on the node. The next steps I stopped the glusterd service on the node with volume test1 I deleted it with: rm -rf /var/lib/glusterd/vols/test1 I started the glusterd service. After starting the gluster service back up, the directory /var/lib/glusterd= /vols/test1 reappears. I=92m guessing syncing with the other nodes? Is this because I have the Volume Option: auth allow * Do I need to remove the directory /var/lib/glusterd/vols/test1 on all nodes= in the cluster individually? thanks ________________________________ From: knarra <knarra@redhat.com> Sent: Tuesday, April 11, 2017 11:51:18 AM To: Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureini= k; Nir Soffer Cc: users Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol= ume On 04/11/2017 11:28 PM, Precht, Andrew wrote: Hi all, The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. On the node I can not find /var/log/glusterfs/glusterd.log However, there i= s a /var/log/glusterfs/glustershd.log can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? = if yes, can you check if there is any error present in that file ? What happens if I follow the four steps outlined here to remove the volume = from the node BUT, I do have another volume present in the cluster. It too = is a test volume. Neither one has any data on them. So, data loss is not an= issue. Running those four steps will remove the volume from your cluster . If the = volumes what you have are test volumes you could just follow the steps outl= ined to delete them (since you are not able to delete from UI) and bring ba= ck the cluster into a normal state. ________________________________ From: knarra <knarra@redhat.com><mailto:knarra@redhat.com> Sent: Tuesday, April 11, 2017 10:32:27 AM To: Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureini= k; Nir Soffer Cc: users Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol= ume On 04/11/2017 10:44 PM, Sandro Bonazzola wrote: Adding some people Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht@sjlibrary.org<mailto:= Andrew.Precht@sjlibrary.org>> ha scritto: Hi Ovirt users, I=92m a newbie to oVirt and I=92m having trouble deleting a test gluster vo= lume. The nodes are 4.1.1 and the engine is 4.1.0 When I try to remove the test volume, I click Remove, the dialog box prompt= ing to confirm the deletion pops up and after I click OK, the dialog box ch= anges to show a little spinning wheel and then it disappears. In the end th= e volume is still there. with the latest version of glusterfs & ovirt we do not see any issue with d= eleting a volume. Can you please check /var/log/glusterfs/glusterd.log file= if there is any error present? The test volume was distributed with two host members. One of the hosts I w= as able to remove from the volume by removing the host form the cluster. Wh= en I try to remove the remaining host in the volume, even with the =93Force= Remove=94 box ticked, I get this response: Cannot remove Host. Server havi= ng Gluster volume. What to try next? since you have already removed the volume from one host in the cluster and = you still see it on another host you can do the following to remove the vol= ume from another host. 1) Login to the host where the volume is present. 2) cd to /var/lib/glusterd/vols 3) rm -rf <vol_name> 4) Restart glusterd on that host. And before doing the above make sure that you do not have any other volume = present in the cluster. Above steps should not be run on a production system as you might loose the= volume and data. Now removing the host from UI should succed. P.S. I=92ve tried to join this user group several times in the past, with n= o response. Is it possible for me to join this group? Regards, Andrew _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users --_000_BY1PR09MB0824E15BF3EEC19B5B776EB5F5000BY1PR09MB0824namp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> </head> <body bgcolor=3D"#FFFFFF" text=3D"#000000"> <style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi= n-bottom:0;} --></style> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <p></p> <div>Here is an update=85</div> <div><br> </div> <div>I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the= node that had the trouble volume (test1). I didn=92t see any errors. So, I= ran a tail -f on the log as I tried to remove the volume using the web UI.= here is what was appended:</div> <div><br> </div> <div>[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:147= 4:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</di= v> <div>[2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:153= 7:__glusterd_handle_cli_get_volume] 0-management: Received get vol req</div=
<div>The message "I [MSGID: 106487] [glusterd-handler.c:1474:__gluster= d_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated= 6 times between [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.5965= 36]</div> <div>The message "I [MSGID: 106488] [glusterd-handler.c:1537:__gluster= d_handle_cli_get_volume] 0-management: Received get vol req" repeated = 20 times between [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.0821= 79]</div> <div>[2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:147= 4:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</di= v> <div><br> </div> <div>I=92m seeing that the timestamps on these log entries do not match the= time on the node.</div> <div><br> </div> <div>The next steps</div> <div>I stopped the glusterd service on the node with volume test1</div> <div>I deleted it with: rm -rf /var/lib/glusterd/vols/test1</div> <div>I started the glusterd service.</div> <div><br> </div> <div>After starting the gluster service back up, the directory /var/lib/glu= sterd/vols/test1 reappears.</div> <div>I=92m guessing syncing with the other nodes?</div> <div>Is this because I have the Volume Option: auth allow *</div> <div>Do I need to remove the directory /var/lib/glusterd/vols/test1 on all = nodes in the cluster individually?</div> <div><br> </div> <div>thanks</div> <br> <p></p> </div> <hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"> <div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" st= yle=3D"font-size:11pt" color=3D"#000000"><b>From:</b> knarra <knarra@red= hat.com><br> <b>Sent:</b> Tuesday, April 11, 2017 11:51:18 AM<br> <b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon = Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus= ter volume</font> <div> </div> </div> <div> <div class=3D"moz-cite-prefix">On 04/11/2017 11:28 PM, Precht, Andrew wrote= :<br> </div> <blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082= 4.namprd09.prod.outlook.com" type=3D"cite"> <style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi= n-bottom:0;} --></style> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <div>Hi all,</div> <div>The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. </div> <div>On the node I can not find /var/log/glusterfs/glusterd.log However, th= ere is a /var/log/glusterfs/glustershd.log <br> </div> </div> </blockquote> can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? = if yes, can you check if there is any error present in that file ?<span sty= le=3D"color: rgb(84, 84, 84); font-family: arial, sans-serif; font-size: small; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); display: inline !important; float: none;"></span> <blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082= 4.namprd09.prod.outlook.com" type=3D"cite"> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <div><br> </div> <div>What happens if I follow the four steps outlined here to remove the vo= lume from the node <u>BUT</u>, I do have another volume present in the cluster. It too is a te= st volume. Neither one has any data on them. So, data loss is not an issue.= </div> </div> </blockquote> Running those four steps will remove the volume from your cluster . If the = volumes what you have are test volumes you could just follow the steps outl= ined to delete them (since you are not able to delete from UI) and bring ba= ck the cluster into a normal state.<br> <blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082= 4.namprd09.prod.outlook.com" type=3D"cite"> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <br> </div> <hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"> <div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color= =3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> knarra <a class=3D"moz-txt-link-rfc2396E" href=3D"mailto:knarra@redhat.com"><kn= arra@redhat.com></a><br> <b>Sent:</b> Tuesday, April 11, 2017 10:32:27 AM<br> <b>To:</b> Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon = Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus= ter volume</font> <div> </div> </div> <div> <div class=3D"moz-cite-prefix">On 04/11/2017 10:44 PM, Sandro Bonazzola wro= te:<br> </div> <blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA= bkkQ@mail.gmail.com" type=3D"cite"> <div dir=3D"auto">Adding some people</div> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote">Il 11/Apr/2017 19:06, "Precht, Andrew"= <<a moz-do-not-send=3D"true" href=3D"mailto:Andrew.Precht@sjlibrary.org= ">Andrew.Precht@sjlibrary.org</a>> ha scritto:<br type=3D"attribution"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir=3D"ltr"> <div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12= pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr= "> <div>Hi Ovirt users,</div> <div>I=92m a newbie to oVirt and I=92m having trouble deleting a test glust= er volume. The nodes are 4.1.1 and the engine is 4.1.0</div> <div><br> </div> <div>When I try to remove the test volume, I click Remove, the dialog box p= rompting to confirm the deletion pops up and after I click OK, the dialog b= ox changes to show a little spinning wheel and then it disappears. In the e= nd the volume is still there.</div> </div> </div> </blockquote> </div> </div> </blockquote> with the latest version of glusterfs & ovirt we do not see any issue wi= th deleting a volume. Can you please check /var/log/glusterfs/glusterd.log = file if there is any error present? <br> <br> <br> <blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA= bkkQ@mail.gmail.com" type=3D"cite"> <div class=3D"gmail_extra"> <div class=3D"gmail_quote"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir=3D"ltr"> <div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12= pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr= "> The test volume was distributed with two host members. One of the hosts I w= as able to remove from the volume by removing the host form the cluster. Wh= en I try to remove the remaining host in the volume, even with the =93Force= Remove=94 box ticked, I get this response: Cannot remove Host. Server having Gluster volume. <div><br> </div> <div>What to try next?</div> </div> </div> </blockquote> </div> </div> </blockquote> since you have already removed the volume from one host in the cluster and = you still see it on another host you can do the following to remove the vol= ume from another host.<br> <br> 1) Login to the host where the volume is present.<br> 2) cd to /var/lib/glusterd/vols<br> 3) rm -rf <vol_name><br> 4) Restart glusterd on that host.<br> <br> And before doing the above make sure that you do not have any other volume = present in the cluster.<br> <br> Above steps should not be run on a production system as you might loose the= volume and data. <br> <br> Now removing the host from UI should succed.<br> <br> <blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA= bkkQ@mail.gmail.com" type=3D"cite"> <div class=3D"gmail_extra"> <div class=3D"gmail_quote"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir=3D"ltr"> <div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12= pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr= "> <div><br> </div> <div>P.S. I=92ve tried to join this user group several times in the past, w= ith no response.</div> <div>Is it possible for me to join this group?</div> <div><br> </div> <div>Regards,</div> <div>Andrew</div> <br> </div> </div> </blockquote> </div> </div> <br> <fieldset class=3D"mimeAttachmentHeader"></fieldset> <br> <pre wrap=3D"">_______________________________________________ Users mailing list <a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"mail= to:Users@ovirt.org">Users@ovirt.org</a> <a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext" href=3D"http://= lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/list= info/users</a> </pre> </blockquote> <p><br> </p> </div> </blockquote> <p><br> </p> </div> </body> </html> --_000_BY1PR09MB0824E15BF3EEC19B5B776EB5F5000BY1PR09MB0824namp_--
--_000_BY1PR09MB0824604879A7F0FED17C039FF5000BY1PR09MB0824namp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable The plot thickens=85 I put all hosts in the cluster into maintenance mode, with the Stop Gluster= service checkbox checked. I then deleted the /var/lib/glusterd/vols/test1 = directory on all hosts. I then took the host that the test1 volume was on o= ut of maintenance mode. Then I tried to remove the test1 volume from within= the web UI. With no luck, I got the message: Could not delete Gluster Volu= me test1 on cluster 8000-1. I went back and checked all host for the test1 directory, it is not on any = host. Yet I still can=92t remove it=85 Any suggestions? ________________________________ From: Precht, Andrew Sent: Tuesday, April 11, 2017 1:15:22 PM To: knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir S= offer Cc: users Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol= ume Here is an update=85 I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the node= that had the trouble volume (test1). I didn=92t see any errors. So, I ran = a tail -f on the log as I tried to remove the volume using the web UI. here= is what was appended: [2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:1474:__g= lusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:1537:__g= lusterd_handle_cli_get_volume] 0-management: Received get vol req The message "I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_c= li_list_friends] 0-glusterd: Received cli list req" repeated 6 times betwee= n [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536] The message "I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_c= li_get_volume] 0-management: Received get vol req" repeated 20 times betwee= n [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179] [2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:1474:__g= lusterd_handle_cli_list_friends] 0-glusterd: Received cli list req I=92m seeing that the timestamps on these log entries do not match the time= on the node. The next steps I stopped the glusterd service on the node with volume test1 I deleted it with: rm -rf /var/lib/glusterd/vols/test1 I started the glusterd service. After starting the gluster service back up, the directory /var/lib/glusterd= /vols/test1 reappears. I=92m guessing syncing with the other nodes? Is this because I have the Volume Option: auth allow * Do I need to remove the directory /var/lib/glusterd/vols/test1 on all nodes= in the cluster individually? thanks ________________________________ From: knarra <knarra@redhat.com> Sent: Tuesday, April 11, 2017 11:51:18 AM To: Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureini= k; Nir Soffer Cc: users Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol= ume On 04/11/2017 11:28 PM, Precht, Andrew wrote: Hi all, The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. On the node I can not find /var/log/glusterfs/glusterd.log However, there i= s a /var/log/glusterfs/glustershd.log can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? = if yes, can you check if there is any error present in that file ? What happens if I follow the four steps outlined here to remove the volume = from the node BUT, I do have another volume present in the cluster. It too = is a test volume. Neither one has any data on them. So, data loss is not an= issue. Running those four steps will remove the volume from your cluster . If the = volumes what you have are test volumes you could just follow the steps outl= ined to delete them (since you are not able to delete from UI) and bring ba= ck the cluster into a normal state. ________________________________ From: knarra <knarra@redhat.com><mailto:knarra@redhat.com> Sent: Tuesday, April 11, 2017 10:32:27 AM To: Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureini= k; Nir Soffer Cc: users Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol= ume On 04/11/2017 10:44 PM, Sandro Bonazzola wrote: Adding some people Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht@sjlibrary.org<mailto:= Andrew.Precht@sjlibrary.org>> ha scritto: Hi Ovirt users, I=92m a newbie to oVirt and I=92m having trouble deleting a test gluster vo= lume. The nodes are 4.1.1 and the engine is 4.1.0 When I try to remove the test volume, I click Remove, the dialog box prompt= ing to confirm the deletion pops up and after I click OK, the dialog box ch= anges to show a little spinning wheel and then it disappears. In the end th= e volume is still there. with the latest version of glusterfs & ovirt we do not see any issue with d= eleting a volume. Can you please check /var/log/glusterfs/glusterd.log file= if there is any error present? The test volume was distributed with two host members. One of the hosts I w= as able to remove from the volume by removing the host form the cluster. Wh= en I try to remove the remaining host in the volume, even with the =93Force= Remove=94 box ticked, I get this response: Cannot remove Host. Server havi= ng Gluster volume. What to try next? since you have already removed the volume from one host in the cluster and = you still see it on another host you can do the following to remove the vol= ume from another host. 1) Login to the host where the volume is present. 2) cd to /var/lib/glusterd/vols 3) rm -rf <vol_name> 4) Restart glusterd on that host. And before doing the above make sure that you do not have any other volume = present in the cluster. Above steps should not be run on a production system as you might loose the= volume and data. Now removing the host from UI should succed. P.S. I=92ve tried to join this user group several times in the past, with n= o response. Is it possible for me to join this group? Regards, Andrew _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users --_000_BY1PR09MB0824604879A7F0FED17C039FF5000BY1PR09MB0824namp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> </head> <body bgcolor=3D"#FFFFFF" text=3D"#000000"> <style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi= n-bottom:0;} --></style> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <p></p> <div>The plot thickens=85</div> <div>I put all hosts in the cluster into maintenance mode, with the Stop Gl= uster service checkbox checked. I then deleted the /var/lib/glusterd/vols/t= est1 directory on all hosts. I then took the host that the test1 volume was= on out of maintenance mode. Then I tried to remove the test1 volume from within the web UI. With no luck, I= got the message: Could not delete Gluster Volume test1 on cluster 8000-1.<= /div> <div><br> </div> <div>I went back and checked all host for the test1 directory, it is not on= any host. Yet I still can=92t remove it=85</div> <div><br> </div> <div>Any suggestions?</div> <br> <p></p> </div> <hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"> <div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" st= yle=3D"font-size:11pt" color=3D"#000000"><b>From:</b> Precht, Andrew<br> <b>Sent:</b> Tuesday, April 11, 2017 1:15:22 PM<br> <b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik= ; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus= ter volume</font> <div> </div> </div> <div><style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;= margin-bottom:0;} --></style> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <p></p> <div>Here is an update=85</div> <div><br> </div> <div>I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the= node that had the trouble volume (test1). I didn=92t see any errors. So, I= ran a tail -f on the log as I tried to remove the volume using the web UI.= here is what was appended:</div> <div><br> </div> <div>[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:147= 4:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</di= v> <div>[2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:153= 7:__glusterd_handle_cli_get_volume] 0-management: Received get vol req</div=
<div>The message "I [MSGID: 106487] [glusterd-handler.c:1474:__gluster= d_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated= 6 times between [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.5965= 36]</div> <div>The message "I [MSGID: 106488] [glusterd-handler.c:1537:__gluster= d_handle_cli_get_volume] 0-management: Received get vol req" repeated = 20 times between [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.0821= 79]</div> <div>[2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:147= 4:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</di= v> <div><br> </div> <div>I=92m seeing that the timestamps on these log entries do not match the= time on the node.</div> <div><br> </div> <div>The next steps</div> <div>I stopped the glusterd service on the node with volume test1</div> <div>I deleted it with: rm -rf /var/lib/glusterd/vols/test1</div> <div>I started the glusterd service.</div> <div><br> </div> <div>After starting the gluster service back up, the directory /var/lib/glu= sterd/vols/test1 reappears.</div> <div>I=92m guessing syncing with the other nodes?</div> <div>Is this because I have the Volume Option: auth allow *</div> <div>Do I need to remove the directory /var/lib/glusterd/vols/test1 on all = nodes in the cluster individually?</div> <div><br> </div> <div>thanks</div> <br> <p></p> </div> <hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"> <div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" st= yle=3D"font-size:11pt" color=3D"#000000"><b>From:</b> knarra <knarra@red= hat.com><br> <b>Sent:</b> Tuesday, April 11, 2017 11:51:18 AM<br> <b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon = Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus= ter volume</font> <div> </div> </div> <div> <div class=3D"moz-cite-prefix">On 04/11/2017 11:28 PM, Precht, Andrew wrote= :<br> </div> <blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082= 4.namprd09.prod.outlook.com" type=3D"cite"> <style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi= n-bottom:0;} --></style> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <div>Hi all,</div> <div>The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. </div> <div>On the node I can not find /var/log/glusterfs/glusterd.log However, th= ere is a /var/log/glusterfs/glustershd.log <br> </div> </div> </blockquote> can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? = if yes, can you check if there is any error present in that file ?<span sty= le=3D"color: rgb(84, 84, 84); font-family: arial, sans-serif; font-size: small; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); display: inline !important; float: none;"></span> <blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082= 4.namprd09.prod.outlook.com" type=3D"cite"> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <div><br> </div> <div>What happens if I follow the four steps outlined here to remove the vo= lume from the node <u>BUT</u>, I do have another volume present in the cluster. It too is a te= st volume. Neither one has any data on them. So, data loss is not an issue.= </div> </div> </blockquote> Running those four steps will remove the volume from your cluster . If the = volumes what you have are test volumes you could just follow the steps outl= ined to delete them (since you are not able to delete from UI) and bring ba= ck the cluster into a normal state.<br> <blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082= 4.namprd09.prod.outlook.com" type=3D"cite"> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <br> </div> <hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"> <div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color= =3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> knarra <a class=3D"moz-txt-link-rfc2396E" href=3D"mailto:knarra@redhat.com"><kn= arra@redhat.com></a><br> <b>Sent:</b> Tuesday, April 11, 2017 10:32:27 AM<br> <b>To:</b> Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon = Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus= ter volume</font> <div> </div> </div> <div> <div class=3D"moz-cite-prefix">On 04/11/2017 10:44 PM, Sandro Bonazzola wro= te:<br> </div> <blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA= bkkQ@mail.gmail.com" type=3D"cite"> <div dir=3D"auto">Adding some people</div> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote">Il 11/Apr/2017 19:06, "Precht, Andrew"= <<a moz-do-not-send=3D"true" href=3D"mailto:Andrew.Precht@sjlibrary.org= ">Andrew.Precht@sjlibrary.org</a>> ha scritto:<br type=3D"attribution"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir=3D"ltr"> <div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12= pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr= "> <div>Hi Ovirt users,</div> <div>I=92m a newbie to oVirt and I=92m having trouble deleting a test glust= er volume. The nodes are 4.1.1 and the engine is 4.1.0</div> <div><br> </div> <div>When I try to remove the test volume, I click Remove, the dialog box p= rompting to confirm the deletion pops up and after I click OK, the dialog b= ox changes to show a little spinning wheel and then it disappears. In the e= nd the volume is still there.</div> </div> </div> </blockquote> </div> </div> </blockquote> with the latest version of glusterfs & ovirt we do not see any issue wi= th deleting a volume. Can you please check /var/log/glusterfs/glusterd.log = file if there is any error present? <br> <br> <br> <blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA= bkkQ@mail.gmail.com" type=3D"cite"> <div class=3D"gmail_extra"> <div class=3D"gmail_quote"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir=3D"ltr"> <div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12= pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr= "> The test volume was distributed with two host members. One of the hosts I w= as able to remove from the volume by removing the host form the cluster. Wh= en I try to remove the remaining host in the volume, even with the =93Force= Remove=94 box ticked, I get this response: Cannot remove Host. Server having Gluster volume. <div><br> </div> <div>What to try next?</div> </div> </div> </blockquote> </div> </div> </blockquote> since you have already removed the volume from one host in the cluster and = you still see it on another host you can do the following to remove the vol= ume from another host.<br> <br> 1) Login to the host where the volume is present.<br> 2) cd to /var/lib/glusterd/vols<br> 3) rm -rf <vol_name><br> 4) Restart glusterd on that host.<br> <br> And before doing the above make sure that you do not have any other volume = present in the cluster.<br> <br> Above steps should not be run on a production system as you might loose the= volume and data. <br> <br> Now removing the host from UI should succed.<br> <br> <blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA= bkkQ@mail.gmail.com" type=3D"cite"> <div class=3D"gmail_extra"> <div class=3D"gmail_quote"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir=3D"ltr"> <div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12= pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr= "> <div><br> </div> <div>P.S. I=92ve tried to join this user group several times in the past, w= ith no response.</div> <div>Is it possible for me to join this group?</div> <div><br> </div> <div>Regards,</div> <div>Andrew</div> <br> </div> </div> </blockquote> </div> </div> <br> <fieldset class=3D"mimeAttachmentHeader"></fieldset> <br> <pre wrap=3D"">_______________________________________________ Users mailing list <a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"mail= to:Users@ovirt.org">Users@ovirt.org</a> <a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext" href=3D"http://= lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/list= info/users</a> </pre> </blockquote> <p><br> </p> </div> </blockquote> <p><br> </p> </div> </div> </body> </html> --_000_BY1PR09MB0824604879A7F0FED17C039FF5000BY1PR09MB0824namp_--
--_000_BY1PR09MB0824E7FF50EC2611720F4D14F5000BY1PR09MB0824namp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable I just noticed this in the Alerts tab: Detected deletion of volume test1 on= cluster 8000-1, and deleted it from engine DB. Yet, It still shows in the web UI? ________________________________ From: Precht, Andrew Sent: Tuesday, April 11, 2017 2:39:31 PM To: knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir S= offer Cc: users Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol= ume The plot thickens=85 I put all hosts in the cluster into maintenance mode, with the Stop Gluster= service checkbox checked. I then deleted the /var/lib/glusterd/vols/test1 = directory on all hosts. I then took the host that the test1 volume was on o= ut of maintenance mode. Then I tried to remove the test1 volume from within= the web UI. With no luck, I got the message: Could not delete Gluster Volu= me test1 on cluster 8000-1. I went back and checked all host for the test1 directory, it is not on any = host. Yet I still can=92t remove it=85 Any suggestions? ________________________________ From: Precht, Andrew Sent: Tuesday, April 11, 2017 1:15:22 PM To: knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir S= offer Cc: users Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol= ume Here is an update=85 I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the node= that had the trouble volume (test1). I didn=92t see any errors. So, I ran = a tail -f on the log as I tried to remove the volume using the web UI. here= is what was appended: [2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:1474:__g= lusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:1537:__g= lusterd_handle_cli_get_volume] 0-management: Received get vol req The message "I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_c= li_list_friends] 0-glusterd: Received cli list req" repeated 6 times betwee= n [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536] The message "I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_c= li_get_volume] 0-management: Received get vol req" repeated 20 times betwee= n [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179] [2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:1474:__g= lusterd_handle_cli_list_friends] 0-glusterd: Received cli list req I=92m seeing that the timestamps on these log entries do not match the time= on the node. The next steps I stopped the glusterd service on the node with volume test1 I deleted it with: rm -rf /var/lib/glusterd/vols/test1 I started the glusterd service. After starting the gluster service back up, the directory /var/lib/glusterd= /vols/test1 reappears. I=92m guessing syncing with the other nodes? Is this because I have the Volume Option: auth allow * Do I need to remove the directory /var/lib/glusterd/vols/test1 on all nodes= in the cluster individually? thanks ________________________________ From: knarra <knarra@redhat.com> Sent: Tuesday, April 11, 2017 11:51:18 AM To: Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureini= k; Nir Soffer Cc: users Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol= ume On 04/11/2017 11:28 PM, Precht, Andrew wrote: Hi all, The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. On the node I can not find /var/log/glusterfs/glusterd.log However, there i= s a /var/log/glusterfs/glustershd.log can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? = if yes, can you check if there is any error present in that file ? What happens if I follow the four steps outlined here to remove the volume = from the node BUT, I do have another volume present in the cluster. It too = is a test volume. Neither one has any data on them. So, data loss is not an= issue. Running those four steps will remove the volume from your cluster . If the = volumes what you have are test volumes you could just follow the steps outl= ined to delete them (since you are not able to delete from UI) and bring ba= ck the cluster into a normal state. ________________________________ From: knarra <knarra@redhat.com><mailto:knarra@redhat.com> Sent: Tuesday, April 11, 2017 10:32:27 AM To: Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureini= k; Nir Soffer Cc: users Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol= ume On 04/11/2017 10:44 PM, Sandro Bonazzola wrote: Adding some people Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht@sjlibrary.org<mailto:= Andrew.Precht@sjlibrary.org>> ha scritto: Hi Ovirt users, I=92m a newbie to oVirt and I=92m having trouble deleting a test gluster vo= lume. The nodes are 4.1.1 and the engine is 4.1.0 When I try to remove the test volume, I click Remove, the dialog box prompt= ing to confirm the deletion pops up and after I click OK, the dialog box ch= anges to show a little spinning wheel and then it disappears. In the end th= e volume is still there. with the latest version of glusterfs & ovirt we do not see any issue with d= eleting a volume. Can you please check /var/log/glusterfs/glusterd.log file= if there is any error present? The test volume was distributed with two host members. One of the hosts I w= as able to remove from the volume by removing the host form the cluster. Wh= en I try to remove the remaining host in the volume, even with the =93Force= Remove=94 box ticked, I get this response: Cannot remove Host. Server havi= ng Gluster volume. What to try next? since you have already removed the volume from one host in the cluster and = you still see it on another host you can do the following to remove the vol= ume from another host. 1) Login to the host where the volume is present. 2) cd to /var/lib/glusterd/vols 3) rm -rf <vol_name> 4) Restart glusterd on that host. And before doing the above make sure that you do not have any other volume = present in the cluster. Above steps should not be run on a production system as you might loose the= volume and data. Now removing the host from UI should succed. P.S. I=92ve tried to join this user group several times in the past, with n= o response. Is it possible for me to join this group? Regards, Andrew _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users --_000_BY1PR09MB0824E7FF50EC2611720F4D14F5000BY1PR09MB0824namp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> </head> <body bgcolor=3D"#FFFFFF" text=3D"#000000"> <style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi= n-bottom:0;} --></style> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <p>I just <span>noticed </span>this in the Alerts tab: <span>Dete= cted deletion of volume test1 on cluster 8000-1, and deleted it from engine= DB.</span></p> <p><span>Yet, It still shows in the web UI?</span></p> </div> <hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"> <div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" st= yle=3D"font-size:11pt" color=3D"#000000"><b>From:</b> Precht, Andrew<br> <b>Sent:</b> Tuesday, April 11, 2017 2:39:31 PM<br> <b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik= ; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus= ter volume</font> <div> </div> </div> <div><style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;= margin-bottom:0;} --></style> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <p></p> <div>The plot thickens=85</div> <div>I put all hosts in the cluster into maintenance mode, with the Stop Gl= uster service checkbox checked. I then deleted the /var/lib/glusterd/vols/t= est1 directory on all hosts. I then took the host that the test1 volume was= on out of maintenance mode. Then I tried to remove the test1 volume from within the web UI. With no luck, I= got the message: Could not delete Gluster Volume test1 on cluster 8000-1.<= /div> <div><br> </div> <div>I went back and checked all host for the test1 directory, it is not on= any host. Yet I still can=92t remove it=85</div> <div><br> </div> <div>Any suggestions?</div> <br> <p></p> </div> <hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"> <div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" st= yle=3D"font-size:11pt" color=3D"#000000"><b>From:</b> Precht, Andrew<br> <b>Sent:</b> Tuesday, April 11, 2017 1:15:22 PM<br> <b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik= ; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus= ter volume</font> <div> </div> </div> <div><style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;= margin-bottom:0;} --></style> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <p></p> <div>Here is an update=85</div> <div><br> </div> <div>I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the= node that had the trouble volume (test1). I didn=92t see any errors. So, I= ran a tail -f on the log as I tried to remove the volume using the web UI.= here is what was appended:</div> <div><br> </div> <div>[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:147= 4:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</di= v> <div>[2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:153= 7:__glusterd_handle_cli_get_volume] 0-management: Received get vol req</div=
<div>The message "I [MSGID: 106487] [glusterd-handler.c:1474:__gluster= d_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated= 6 times between [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.5965= 36]</div> <div>The message "I [MSGID: 106488] [glusterd-handler.c:1537:__gluster= d_handle_cli_get_volume] 0-management: Received get vol req" repeated = 20 times between [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.0821= 79]</div> <div>[2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:147= 4:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</di= v> <div><br> </div> <div>I=92m seeing that the timestamps on these log entries do not match the= time on the node.</div> <div><br> </div> <div>The next steps</div> <div>I stopped the glusterd service on the node with volume test1</div> <div>I deleted it with: rm -rf /var/lib/glusterd/vols/test1</div> <div>I started the glusterd service.</div> <div><br> </div> <div>After starting the gluster service back up, the directory /var/lib/glu= sterd/vols/test1 reappears.</div> <div>I=92m guessing syncing with the other nodes?</div> <div>Is this because I have the Volume Option: auth allow *</div> <div>Do I need to remove the directory /var/lib/glusterd/vols/test1 on all = nodes in the cluster individually?</div> <div><br> </div> <div>thanks</div> <br> <p></p> </div> <hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"> <div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" st= yle=3D"font-size:11pt" color=3D"#000000"><b>From:</b> knarra <knarra@red= hat.com><br> <b>Sent:</b> Tuesday, April 11, 2017 11:51:18 AM<br> <b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon = Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus= ter volume</font> <div> </div> </div> <div> <div class=3D"moz-cite-prefix">On 04/11/2017 11:28 PM, Precht, Andrew wrote= :<br> </div> <blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082= 4.namprd09.prod.outlook.com" type=3D"cite"> <style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi= n-bottom:0;} --></style> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <div>Hi all,</div> <div>The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. </div> <div>On the node I can not find /var/log/glusterfs/glusterd.log However, th= ere is a /var/log/glusterfs/glustershd.log <br> </div> </div> </blockquote> can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? = if yes, can you check if there is any error present in that file ?<span sty= le=3D"color: rgb(84, 84, 84); font-family: arial, sans-serif; font-size: small; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); display: inline !important; float: none;"></span> <blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082= 4.namprd09.prod.outlook.com" type=3D"cite"> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <div><br> </div> <div>What happens if I follow the four steps outlined here to remove the vo= lume from the node <u>BUT</u>, I do have another volume present in the cluster. It too is a te= st volume. Neither one has any data on them. So, data loss is not an issue.= </div> </div> </blockquote> Running those four steps will remove the volume from your cluster . If the = volumes what you have are test volumes you could just follow the steps outl= ined to delete them (since you are not able to delete from UI) and bring ba= ck the cluster into a normal state.<br> <blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082= 4.namprd09.prod.outlook.com" type=3D"cite"> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> <br> </div> <hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"> <div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color= =3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> knarra <a class=3D"moz-txt-link-rfc2396E" href=3D"mailto:knarra@redhat.com"><kn= arra@redhat.com></a><br> <b>Sent:</b> Tuesday, April 11, 2017 10:32:27 AM<br> <b>To:</b> Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon = Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus= ter volume</font> <div> </div> </div> <div> <div class=3D"moz-cite-prefix">On 04/11/2017 10:44 PM, Sandro Bonazzola wro= te:<br> </div> <blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA= bkkQ@mail.gmail.com" type=3D"cite"> <div dir=3D"auto">Adding some people</div> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote">Il 11/Apr/2017 19:06, "Precht, Andrew"= <<a moz-do-not-send=3D"true" href=3D"mailto:Andrew.Precht@sjlibrary.org= ">Andrew.Precht@sjlibrary.org</a>> ha scritto:<br type=3D"attribution"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir=3D"ltr"> <div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12= pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr= "> <div>Hi Ovirt users,</div> <div>I=92m a newbie to oVirt and I=92m having trouble deleting a test glust= er volume. The nodes are 4.1.1 and the engine is 4.1.0</div> <div><br> </div> <div>When I try to remove the test volume, I click Remove, the dialog box p= rompting to confirm the deletion pops up and after I click OK, the dialog b= ox changes to show a little spinning wheel and then it disappears. In the e= nd the volume is still there.</div> </div> </div> </blockquote> </div> </div> </blockquote> with the latest version of glusterfs & ovirt we do not see any issue wi= th deleting a volume. Can you please check /var/log/glusterfs/glusterd.log = file if there is any error present? <br> <br> <br> <blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA= bkkQ@mail.gmail.com" type=3D"cite"> <div class=3D"gmail_extra"> <div class=3D"gmail_quote"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir=3D"ltr"> <div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12= pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr= "> The test volume was distributed with two host members. One of the hosts I w= as able to remove from the volume by removing the host form the cluster. Wh= en I try to remove the remaining host in the volume, even with the =93Force= Remove=94 box ticked, I get this response: Cannot remove Host. Server having Gluster volume. <div><br> </div> <div>What to try next?</div> </div> </div> </blockquote> </div> </div> </blockquote> since you have already removed the volume from one host in the cluster and = you still see it on another host you can do the following to remove the vol= ume from another host.<br> <br> 1) Login to the host where the volume is present.<br> 2) cd to /var/lib/glusterd/vols<br> 3) rm -rf <vol_name><br> 4) Restart glusterd on that host.<br> <br> And before doing the above make sure that you do not have any other volume = present in the cluster.<br> <br> Above steps should not be run on a production system as you might loose the= volume and data. <br> <br> Now removing the host from UI should succed.<br> <br> <blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA= bkkQ@mail.gmail.com" type=3D"cite"> <div class=3D"gmail_extra"> <div class=3D"gmail_quote"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir=3D"ltr"> <div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12= pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr= "> <div><br> </div> <div>P.S. I=92ve tried to join this user group several times in the past, w= ith no response.</div> <div>Is it possible for me to join this group?</div> <div><br> </div> <div>Regards,</div> <div>Andrew</div> <br> </div> </div> </blockquote> </div> </div> <br> <fieldset class=3D"mimeAttachmentHeader"></fieldset> <br> <pre wrap=3D"">_______________________________________________ Users mailing list <a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"mail= to:Users@ovirt.org">Users@ovirt.org</a> <a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext" href=3D"http://= lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/list= info/users</a> </pre> </blockquote> <p><br> </p> </div> </blockquote> <p><br> </p> </div> </div> </div> </body> </html> --_000_BY1PR09MB0824E7FF50EC2611720F4D14F5000BY1PR09MB0824namp_--
This is a multi-part message in MIME format. --------------FBCB929546CA281A03A4314D Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit On 04/12/2017 03:35 AM, Precht, Andrew wrote:
I just noticed this in the Alerts tab: Detected deletion of volume test1 on cluster 8000-1, and deleted it from engine DB.
Yet, It still shows in the web UI?
Any errors in ovirt-engine.log file ? if the volume is deleted from db ideally it should be deleted from UI too. Can you go to gluster nodes and check for the following: 1) gluster volume list -> should not return anything since you have deleted the volumes. 2) gluster peer status -> on all the nodes should show that all the peers are in connected state. can you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster log and capture the error messages when you try deleting the volume from UI? Log what you have pasted in the previous mail only gives info and i could not get any details from that on why volume delete is failing
------------------------------------------------------------------------ *From:* Precht, Andrew *Sent:* Tuesday, April 11, 2017 2:39:31 PM *To:* knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume The plot thickens I put all hosts in the cluster into maintenance mode, with the Stop Gluster service checkbox checked. I then deleted the /var/lib/glusterd/vols/test1 directory on all hosts. I then took the host that the test1 volume was on out of maintenance mode. Then I tried to remove the test1 volume from within the web UI. With no luck, I got the message: Could not delete Gluster Volume test1 on cluster 8000-1.
I went back and checked all host for the test1 directory, it is not on any host. Yet I still cant remove it
Any suggestions?
------------------------------------------------------------------------ *From:* Precht, Andrew *Sent:* Tuesday, April 11, 2017 1:15:22 PM *To:* knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume Here is an update
I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the node that had the trouble volume (test1). I didnt see any errors. So, I ran a tail -f on the log as I tried to remove the volume using the web UI. here is what was appended:
[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req The message "I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 6 times between [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536] The message "I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 20 times between [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179] [2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
Im seeing that the timestamps on these log entries do not match the time on the node.
The next steps I stopped the glusterd service on the node with volume test1 I deleted it with: rm -rf /var/lib/glusterd/vols/test1 I started the glusterd service.
After starting the gluster service back up, the directory /var/lib/glusterd/vols/test1 reappears. Im guessing syncing with the other nodes? Is this because I have the Volume Option: auth allow * Do I need to remove the directory /var/lib/glusterd/vols/test1 on all nodes in the cluster individually?
thanks
Hi all, The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. On the node I can not find /var/log/glusterfs/glusterd.log However, there is a /var/log/glusterfs/glustershd.log can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? if yes, can you check if there is any error present in that file ?
What happens if I follow the four steps outlined here to remove the volume from the node _BUT_, I do have another volume present in the cluster. It too is a test volume. Neither one has any data on them. So, data loss is not an issue. Running those four steps will remove the volume from your cluster . If
------------------------------------------------------------------------ *From:* knarra <knarra@redhat.com> *Sent:* Tuesday, April 11, 2017 11:51:18 AM *To:* Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume On 04/11/2017 11:28 PM, Precht, Andrew wrote: the volumes what you have are test volumes you could just follow the steps outlined to delete them (since you are not able to delete from UI) and bring back the cluster into a normal state.
------------------------------------------------------------------------ *From:* knarra <knarra@redhat.com> *Sent:* Tuesday, April 11, 2017 10:32:27 AM *To:* Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:
Adding some people
Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht@sjlibrary.org <mailto:Andrew.Precht@sjlibrary.org>> ha scritto:
Hi Ovirt users, Im a newbie to oVirt and Im having trouble deleting a test gluster volume. The nodes are 4.1.1 and the engine is 4.1.0
When I try to remove the test volume, I click Remove, the dialog box prompting to confirm the deletion pops up and after I click OK, the dialog box changes to show a little spinning wheel and then it disappears. In the end the volume is still there.
with the latest version of glusterfs & ovirt we do not see any issue with deleting a volume. Can you please check /var/log/glusterfs/glusterd.log file if there is any error present?
The test volume was distributed with two host members. One of the hosts I was able to remove from the volume by removing the host form the cluster. When I try to remove the remaining host in the volume, even with the Force Remove box ticked, I get this response: Cannot remove Host. Server having Gluster volume.
What to try next?
since you have already removed the volume from one host in the cluster and you still see it on another host you can do the following to remove the volume from another host.
1) Login to the host where the volume is present. 2) cd to /var/lib/glusterd/vols 3) rm -rf <vol_name> 4) Restart glusterd on that host.
And before doing the above make sure that you do not have any other volume present in the cluster.
Above steps should not be run on a production system as you might loose the volume and data.
Now removing the host from UI should succed.
P.S. Ive tried to join this user group several times in the past, with no response. Is it possible for me to join this group?
Regards, Andrew
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--------------FBCB929546CA281A03A4314D Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">On 04/12/2017 03:35 AM, Precht, Andrew wrote:<br> </div> <blockquote cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <meta http-equiv="Content-Type" content="text/html; charset=windows-1252"> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p>I just <span>noticed </span>this in the Alerts tab: <span>Detected deletion of volume test1 on cluster 8000-1, and deleted it from engine DB.</span><br> </p> </div> </blockquote> <blockquote cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p><span>Yet, It still shows in the web UI?</span></p> </div> </blockquote> Any errors in ovirt-engine.log file ? if the volume is deleted from db ideally it should be deleted from UI too. Can you go to gluster nodes and check for the following:<br> <br> 1) gluster volume list -> should not return anything since you have deleted the volumes.<br> <br> 2) gluster peer status -> on all the nodes should show that all the peers are in connected state.<br> <br> can you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster log and capture the error messages when you try deleting the volume from UI?<br> <br> Log what you have pasted in the previous mail only gives info and i could not get any details from that on why volume delete is failing<br> <br> <blockquote cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Precht, Andrew<br> <b>Sent:</b> Tuesday, April 11, 2017 2:39:31 PM<br> <b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div>The plot thickens </div> <div>I put all hosts in the cluster into maintenance mode, with the Stop Gluster service checkbox checked. I then deleted the /var/lib/glusterd/vols/test1 directory on all hosts. I then took the host that the test1 volume was on out of maintenance mode. Then I tried to remove the test1 volume from within the web UI. With no luck, I got the message: Could not delete Gluster Volume test1 on cluster 8000-1.</div> <div><br> </div> <div>I went back and checked all host for the test1 directory, it is not on any host. Yet I still cant remove it </div> <div><br> </div> <div>Any suggestions?</div> <br> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Precht, Andrew<br> <b>Sent:</b> Tuesday, April 11, 2017 1:15:22 PM<br> <b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div>Here is an update </div> <div><br> </div> <div>I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the node that had the trouble volume (test1). I didnt see any errors. So, I ran a tail -f on the log as I tried to remove the volume using the web UI. here is what was appended:</div> <div><br> </div> <div>[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</div> <div>[2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req</div> <div>The message "I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 6 times between [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536]</div> <div>The message "I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 20 times between [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179]</div> <div>[2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</div> <div><br> </div> <div>Im seeing that the timestamps on these log entries do not match the time on the node.</div> <div><br> </div> <div>The next steps</div> <div>I stopped the glusterd service on the node with volume test1</div> <div>I deleted it with: rm -rf /var/lib/glusterd/vols/test1</div> <div>I started the glusterd service.</div> <div><br> </div> <div>After starting the gluster service back up, the directory /var/lib/glusterd/vols/test1 reappears.</div> <div>Im guessing syncing with the other nodes?</div> <div>Is this because I have the Volume Option: auth allow *</div> <div>Do I need to remove the directory /var/lib/glusterd/vols/test1 on all nodes in the cluster individually?</div> <div><br> </div> <div>thanks</div> <br> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> knarra <a class="moz-txt-link-rfc2396E" href="mailto:knarra@redhat.com"><knarra@redhat.com></a><br> <b>Sent:</b> Tuesday, April 11, 2017 11:51:18 AM<br> <b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <div class="moz-cite-prefix">On 04/11/2017 11:28 PM, Precht, Andrew wrote:<br> </div> <blockquote cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div>Hi all,</div> <div>The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. </div> <div>On the node I can not find /var/log/glusterfs/glusterd.log However, there is a /var/log/glusterfs/glustershd.log <br> </div> </div> </blockquote> can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? if yes, can you check if there is any error present in that file ?<span style="color: rgb(84, 84, 84); font-family: arial, sans-serif; font-size: small; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); display: inline !important; float: none;"></span> <blockquote cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div><br> </div> <div>What happens if I follow the four steps outlined here to remove the volume from the node <u>BUT</u>, I do have another volume present in the cluster. It too is a test volume. Neither one has any data on them. So, data loss is not an issue.</div> </div> </blockquote> Running those four steps will remove the volume from your cluster . If the volumes what you have are test volumes you could just follow the steps outlined to delete them (since you are not able to delete from UI) and bring back the cluster into a normal state.<br> <blockquote cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <br> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> knarra <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:knarra@redhat.com"><knarra@redhat.com></a><br> <b>Sent:</b> Tuesday, April 11, 2017 10:32:27 AM<br> <b>To:</b> Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <div class="moz-cite-prefix">On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:<br> </div> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div dir="auto">Adding some people</div> <div class="gmail_extra"><br> <div class="gmail_quote">Il 11/Apr/2017 19:06, "Precht, Andrew" <<a moz-do-not-send="true" href="mailto:Andrew.Precht@sjlibrary.org">Andrew.Precht@sjlibrary.org</a>> ha scritto:<br type="attribution"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> <div>Hi Ovirt users,</div> <div>Im a newbie to oVirt and Im having trouble deleting a test gluster volume. The nodes are 4.1.1 and the engine is 4.1.0</div> <div><br> </div> <div>When I try to remove the test volume, I click Remove, the dialog box prompting to confirm the deletion pops up and after I click OK, the dialog box changes to show a little spinning wheel and then it disappears. In the end the volume is still there.</div> </div> </div> </blockquote> </div> </div> </blockquote> with the latest version of glusterfs & ovirt we do not see any issue with deleting a volume. Can you please check /var/log/glusterfs/glusterd.log file if there is any error present? <br> <br> <br> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> The test volume was distributed with two host members. One of the hosts I was able to remove from the volume by removing the host form the cluster. When I try to remove the remaining host in the volume, even with the Force Remove box ticked, I get this response: Cannot remove Host. Server having Gluster volume. <div><br> </div> <div>What to try next?</div> </div> </div> </blockquote> </div> </div> </blockquote> since you have already removed the volume from one host in the cluster and you still see it on another host you can do the following to remove the volume from another host.<br> <br> 1) Login to the host where the volume is present.<br> 2) cd to /var/lib/glusterd/vols<br> 3) rm -rf <vol_name><br> 4) Restart glusterd on that host.<br> <br> And before doing the above make sure that you do not have any other volume present in the cluster.<br> <br> Above steps should not be run on a production system as you might loose the volume and data. <br> <br> Now removing the host from UI should succed.<br> <br> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> <div><br> </div> <div>P.S. Ive tried to join this user group several times in the past, with no response.</div> <div>Is it possible for me to join this group?</div> <div><br> </div> <div>Regards,</div> <div>Andrew</div> <br> </div> </div> </blockquote> </div> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <p><br> </p> </div> </blockquote> <p><br> </p> </div> </div> </div> </blockquote> <p><br> </p> </body> </html> --------------FBCB929546CA281A03A4314D--
--_000_BY1PR09MB0824A056B7185A98CDF374C7F5030BY1PR09MB0824namp_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
Hi all,
You asked: Any errors in ovirt-engine.log file ?
Yes, In the engine.log this error is repeated about every 3 minutes:
2017-04-12 07:16:12,554-07 ERROR [org.ovirt.engine.core.bll.gluster.Gluster=
TasksSyncJob] (DefaultQuartzScheduler3) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7=
b8] Error updating tasks from CLI: org.ovirt.engine.core.common.errors.Engi=
neException: EngineException: Command execution failed
error: Error : Request timed out
return code: 1 (Failed with error GlusterVolumeStatusAllFailedException and=
code 4161)
error: Error : Request timed out
1) gluster volume list -> Still shows the deleted volume (test1)
2) gluster peer status -> Shows one of the peers twice with different uuid=
=92s:
Hostname: 192.168.10.109
Uuid: 42fbb7de-8e6f-4159-a601-3f858fa65f6c
State: Peer in Cluster (Connected)
Hostname: 192.168.10.109
Uuid: e058babe-7f9d-49fe-a3ea-ccdc98d7e5b5
State: Peer in Cluster (Connected)
I tried a gluster volume stop test1, with this result: volume stop: test1: =
failed: Another transaction is in progress for test1. Please try again afte=
r sometime.
The etc-glusterfs-glusterd.vol.log shows no activity triggered by trying to=
remove the test1 volume from the UI.
The ovirt-engine.log shows this repeating many times, when trying to remove=
the test1 volume from the UI:
2017-04-12 07:57:38,049-07 INFO [org.ovirt.engine.core.bll.lock.InMemoryLo=
ckManager] (DefaultQuartzScheduler9) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8]=
Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks=3D'[b0e1b=
909-9a6a-49dc-8e20-3a027218f7e1=3D<GLUSTER, ACTION_TYPE_FAILED_GLUSTER_OPER=
ATION_INPROGRESS>]', sharedLocks=3D'null'}'
Thanks much,
Andrew
________________________________
From: knarra <knarra@redhat.com>
Sent: Tuesday, April 11, 2017 11:10:04 PM
To: Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureini=
k; Nir Soffer
Cc: users
Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol=
ume
On 04/12/2017 03:35 AM, Precht, Andrew wrote:
I just noticed this in the Alerts tab: Detected deletion of volume test1 on=
cluster 8000-1, and deleted it from engine DB.
Yet, It still shows in the web UI?
Any errors in ovirt-engine.log file ? if the volume is deleted from db idea=
lly it should be deleted from UI too. Can you go to gluster nodes and chec=
k for the following:
1) gluster volume list -> should not return anything since you have deleted=
the volumes.
2) gluster peer status -> on all the nodes should show that all the peers a=
re in connected state.
can you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster log and =
capture the error messages when you try deleting the volume from UI?
Log what you have pasted in the previous mail only gives info and i could n=
ot get any details from that on why volume delete is failing
________________________________
From: Precht, Andrew
Sent: Tuesday, April 11, 2017 2:39:31 PM
To: knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir S=
offer
Cc: users
Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol=
ume
The plot thickens=85
I put all hosts in the cluster into maintenance mode, with the Stop Gluster=
service checkbox checked. I then deleted the /var/lib/glusterd/vols/test1 =
directory on all hosts. I then took the host that the test1 volume was on o=
ut of maintenance mode. Then I tried to remove the test1 volume from within=
the web UI. With no luck, I got the message: Could not delete Gluster Volu=
me test1 on cluster 8000-1.
I went back and checked all host for the test1 directory, it is not on any =
host. Yet I still can=92t remove it=85
Any suggestions?
________________________________
From: Precht, Andrew
Sent: Tuesday, April 11, 2017 1:15:22 PM
To: knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir S=
offer
Cc: users
Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol=
ume
Here is an update=85
I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the node=
that had the trouble volume (test1). I didn=92t see any errors. So, I ran =
a tail -f on the log as I tried to remove the volume using the web UI. here=
is what was appended:
[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:1474:__g=
lusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:1537:__g=
lusterd_handle_cli_get_volume] 0-management: Received get vol req
The message "I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_c=
li_list_friends] 0-glusterd: Received cli list req" repeated 6 times betwee=
n [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536]
The message "I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_c=
li_get_volume] 0-management: Received get vol req" repeated 20 times betwee=
n [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179]
[2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:1474:__g=
lusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
I=92m seeing that the timestamps on these log entries do not match the time=
on the node.
The next steps
I stopped the glusterd service on the node with volume test1
I deleted it with: rm -rf /var/lib/glusterd/vols/test1
I started the glusterd service.
After starting the gluster service back up, the directory /var/lib/glusterd=
/vols/test1 reappears.
I=92m guessing syncing with the other nodes?
Is this because I have the Volume Option: auth allow *
Do I need to remove the directory /var/lib/glusterd/vols/test1 on all nodes=
in the cluster individually?
thanks
________________________________
From: knarra <knarra@redhat.com><mailto:knarra@redhat.com>
Sent: Tuesday, April 11, 2017 11:51:18 AM
To: Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureini=
k; Nir Soffer
Cc: users
Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol=
ume
On 04/11/2017 11:28 PM, Precht, Andrew wrote:
Hi all,
The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7.
On the node I can not find /var/log/glusterfs/glusterd.log However, there i=
s a /var/log/glusterfs/glustershd.log
can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? =
if yes, can you check if there is any error present in that file ?
What happens if I follow the four steps outlined here to remove the volume =
from the node BUT, I do have another volume present in the cluster. It too =
is a test volume. Neither one has any data on them. So, data loss is not an=
issue.
Running those four steps will remove the volume from your cluster . If the =
volumes what you have are test volumes you could just follow the steps outl=
ined to delete them (since you are not able to delete from UI) and bring ba=
ck the cluster into a normal state.
________________________________
From: knarra <knarra@redhat.com><mailto:knarra@redhat.com>
Sent: Tuesday, April 11, 2017 10:32:27 AM
To: Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureini=
k; Nir Soffer
Cc: users
Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol=
ume
On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:
Adding some people
Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht@sjlibrary.org<mailto:=
Andrew.Precht@sjlibrary.org>> ha scritto:
Hi Ovirt users,
I=92m a newbie to oVirt and I=92m having trouble deleting a test gluster vo=
lume. The nodes are 4.1.1 and the engine is 4.1.0
When I try to remove the test volume, I click Remove, the dialog box prompt=
ing to confirm the deletion pops up and after I click OK, the dialog box ch=
anges to show a little spinning wheel and then it disappears. In the end th=
e volume is still there.
with the latest version of glusterfs & ovirt we do not see any issue with d=
eleting a volume. Can you please check /var/log/glusterfs/glusterd.log file=
if there is any error present?
The test volume was distributed with two host members. One of the hosts I w=
as able to remove from the volume by removing the host form the cluster. Wh=
en I try to remove the remaining host in the volume, even with the =93Force=
Remove=94 box ticked, I get this response: Cannot remove Host. Server havi=
ng Gluster volume.
What to try next?
since you have already removed the volume from one host in the cluster and =
you still see it on another host you can do the following to remove the vol=
ume from another host.
1) Login to the host where the volume is present.
2) cd to /var/lib/glusterd/vols
3) rm -rf <vol_name>
4) Restart glusterd on that host.
And before doing the above make sure that you do not have any other volume =
present in the cluster.
Above steps should not be run on a production system as you might loose the=
volume and data.
Now removing the host from UI should succed.
P.S. I=92ve tried to join this user group several times in the past, with n=
o response.
Is it possible for me to join this group?
Regards,
Andrew
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
--_000_BY1PR09MB0824A056B7185A98CDF374C7F5030BY1PR09MB0824namp_
Content-Type: text/html; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1=
252">
</head>
<body bgcolor=3D"#FFFFFF" text=3D"#000000">
<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style>
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<p>Hi all,</p>
<p><span id=3D"docs-internal-guid-44c81590-62b4-32c0-d6d1-82b627328ebb"></p=
>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;">You asked: Any errors=
in ovirt-engine.log file ?</span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;t=
ext-indent: 36pt;">
<span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33=
); vertical-align: baseline; white-space: pre-wrap;">Yes, In the engine.log=
this error is repeated about every 3 minutes:</span></p>
<br>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;">2017-04-12 07:16:12,5=
54-07 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob]
(DefaultQuartzScheduler3) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Error upd=
ating tasks from CLI: org.ovirt.engine.core.common.errors.EngineException: =
EngineException: Command execution failed</span><span style=3D"font-size: 1=
1.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline=
; white-space: pre-wrap;"><br class=3D"kix-line-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">error: Error : =
Request timed out</span><span style=3D"font-size: 11.5pt; font-family: Aria=
l; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;=
"><br class=3D"kix-line-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">return code: 1 =
(Failed with error GlusterVolumeStatusAllFailedException and code 4161)</sp=
an><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33,=
33); vertical-align: baseline; white-space: pre-wrap;"><br class=3D"kix-li=
ne-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">error: Error : =
Request timed out</span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;t=
ext-indent: 36pt;">
<span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33=
); vertical-align: baseline; white-space: pre-wrap;"><br class=3D"kix-line-=
break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">1) gluster volu=
me list -> Still shows the deleted volume (test1)</span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;t=
ext-indent: 36pt;">
<span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33=
); vertical-align: baseline; white-space: pre-wrap;"><br class=3D"kix-line-=
break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">2) gluster peer=
status -> Shows one of the peers twice with different uuid=92s:</span><=
/p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;">Hostname: 192.168.10.=
109</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(=
33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"><br class=3D=
"kix-line-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">Uuid: 42fbb7de-=
8e6f-4159-a601-3f858fa65f6c</span><span style=3D"font-size: 11.5pt; font-fa=
mily: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space:=
pre-wrap;"><br class=3D"kix-line-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">State: Peer in =
Cluster (Connected)</span><span style=3D"font-size: 11.5pt; font-family: Ar=
ial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wra=
p;"><br class=3D"kix-line-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;"><br class=3D"ki=
x-line-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">Hostname: 192.1=
68.10.109</span><span style=3D"font-size: 11.5pt; font-family: Arial; color=
: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"><br cl=
ass=3D"kix-line-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">Uuid: e058babe-=
7f9d-49fe-a3ea-ccdc98d7e5b5</span><span style=3D"font-size: 11.5pt; font-fa=
mily: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space:=
pre-wrap;"><br class=3D"kix-line-break">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">State: Peer in =
Cluster (Connected)</span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;"><br>
</span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"vertical-align: baseline;"><font color=3D"#212121" face=3D"=
Arial"><span style=3D"font-size: 11.5pt; white-space: pre-wrap;">I tried a =
gluster volume stop test1, with this
</span><span style=3D"font-size: 15.3333px; white-space: pre-wrap;">result<=
/span><span style=3D"font-size: 11.5pt; white-space: pre-wrap;">:
<span>volume stop: test1: failed: Another transaction is in progress for te=
st1. Please try again after sometime.</span></span></font></span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5=
pt; white-space: pre-wrap; text-indent: 36pt;"><br>
</span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5=
pt; white-space: pre-wrap; text-indent: 36pt;"><br>
</span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;"><span></span></span><=
span style=3D"color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt=
; white-space: pre-wrap; text-indent: 36pt;">The
etc-glusterfs-glusterd.vol.log shows no activity triggered by trying to re=
move the test1 volume from the UI</span><span style=3D"color: rgb(33, 33, 3=
3); font-family: Arial; font-size: 11.5pt; white-space: pre-wrap; text-inde=
nt: 36pt;">.</span></p>
<br>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;">The ovirt-engine.log =
shows this repeating many times, when
<span style=3D"color: rgb(33, 33, 33); font-family: Arial; font-size: 15.33=
33px; text-indent: 48px; white-space: pre-wrap;">
trying to remove the test1 volume from the UI</span>:</span></p>
<br>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;">2017-04-12 07:57:38,0=
49-07 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler9) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Failed to=
acquire lock and wait lock 'EngineLock:{exclusiveLocks=3D'[b0e1b909-9a6a-4=
9dc-8e20-3a027218f7e1=3D<GLUSTER, ACTION_TYPE_FAILED_GLUSTER_OPERATION_I=
NPROGRESS>]', sharedLocks=3D'null'}'</span><span style=3D"font-size: 11.=
5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; =
white-space: pre-wrap;"><br class=3D"kix-line-break">
<br class=3D"kix-line-break">
</span></p>
</span>Thanks much,
<p></p>
<p>Andrew</p>
</div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" st=
yle=3D"font-size:11pt" color=3D"#000000"><b>From:</b> knarra <knarra@red=
hat.com><br>
<b>Sent:</b> Tuesday, April 11, 2017 11:10:04 PM<br>
<b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon =
Mureinik; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus=
ter volume</font>
<div> </div>
</div>
<div>
<div class=3D"moz-cite-prefix">On 04/12/2017 03:35 AM, Precht, Andrew wrote=
:<br>
</div>
<blockquote cite=3D"mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style>
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<p>I just <span>noticed </span>this in the Alerts tab: <span>Dete=
cted deletion of volume test1 on cluster 8000-1, and deleted it from engine=
DB.</span><br>
</p>
</div>
</blockquote>
<blockquote cite=3D"mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<p><span>Yet, It still shows in the web UI?</span></p>
</div>
</blockquote>
Any errors in ovirt-engine.log file ? if the volume is deleted from db idea=
lly it should be deleted from UI too. Can you go to gluster nodes and=
check for the following:<br>
<br>
1) gluster volume list -> should not return anything since you have dele=
ted the volumes.<br>
<br>
2) gluster peer status -> on all the nodes should show that all the peer=
s are in connected state.<br>
<br>
can you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster log and =
capture the error messages when you try deleting the volume from UI?<br>
<br>
Log what you have pasted in the previous mail only gives info and i could n=
ot get any details from that on why volume delete is failing<br>
<br>
<blockquote cite=3D"mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
</div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color=
=3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> Precht, Andrew<br>
<b>Sent:</b> Tuesday, April 11, 2017 2:39:31 PM<br>
<b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik=
; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus=
ter volume</font>
<div> </div>
</div>
<div><style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;=
margin-bottom:0;} --></style>
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<div>The plot thickens=85</div>
<div>I put all hosts in the cluster into maintenance mode, with the Stop Gl=
uster service checkbox checked. I then deleted the /var/lib/glusterd/vols/t=
est1 directory on all hosts. I then took the host that the test1 volume was=
on out of maintenance mode. Then
I tried to remove the test1 volume from within the web UI. With no luck, I=
got the message: Could not delete Gluster Volume test1 on cluster 8000-1.<=
/div>
<div><br>
</div>
<div>I went back and checked all host for the test1 directory, it is not on=
any host. Yet I still can=92t remove it=85</div>
<div><br>
</div>
<div>Any suggestions?</div>
<br>
</div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color=
=3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> Precht, Andrew<br>
<b>Sent:</b> Tuesday, April 11, 2017 1:15:22 PM<br>
<b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik=
; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus=
ter volume</font>
<div> </div>
</div>
<div><style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;=
margin-bottom:0;} --></style>
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<div>Here is an update=85</div>
<div><br>
</div>
<div>I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the=
node that had the trouble volume (test1). I didn=92t see any errors. So, I=
ran a tail -f on the log as I tried to remove the volume using the web UI.=
here is what was appended:</div>
<div><br>
</div>
<div>[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:147=
4:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</di=
v>
<div>[2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:153=
7:__glusterd_handle_cli_get_volume] 0-management: Received get vol req</div=
>
<div>The message "I [MSGID: 106487] [glusterd-handler.c:1474:__gluster=
d_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated=
6 times between [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.5965=
36]</div>
<div>The message "I [MSGID: 106488] [glusterd-handler.c:1537:__gluster=
d_handle_cli_get_volume] 0-management: Received get vol req" repeated =
20 times between [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.0821=
79]</div>
<div>[2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:147=
4:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</di=
v>
<div><br>
</div>
<div>I=92m seeing that the timestamps on these log entries do not match the=
time on the node.</div>
<div><br>
</div>
<div>The next steps</div>
<div>I stopped the glusterd service on the node with volume test1</div>
<div>I deleted it with: rm -rf /var/lib/glusterd/vols/test1</div>
<div>I started the glusterd service.</div>
<div><br>
</div>
<div>After starting the gluster service back up, the directory /var/lib/glu=
sterd/vols/test1 reappears.</div>
<div>I=92m guessing syncing with the other nodes?</div>
<div>Is this because I have the Volume Option: auth allow *</div>
<div>Do I need to remove the directory /var/lib/glusterd/vols/test1 on all =
nodes in the cluster individually?</div>
<div><br>
</div>
<div>thanks</div>
<br>
</div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color=
=3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> knarra
<a class=3D"moz-txt-link-rfc2396E" href=3D"mailto:knarra@redhat.com"><kn=
arra@redhat.com></a><br>
<b>Sent:</b> Tuesday, April 11, 2017 11:51:18 AM<br>
<b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon =
Mureinik; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus=
ter volume</font>
<div> </div>
</div>
<div>
<div class=3D"moz-cite-prefix">On 04/11/2017 11:28 PM, Precht, Andrew wrote=
:<br>
</div>
<blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style>
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<div>Hi all,</div>
<div>The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. </div>
<div>On the node I can not find /var/log/glusterfs/glusterd.log However, th=
ere is a /var/log/glusterfs/glustershd.log
<br>
</div>
</div>
</blockquote>
can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? =
if yes, can you check if there is any error present in that file ?<span sty=
le=3D"color: rgb(84, 84, 84); font-family:
arial, sans-serif; font-size: small; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; orphans: 2;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); display: inline !important; float: none;"></span>
<blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<div><br>
</div>
<div>What happens if I follow the four steps outlined here to remove the vo=
lume from the node
<u>BUT</u>, I do have another volume present in the cluster. It too is a te=
st volume. Neither one has any data on them. So, data loss is not an issue.=
</div>
</div>
</blockquote>
Running those four steps will remove the volume from your cluster . If the =
volumes what you have are test volumes you could just follow the steps outl=
ined to delete them (since you are not able to delete from UI) and bring ba=
ck the cluster into a normal state.<br>
<blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<br>
</div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color=
=3D"#000000" face=3D"Calibri,
sans-serif"><b>From:</b> knarra
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-rfc2396E" href=3D"mailto:=
knarra@redhat.com">
<knarra@redhat.com></a><br>
<b>Sent:</b> Tuesday, April 11, 2017 10:32:27 AM<br>
<b>To:</b> Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon =
Mureinik; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus=
ter volume</font>
<div> </div>
</div>
<div>
<div class=3D"moz-cite-prefix">On 04/11/2017 10:44 PM, Sandro Bonazzola wro=
te:<br>
</div>
<blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA=
bkkQ@mail.gmail.com" type=3D"cite">
<div dir=3D"auto">Adding some people</div>
<div class=3D"gmail_extra"><br>
<div class=3D"gmail_quote">Il 11/Apr/2017 19:06, "Precht, Andrew"=
<<a moz-do-not-send=3D"true" href=3D"mailto:Andrew.Precht@sjlibrary.org=
">Andrew.Precht@sjlibrary.org</a>> ha scritto:<br type=3D"attribution">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir=3D"ltr">
<div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12=
pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr=
">
<div>Hi Ovirt users,</div>
<div>I=92m a newbie to oVirt and I=92m having trouble deleting a test glust=
er volume. The nodes are 4.1.1 and the engine is 4.1.0</div>
<div><br>
</div>
<div>When I try to remove the test volume, I click Remove, the dialog box p=
rompting to confirm the deletion pops up and after I click OK, the dialog b=
ox changes to show a little spinning wheel and then it disappears. In the e=
nd the volume is still there.</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
with the latest version of glusterfs & ovirt we do not see any issue wi=
th deleting a volume. Can you please check /var/log/glusterfs/glusterd.log =
file if there is any error present?
<br>
<br>
<br>
<blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA=
bkkQ@mail.gmail.com" type=3D"cite">
<div class=3D"gmail_extra">
<div class=3D"gmail_quote">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir=3D"ltr">
<div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12=
pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr=
">
The test volume was distributed with two host members. One of the hosts I w=
as able to remove from the volume by removing the host form the cluster. Wh=
en I try to remove the remaining host in the volume, even with the =93Force=
Remove=94 box ticked, I get this response:
Cannot remove Host. Server having Gluster volume.
<div><br>
</div>
<div>What to try next?</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
since you have already removed the volume from one host in the cluster and =
you still see it on another host you can do the following to remove the vol=
ume from another host.<br>
<br>
1) Login to the host where the volume is present.<br>
2) cd to /var/lib/glusterd/vols<br>
3) rm -rf <vol_name><br>
4) Restart glusterd on that host.<br>
<br>
And before doing the above make sure that you do not have any other volume =
present in the cluster.<br>
<br>
Above steps should not be run on a production system as you might loose the=
volume and data.
<br>
<br>
Now removing the host from UI should succed.<br>
<br>
<blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA=
bkkQ@mail.gmail.com" type=3D"cite">
<div class=3D"gmail_extra">
<div class=3D"gmail_quote">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir=3D"ltr">
<div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12=
pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr=
">
<div><br>
</div>
<div>P.S. I=92ve tried to join this user group several times in the past, w=
ith no response.</div>
<div>Is it possible for me to join this group?</div>
<div><br>
</div>
<div>Regards,</div>
<div>Andrew</div>
<br>
</div>
</div>
</blockquote>
</div>
</div>
<br>
<fieldset class=3D"mimeAttachmentHeader"></fieldset> <br>
<pre wrap=3D"">_______________________________________________
Users mailing list
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"mail=
to:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext" href=3D"http://=
lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/list=
info/users</a>
</pre>
</blockquote>
<p><br>
</p>
</div>
</blockquote>
<p><br>
</p>
</div>
</div>
</div>
</blockquote>
<p><br>
</p>
</div>
</body>
</html>
--_000_BY1PR09MB0824A056B7185A98CDF374C7F5030BY1PR09MB0824namp_--
This is a multi-part message in MIME format. --------------47436FFB8B4C5BD03AEEA978 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit On 04/12/2017 08:45 PM, Precht, Andrew wrote:
Hi all,
You asked: Any errors in ovirt-engine.log file ?
Yes, In the engine.log this error is repeated about every 3 minutes:
2017-04-12 07:16:12,554-07 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob] (DefaultQuartzScheduler3) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Error updating tasks from CLI: org.ovirt.engine.core.common.errors.EngineException: EngineException: Command execution failederror: Error : Request timed outreturn code: 1 (Failed with error GlusterVolumeStatusAllFailedException and code 4161)error: Error : Request timed out
I am not sure why this says "Request timed out".
1) gluster volume list -> Still shows the deleted volume (test1)
2) gluster peer status -> Shows one of the peers twice with different uuids:
Hostname: 192.168.10.109Uuid: 42fbb7de-8e6f-4159-a601-3f858fa65f6cState: Peer in Cluster (Connected)Hostname: 192.168.10.109Uuid: e058babe-7f9d-49fe-a3ea-ccdc98d7e5b5State: Peer in Cluster (Connected)
How did this happen? Are the hostname same for two hosts ?
I tried a gluster volume stop test1, with this result: volume stop: test1: failed: Another transaction is in progress for test1. Please try again after sometime.
can you restart glusterd and try to stop and delete the volume?
The etc-glusterfs-glusterd.vol.log shows no activity triggered by trying to remove the test1 volume from the UI.
The ovirt-engine.log shows this repeating many times, when trying to remove the test1 volume from the UI:
2017-04-12 07:57:38,049-07 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler9) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[b0e1b909-9a6a-49dc-8e20-3a027218f7e1=<GLUSTER, ACTION_TYPE_FAILED_GLUSTER_OPERATION_INPROGRESS>]', sharedLocks='null'}'
can you restart ovirt-engine service because i see that "failed to acquire lock". Once ovirt-engine is restarted some one who is holding the lock should be release and things should work fine. Last but not least, if none of the above works: Login to all your nodes in the cluster. rm -rf /var/lib/glusterd/vols/* rm -rf /var/lib/glusterd/peers/* systemctl restart glusterd on all the nodes. Login to UI and see if any volumes / hosts are present. If yes, remove them. This should clear things for you and you can start from basic.
Thanks much,
Andrew
------------------------------------------------------------------------ *From:* knarra <knarra@redhat.com> *Sent:* Tuesday, April 11, 2017 11:10:04 PM *To:* Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume On 04/12/2017 03:35 AM, Precht, Andrew wrote:
I just noticed this in the Alerts tab: Detected deletion of volume test1 on cluster 8000-1, and deleted it from engine DB.
Yet, It still shows in the web UI?
Any errors in ovirt-engine.log file ? if the volume is deleted from db ideally it should be deleted from UI too. Can you go to gluster nodes and check for the following:
1) gluster volume list -> should not return anything since you have deleted the volumes.
2) gluster peer status -> on all the nodes should show that all the peers are in connected state.
can you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster log and capture the error messages when you try deleting the volume from UI?
Log what you have pasted in the previous mail only gives info and i could not get any details from that on why volume delete is failing
------------------------------------------------------------------------ *From:* Precht, Andrew *Sent:* Tuesday, April 11, 2017 2:39:31 PM *To:* knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume The plot thickens I put all hosts in the cluster into maintenance mode, with the Stop Gluster service checkbox checked. I then deleted the /var/lib/glusterd/vols/test1 directory on all hosts. I then took the host that the test1 volume was on out of maintenance mode. Then I tried to remove the test1 volume from within the web UI. With no luck, I got the message: Could not delete Gluster Volume test1 on cluster 8000-1.
I went back and checked all host for the test1 directory, it is not on any host. Yet I still cant remove it
Any suggestions?
------------------------------------------------------------------------ *From:* Precht, Andrew *Sent:* Tuesday, April 11, 2017 1:15:22 PM *To:* knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume Here is an update
I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the node that had the trouble volume (test1). I didnt see any errors. So, I ran a tail -f on the log as I tried to remove the volume using the web UI. here is what was appended:
[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req The message "I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 6 times between [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536] The message "I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 20 times between [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179] [2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
Im seeing that the timestamps on these log entries do not match the time on the node.
The next steps I stopped the glusterd service on the node with volume test1 I deleted it with: rm -rf /var/lib/glusterd/vols/test1 I started the glusterd service.
After starting the gluster service back up, the directory /var/lib/glusterd/vols/test1 reappears. Im guessing syncing with the other nodes? Is this because I have the Volume Option: auth allow * Do I need to remove the directory /var/lib/glusterd/vols/test1 on all nodes in the cluster individually?
thanks
Hi all, The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. On the node I can not find /var/log/glusterfs/glusterd.log However, there is a /var/log/glusterfs/glustershd.log can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? if yes, can you check if there is any error present in that file ?
What happens if I follow the four steps outlined here to remove the volume from the node _BUT_, I do have another volume present in the cluster. It too is a test volume. Neither one has any data on them. So, data loss is not an issue. Running those four steps will remove the volume from your cluster . If the volumes what you have are test volumes you could just follow
------------------------------------------------------------------------ *From:* knarra <knarra@redhat.com> *Sent:* Tuesday, April 11, 2017 11:51:18 AM *To:* Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume On 04/11/2017 11:28 PM, Precht, Andrew wrote: the steps outlined to delete them (since you are not able to delete from UI) and bring back the cluster into a normal state.
------------------------------------------------------------------------ *From:* knarra <knarra@redhat.com> *Sent:* Tuesday, April 11, 2017 10:32:27 AM *To:* Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:
Adding some people
Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht@sjlibrary.org <mailto:Andrew.Precht@sjlibrary.org>> ha scritto:
Hi Ovirt users, Im a newbie to oVirt and Im having trouble deleting a test gluster volume. The nodes are 4.1.1 and the engine is 4.1.0
When I try to remove the test volume, I click Remove, the dialog box prompting to confirm the deletion pops up and after I click OK, the dialog box changes to show a little spinning wheel and then it disappears. In the end the volume is still there.
with the latest version of glusterfs & ovirt we do not see any issue with deleting a volume. Can you please check /var/log/glusterfs/glusterd.log file if there is any error present?
The test volume was distributed with two host members. One of the hosts I was able to remove from the volume by removing the host form the cluster. When I try to remove the remaining host in the volume, even with the Force Remove box ticked, I get this response: Cannot remove Host. Server having Gluster volume.
What to try next?
since you have already removed the volume from one host in the cluster and you still see it on another host you can do the following to remove the volume from another host.
1) Login to the host where the volume is present. 2) cd to /var/lib/glusterd/vols 3) rm -rf <vol_name> 4) Restart glusterd on that host.
And before doing the above make sure that you do not have any other volume present in the cluster.
Above steps should not be run on a production system as you might loose the volume and data.
Now removing the host from UI should succed.
P.S. Ive tried to join this user group several times in the past, with no response. Is it possible for me to join this group?
Regards, Andrew
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--------------47436FFB8B4C5BD03AEEA978 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">On 04/12/2017 08:45 PM, Precht, Andrew wrote:<br> </div> <blockquote cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <meta http-equiv="Content-Type" content="text/html; charset=windows-1252"> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p>Hi all,</p> <p><span id="docs-internal-guid-44c81590-62b4-32c0-d6d1-82b627328ebb"></span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">You asked: Any errors in ovirt-engine.log file ?</span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent: 36pt;"> <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">Yes, In the engine.log this error is repeated about every 3 minutes:</span></p> <br> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">2017-04-12 07:16:12,554-07 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob] (DefaultQuartzScheduler3) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Error updating tasks from CLI: org.ovirt.engine.core.common.errors.EngineException: EngineException: Command execution failed</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">error: Error : Request timed out</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">return code: 1 (Failed with error GlusterVolumeStatusAllFailedException and code 4161)</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">error: Error : Request timed out</span></p> </div> </blockquote> I am not sure why this says "Request timed out".<br> <blockquote cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent: 36pt;"> <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">1) gluster volume list -> Still shows the deleted volume (test1)</span></p> </div> </blockquote> <blockquote cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent: 36pt;"> <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">2) gluster peer status -> Shows one of the peers twice with different uuids:</span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">Hostname: 192.168.10.109</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">Uuid: 42fbb7de-8e6f-4159-a601-3f858fa65f6c</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">State: Peer in Cluster (Connected)</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">Hostname: 192.168.10.109</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">Uuid: e058babe-7f9d-49fe-a3ea-ccdc98d7e5b5</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">State: Peer in Cluster (Connected)</span></p> </div> </blockquote> How did this happen? Are the hostname same for two hosts ?<br> <blockquote cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="vertical-align: baseline;"><font color="#212121" face="Arial"><span style="font-size: 11.5pt; white-space: pre-wrap;">I tried a gluster volume stop test1, with this </span><span style="font-size: 15.3333px; white-space: pre-wrap;">result</span><span style="font-size: 11.5pt; white-space: pre-wrap;">: <span>volume stop: test1: failed: Another transaction is in progress for test1. Please try again after sometime.</span></span></font></span></p> </div> </blockquote> <font color="#212121"><font face="Arial">can you restart glusterd and try to stop and delete the volume? </font></font><br> <blockquote cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt; white-space: pre-wrap; text-indent: 36pt;"> </span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt; white-space: pre-wrap; text-indent: 36pt;"> </span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"><span></span></span><span style="color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt; white-space: pre-wrap; text-indent: 36pt;">The etc-glusterfs-glusterd.vol.log shows no activity triggered by trying to remove the test1 volume from the UI</span><span style="color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt; white-space: pre-wrap; text-indent: 36pt;">.</span></p> <br> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">The ovirt-engine.log shows this repeating many times, when <span style="color: rgb(33, 33, 33); font-family: Arial; font-size: 15.3333px; text-indent: 48px; white-space: pre-wrap;"> trying to remove the test1 volume from the UI</span>:</span></p> <br> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">2017-04-12 07:57:38,049-07 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler9) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[b0e1b909-9a6a-49dc-8e20-3a027218f7e1=<GLUSTER, ACTION_TYPE_FAILED_GLUSTER_OPERATION_INPROGRESS>]', sharedLocks='null'}'</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span></p> </div> </blockquote> can you restart ovirt-engine service because i see that "failed to acquire lock". Once ovirt-engine is restarted some one who is holding the lock should be release and things should work fine.<br> <br> Last but not least, if none of the above works:<br> <br> Login to all your nodes in the cluster.<br> rm -rf /var/lib/glusterd/vols/*<br> rm -rf /var/lib/glusterd/peers/*<br> systemctl restart glusterd on all the nodes.<br> <br> Login to UI and see if any volumes / hosts are present. If yes, remove them.<br> <br> This should clear things for you and you can start from basic.<br> <br> <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"></span><br class="kix-line-break"> <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"></span> <blockquote cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span></p> Thanks much, <p>Andrew</p> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> knarra <a class="moz-txt-link-rfc2396E" href="mailto:knarra@redhat.com"><knarra@redhat.com></a><br> <b>Sent:</b> Tuesday, April 11, 2017 11:10:04 PM<br> <b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <div class="moz-cite-prefix">On 04/12/2017 03:35 AM, Precht, Andrew wrote:<br> </div> <blockquote cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p>I just <span>noticed </span>this in the Alerts tab: <span>Detected deletion of volume test1 on cluster 8000-1, and deleted it from engine DB.</span><br> </p> </div> </blockquote> <blockquote cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p><span>Yet, It still shows in the web UI?</span></p> </div> </blockquote> Any errors in ovirt-engine.log file ? if the volume is deleted from db ideally it should be deleted from UI too. Can you go to gluster nodes and check for the following:<br> <br> 1) gluster volume list -> should not return anything since you have deleted the volumes.<br> <br> 2) gluster peer status -> on all the nodes should show that all the peers are in connected state.<br> <br> can you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster log and capture the error messages when you try deleting the volume from UI?<br> <br> Log what you have pasted in the previous mail only gives info and i could not get any details from that on why volume delete is failing<br> <br> <blockquote cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Precht, Andrew<br> <b>Sent:</b> Tuesday, April 11, 2017 2:39:31 PM<br> <b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div>The plot thickens </div> <div>I put all hosts in the cluster into maintenance mode, with the Stop Gluster service checkbox checked. I then deleted the /var/lib/glusterd/vols/test1 directory on all hosts. I then took the host that the test1 volume was on out of maintenance mode. Then I tried to remove the test1 volume from within the web UI. With no luck, I got the message: Could not delete Gluster Volume test1 on cluster 8000-1.</div> <div><br> </div> <div>I went back and checked all host for the test1 directory, it is not on any host. Yet I still cant remove it </div> <div><br> </div> <div>Any suggestions?</div> <br> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Precht, Andrew<br> <b>Sent:</b> Tuesday, April 11, 2017 1:15:22 PM<br> <b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div>Here is an update </div> <div><br> </div> <div>I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the node that had the trouble volume (test1). I didnt see any errors. So, I ran a tail -f on the log as I tried to remove the volume using the web UI. here is what was appended:</div> <div><br> </div> <div>[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</div> <div>[2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req</div> <div>The message "I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 6 times between [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536]</div> <div>The message "I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 20 times between [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179]</div> <div>[2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</div> <div><br> </div> <div>Im seeing that the timestamps on these log entries do not match the time on the node.</div> <div><br> </div> <div>The next steps</div> <div>I stopped the glusterd service on the node with volume test1</div> <div>I deleted it with: rm -rf /var/lib/glusterd/vols/test1</div> <div>I started the glusterd service.</div> <div><br> </div> <div>After starting the gluster service back up, the directory /var/lib/glusterd/vols/test1 reappears.</div> <div>Im guessing syncing with the other nodes?</div> <div>Is this because I have the Volume Option: auth allow *</div> <div>Do I need to remove the directory /var/lib/glusterd/vols/test1 on all nodes in the cluster individually?</div> <div><br> </div> <div>thanks</div> <br> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> knarra <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:knarra@redhat.com"><knarra@redhat.com></a><br> <b>Sent:</b> Tuesday, April 11, 2017 11:51:18 AM<br> <b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <div class="moz-cite-prefix">On 04/11/2017 11:28 PM, Precht, Andrew wrote:<br> </div> <blockquote cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div>Hi all,</div> <div>The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. </div> <div>On the node I can not find /var/log/glusterfs/glusterd.log However, there is a /var/log/glusterfs/glustershd.log <br> </div> </div> </blockquote> can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? if yes, can you check if there is any error present in that file ?<span style="color: rgb(84, 84, 84); font-family: arial, sans-serif; font-size: small; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); display: inline !important; float: none;"></span> <blockquote cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div><br> </div> <div>What happens if I follow the four steps outlined here to remove the volume from the node <u>BUT</u>, I do have another volume present in the cluster. It too is a test volume. Neither one has any data on them. So, data loss is not an issue.</div> </div> </blockquote> Running those four steps will remove the volume from your cluster . If the volumes what you have are test volumes you could just follow the steps outlined to delete them (since you are not able to delete from UI) and bring back the cluster into a normal state.<br> <blockquote cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <br> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> knarra <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:knarra@redhat.com"> <knarra@redhat.com></a><br> <b>Sent:</b> Tuesday, April 11, 2017 10:32:27 AM<br> <b>To:</b> Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <div class="moz-cite-prefix">On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:<br> </div> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div dir="auto">Adding some people</div> <div class="gmail_extra"><br> <div class="gmail_quote">Il 11/Apr/2017 19:06, "Precht, Andrew" <<a moz-do-not-send="true" href="mailto:Andrew.Precht@sjlibrary.org">Andrew.Precht@sjlibrary.org</a>> ha scritto:<br type="attribution"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> <div>Hi Ovirt users,</div> <div>Im a newbie to oVirt and Im having trouble deleting a test gluster volume. The nodes are 4.1.1 and the engine is 4.1.0</div> <div><br> </div> <div>When I try to remove the test volume, I click Remove, the dialog box prompting to confirm the deletion pops up and after I click OK, the dialog box changes to show a little spinning wheel and then it disappears. In the end the volume is still there.</div> </div> </div> </blockquote> </div> </div> </blockquote> with the latest version of glusterfs & ovirt we do not see any issue with deleting a volume. Can you please check /var/log/glusterfs/glusterd.log file if there is any error present? <br> <br> <br> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> The test volume was distributed with two host members. One of the hosts I was able to remove from the volume by removing the host form the cluster. When I try to remove the remaining host in the volume, even with the Force Remove box ticked, I get this response: Cannot remove Host. Server having Gluster volume. <div><br> </div> <div>What to try next?</div> </div> </div> </blockquote> </div> </div> </blockquote> since you have already removed the volume from one host in the cluster and you still see it on another host you can do the following to remove the volume from another host.<br> <br> 1) Login to the host where the volume is present.<br> 2) cd to /var/lib/glusterd/vols<br> 3) rm -rf <vol_name><br> 4) Restart glusterd on that host.<br> <br> And before doing the above make sure that you do not have any other volume present in the cluster.<br> <br> Above steps should not be run on a production system as you might loose the volume and data. <br> <br> Now removing the host from UI should succed.<br> <br> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> <div><br> </div> <div>P.S. Ive tried to join this user group several times in the past, with no response.</div> <div>Is it possible for me to join this group?</div> <div><br> </div> <div>Regards,</div> <div>Andrew</div> <br> </div> </div> </blockquote> </div> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <p><br> </p> </div> </blockquote> <p><br> </p> </div> </div> </div> </blockquote> <p><br> </p> </div> </blockquote> <p><br> </p> </body> </html> --------------47436FFB8B4C5BD03AEEA978--
--_000_BY1PR09MB0824328CE2138DBDA861057EF5030BY1PR09MB0824namp_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
Hi all,
In the end, I ran this on each host node and is what worked:
systemctl stop glusterd && rm -rf /var/lib/glusterd/vols/* && rm -rf /var/l=
ib/glusterd/peers/*
Thanks so much for your help.
P.S. I work as a sys admin for the San Jose library. Part of my job satisfa=
ction comes from knowing that the work I do here goes directly back into th=
is community. We=92r fortunate that you, your coworkers, and Red Hat do so =
much to give back. I have to imagine you too feel this sense of satisfactio=
n. Thanks again=85
P.S.S. I never did hear back from users@ovirt.org mailing list. I did fill =
out the fields on this page: https://lists.ovirt.org/mailman/listinfo/users=
. Yet, everytime I send them an email I get: Your message to Users awaits m=
oderator approval. Is there a secret handshake, I=92m not aware of?
Regards,
Andrew
________________________________
From: knarra <knarra@redhat.com>
Sent: Wednesday, April 12, 2017 10:01:33 AM
To: Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureini=
k; Nir Soffer
Cc: users
Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol=
ume
On 04/12/2017 08:45 PM, Precht, Andrew wrote:
Hi all,
You asked: Any errors in ovirt-engine.log file ?
Yes, In the engine.log this error is repeated about every 3 minutes:
2017-04-12 07:16:12,554-07 ERROR [org.ovirt.engine.core.bll.gluster.Gluster=
TasksSyncJob] (DefaultQuartzScheduler3) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7=
b8] Error updating tasks from CLI: org.ovirt.engine.core.common.errors.Engi=
neException: EngineException: Command execution failed error: Error : Reque=
st timed out return code: 1 (Failed with error GlusterVolumeStatusAllFailed=
Exception and code 4161) error: Error : Request timed out
I am not sure why this says "Request timed out".
1) gluster volume list -> Still shows the deleted volume (test1)
2) gluster peer status -> Shows one of the peers twice with different uuid=
=92s:
Hostname: 192.168.10.109 Uuid: 42fbb7de-8e6f-4159-a601-3f858fa65f6c State: =
Peer in Cluster (Connected) Hostname: 192.168.10.109 Uuid: e058babe-7f9d-49=
fe-a3ea-ccdc98d7e5b5 State: Peer in Cluster (Connected)
How did this happen? Are the hostname same for two hosts ?
I tried a gluster volume stop test1, with this result: volume stop: test1: =
failed: Another transaction is in progress for test1. Please try again afte=
r sometime.
can you restart glusterd and try to stop and delete the volume?
The etc-glusterfs-glusterd.vol.log shows no activity triggered by trying to=
remove the test1 volume from the UI.
The ovirt-engine.log shows this repeating many times, when trying to remove=
the test1 volume from the UI:
2017-04-12 07:57:38,049-07 INFO [org.ovirt.engine.core.bll.lock.InMemoryLo=
ckManager] (DefaultQuartzScheduler9) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8]=
Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks=3D'[b0e1b=
909-9a6a-49dc-8e20-3a027218f7e1=3D<GLUSTER, ACTION_TYPE_FAILED_GLUSTER_OPER=
ATION_INPROGRESS>]', sharedLocks=3D'null'}'
can you restart ovirt-engine service because i see that "failed to acquire =
lock". Once ovirt-engine is restarted some one who is holding the lock sho=
uld be release and things should work fine.
Last but not least, if none of the above works:
Login to all your nodes in the cluster.
rm -rf /var/lib/glusterd/vols/*
rm -rf /var/lib/glusterd/peers/*
systemctl restart glusterd on all the nodes.
Login to UI and see if any volumes / hosts are present. If yes, remove them=
.
This should clear things for you and you can start from basic.
Thanks much,
Andrew
________________________________
From: knarra <knarra@redhat.com><mailto:knarra@redhat.com>
Sent: Tuesday, April 11, 2017 11:10:04 PM
To: Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureini=
k; Nir Soffer
Cc: users
Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol=
ume
On 04/12/2017 03:35 AM, Precht, Andrew wrote:
I just noticed this in the Alerts tab: Detected deletion of volume test1 on=
cluster 8000-1, and deleted it from engine DB.
Yet, It still shows in the web UI?
Any errors in ovirt-engine.log file ? if the volume is deleted from db idea=
lly it should be deleted from UI too. Can you go to gluster nodes and chec=
k for the following:
1) gluster volume list -> should not return anything since you have deleted=
the volumes.
2) gluster peer status -> on all the nodes should show that all the peers a=
re in connected state.
can you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster log and =
capture the error messages when you try deleting the volume from UI?
Log what you have pasted in the previous mail only gives info and i could n=
ot get any details from that on why volume delete is failing
________________________________
From: Precht, Andrew
Sent: Tuesday, April 11, 2017 2:39:31 PM
To: knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir S=
offer
Cc: users
Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol=
ume
The plot thickens=85
I put all hosts in the cluster into maintenance mode, with the Stop Gluster=
service checkbox checked. I then deleted the /var/lib/glusterd/vols/test1 =
directory on all hosts. I then took the host that the test1 volume was on o=
ut of maintenance mode. Then I tried to remove the test1 volume from within=
the web UI. With no luck, I got the message: Could not delete Gluster Volu=
me test1 on cluster 8000-1.
I went back and checked all host for the test1 directory, it is not on any =
host. Yet I still can=92t remove it=85
Any suggestions?
________________________________
From: Precht, Andrew
Sent: Tuesday, April 11, 2017 1:15:22 PM
To: knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir S=
offer
Cc: users
Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol=
ume
Here is an update=85
I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the node=
that had the trouble volume (test1). I didn=92t see any errors. So, I ran =
a tail -f on the log as I tried to remove the volume using the web UI. here=
is what was appended:
[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:1474:__g=
lusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:1537:__g=
lusterd_handle_cli_get_volume] 0-management: Received get vol req
The message "I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_c=
li_list_friends] 0-glusterd: Received cli list req" repeated 6 times betwee=
n [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536]
The message "I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_c=
li_get_volume] 0-management: Received get vol req" repeated 20 times betwee=
n [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179]
[2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:1474:__g=
lusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
I=92m seeing that the timestamps on these log entries do not match the time=
on the node.
The next steps
I stopped the glusterd service on the node with volume test1
I deleted it with: rm -rf /var/lib/glusterd/vols/test1
I started the glusterd service.
After starting the gluster service back up, the directory /var/lib/glusterd=
/vols/test1 reappears.
I=92m guessing syncing with the other nodes?
Is this because I have the Volume Option: auth allow *
Do I need to remove the directory /var/lib/glusterd/vols/test1 on all nodes=
in the cluster individually?
thanks
________________________________
From: knarra <knarra@redhat.com><mailto:knarra@redhat.com>
Sent: Tuesday, April 11, 2017 11:51:18 AM
To: Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureini=
k; Nir Soffer
Cc: users
Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol=
ume
On 04/11/2017 11:28 PM, Precht, Andrew wrote:
Hi all,
The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7.
On the node I can not find /var/log/glusterfs/glusterd.log However, there i=
s a /var/log/glusterfs/glustershd.log
can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? =
if yes, can you check if there is any error present in that file ?
What happens if I follow the four steps outlined here to remove the volume =
from the node BUT, I do have another volume present in the cluster. It too =
is a test volume. Neither one has any data on them. So, data loss is not an=
issue.
Running those four steps will remove the volume from your cluster . If the =
volumes what you have are test volumes you could just follow the steps outl=
ined to delete them (since you are not able to delete from UI) and bring ba=
ck the cluster into a normal state.
________________________________
From: knarra <knarra@redhat.com><mailto:knarra@redhat.com>
Sent: Tuesday, April 11, 2017 10:32:27 AM
To: Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureini=
k; Nir Soffer
Cc: users
Subject: Re: [ovirt-users] I=92m having trouble deleting a test gluster vol=
ume
On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:
Adding some people
Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht@sjlibrary.org<mailto:=
Andrew.Precht@sjlibrary.org>> ha scritto:
Hi Ovirt users,
I=92m a newbie to oVirt and I=92m having trouble deleting a test gluster vo=
lume. The nodes are 4.1.1 and the engine is 4.1.0
When I try to remove the test volume, I click Remove, the dialog box prompt=
ing to confirm the deletion pops up and after I click OK, the dialog box ch=
anges to show a little spinning wheel and then it disappears. In the end th=
e volume is still there.
with the latest version of glusterfs & ovirt we do not see any issue with d=
eleting a volume. Can you please check /var/log/glusterfs/glusterd.log file=
if there is any error present?
The test volume was distributed with two host members. One of the hosts I w=
as able to remove from the volume by removing the host form the cluster. Wh=
en I try to remove the remaining host in the volume, even with the =93Force=
Remove=94 box ticked, I get this response: Cannot remove Host. Server havi=
ng Gluster volume.
What to try next?
since you have already removed the volume from one host in the cluster and =
you still see it on another host you can do the following to remove the vol=
ume from another host.
1) Login to the host where the volume is present.
2) cd to /var/lib/glusterd/vols
3) rm -rf <vol_name>
4) Restart glusterd on that host.
And before doing the above make sure that you do not have any other volume =
present in the cluster.
Above steps should not be run on a production system as you might loose the=
volume and data.
Now removing the host from UI should succed.
P.S. I=92ve tried to join this user group several times in the past, with n=
o response.
Is it possible for me to join this group?
Regards,
Andrew
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
--_000_BY1PR09MB0824328CE2138DBDA861057EF5030BY1PR09MB0824namp_
Content-Type: text/html; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1=
252">
</head>
<body bgcolor=3D"#FFFFFF" text=3D"#000000">
<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style>
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<p></p>
<div>Hi all,</div>
<div>In the end, I ran this on each host node and is what worked:</div>
<div>systemctl stop glusterd && rm -rf /var/lib/glusterd/vols/* &am=
p;& rm -rf /var/lib/glusterd/peers/*</div>
<div><br>
</div>
<div>Thanks so much for your help.</div>
<div><br>
</div>
<div>P.S. I work as a sys admin for the San Jose library. Part of my job sa=
tisfaction comes from knowing that the work I do here goes directly back in=
to this community. We=92r fortunate that you, your coworkers, and Red Hat d=
o so much to give back. I have to
imagine you too feel this sense of satisfaction. Thanks again=85</div>
<div><br>
</div>
<div>P.S.S. I never did hear back from users@ovirt.org mailing list. I did =
fill out the fields on this page: https://lists.ovirt.org/mailman/listinfo/=
users. Yet, everytime I send them an email I get: Your message to Users awa=
its moderator approval. Is there
a secret handshake, I=92m not aware of?</div>
<div><br>
</div>
<div>Regards,</div>
<div>Andrew</div>
<br>
<p></p>
</div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" st=
yle=3D"font-size:11pt" color=3D"#000000"><b>From:</b> knarra <knarra@red=
hat.com><br>
<b>Sent:</b> Wednesday, April 12, 2017 10:01:33 AM<br>
<b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon =
Mureinik; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus=
ter volume</font>
<div> </div>
</div>
<div>
<div class=3D"moz-cite-prefix">On 04/12/2017 08:45 PM, Precht, Andrew wrote=
:<br>
</div>
<blockquote cite=3D"mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style>
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<p>Hi all,</p>
<p><span id=3D"docs-internal-guid-44c81590-62b4-32c0-d6d1-82b627328ebb"></s=
pan></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;">You asked: Any errors=
in ovirt-engine.log file ?</span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;t=
ext-indent:
36pt;">
<span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33=
); vertical-align: baseline; white-space: pre-wrap;">Yes, In the engine.log=
this error is repeated about every 3 minutes:</span></p>
<br>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;">2017-04-12 07:16:12,5=
54-07 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob]
(DefaultQuartzScheduler3) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Error upd=
ating tasks from CLI: org.ovirt.engine.core.common.errors.EngineException: =
EngineException: Command execution failed</span><span style=3D"font-size: 1=
1.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline=
; white-space: pre-wrap;">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">error: Error : =
Request timed out</span><span style=3D"font-size: 11.5pt; font-family: Aria=
l; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;=
">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">return code: 1 =
(Failed with error GlusterVolumeStatusAllFailedException and code 4161)</sp=
an><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33,=
33); vertical-align: baseline; white-space: pre-wrap;">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">error: Error : =
Request timed out</span></p>
</div>
</blockquote>
I am not sure why this says "Request timed out".<br>
<blockquote cite=3D"mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;t=
ext-indent:
36pt;">
<span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33=
); vertical-align: baseline; white-space: pre-wrap;"></span><span style=3D"=
font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-ali=
gn: baseline; white-space: pre-wrap;">1)
gluster volume list -> Still shows the deleted volume (test1)</sp=
an></p>
</div>
</blockquote>
<blockquote cite=3D"mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;t=
ext-indent:
36pt;">
<span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33=
); vertical-align: baseline; white-space: pre-wrap;"></span><span style=3D"=
font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-ali=
gn: baseline; white-space: pre-wrap;">2)
gluster peer status -> Shows one of the peers twice with different uuid=
=92s:</span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;">Hostname: 192.168.10.=
109</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(=
33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">Uuid: 42fbb7de-=
8e6f-4159-a601-3f858fa65f6c</span><span style=3D"font-size: 11.5pt; font-fa=
mily: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space:=
pre-wrap;">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">State: Peer in =
Cluster (Connected)</span><span style=3D"font-size: 11.5pt; font-family: Ar=
ial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wra=
p;">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;"></span><span st=
yle=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); verti=
cal-align: baseline; white-space: pre-wrap;">Hostname:
192.168.10.109</span><span style=3D"font-size: 11.5pt; font-family: Arial;=
color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">Uuid: e058babe-=
7f9d-49fe-a3ea-ccdc98d7e5b5</span><span style=3D"font-size: 11.5pt; font-fa=
mily: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space:=
pre-wrap;">
</span><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33,=
33, 33); vertical-align: baseline; white-space: pre-wrap;">State: Peer in =
Cluster (Connected)</span></p>
</div>
</blockquote>
How did this happen? Are the hostname same for two hosts ?<br>
<blockquote cite=3D"mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;"></span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"vertical-align: baseline;"><font color=3D"#212121" face=3D"=
Arial"><span style=3D"font-size: 11.5pt; white-space: pre-wrap;">I tried a =
gluster volume stop test1, with this
</span><span style=3D"font-size: 15.3333px; white-space: pre-wrap;">result<=
/span><span style=3D"font-size: 11.5pt; white-space: pre-wrap;">:
<span>volume stop: test1: failed: Another transaction is in progress for te=
st1. Please try again after sometime.</span></span></font></span></p>
</div>
</blockquote>
<font color=3D"#212121"><font face=3D"Arial">can you restart glusterd and t=
ry to stop and delete the volume?
</font></font><br>
<blockquote cite=3D"mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5=
pt; white-space: pre-wrap; text-indent: 36pt;"></span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5=
pt; white-space: pre-wrap; text-indent: 36pt;"></span></p>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;"><span></span></span><=
span style=3D"color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt=
; white-space: pre-wrap; text-indent: 36pt;">The
etc-glusterfs-glusterd.vol.log shows no activity triggered by trying to re=
move the test1 volume from the UI</span><span style=3D"color: rgb(33, 33, 3=
3); font-family: Arial; font-size: 11.5pt; white-space: pre-wrap; text-inde=
nt: 36pt;">.</span></p>
<br>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;">The ovirt-engine.log =
shows this repeating many times, when
<span style=3D"color: rgb(33, 33, 33); font-family: Arial; font-size: 15.33=
33px; text-indent: 48px; white-space: pre-wrap;">
trying to remove the test1 volume from the UI</span>:</span></p>
<br>
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;">2017-04-12 07:57:38,0=
49-07 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler9) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Failed to=
acquire lock and wait lock 'EngineLock:{exclusiveLocks=3D'[b0e1b909-9a6a-4=
9dc-8e20-3a027218f7e1=3D<GLUSTER, ACTION_TYPE_FAILED_GLUSTER_OPERATION_I=
NPROGRESS>]', sharedLocks=3D'null'}'</span><span style=3D"font-size: 11.=
5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; =
white-space: pre-wrap;">
</span></p>
</div>
</blockquote>
can you restart ovirt-engine service because i see that "failed to acq=
uire lock". Once ovirt-engine is restarted some one who is holdi=
ng the lock should be release and things should work fine.<br>
<br>
Last but not least, if none of the above works:<br>
<br>
Login to all your nodes in the cluster.<br>
rm -rf /var/lib/glusterd/vols/*<br>
rm -rf /var/lib/glusterd/peers/*<br>
systemctl restart glusterd on all the nodes.<br>
<br>
Login to UI and see if any volumes / hosts are present. If yes, remove them=
.<br>
<br>
This should clear things for you and you can start from basic.<br>
<br>
<span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33=
); vertical-align: baseline; white-space: pre-wrap;"></span><br class=3D"ki=
x-line-break">
<span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33=
); vertical-align: baseline; white-space: pre-wrap;"></span>
<blockquote cite=3D"mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;"=
><span style=3D"font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 3=
3); vertical-align: baseline; white-space: pre-wrap;"></span></p>
Thanks much,
<p>Andrew</p>
</div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color=
=3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> knarra
<a class=3D"moz-txt-link-rfc2396E" href=3D"mailto:knarra@redhat.com"><kn=
arra@redhat.com></a><br>
<b>Sent:</b> Tuesday, April 11, 2017 11:10:04 PM<br>
<b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon =
Mureinik; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus=
ter volume</font>
<div> </div>
</div>
<div>
<div class=3D"moz-cite-prefix">On 04/12/2017 03:35 AM, Precht, Andrew wrote=
:<br>
</div>
<blockquote cite=3D"mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style>
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<p>I just <span>noticed </span>this in the Alerts tab: <span>Dete=
cted deletion of volume test1 on cluster 8000-1, and deleted it from engine=
DB.</span><br>
</p>
</div>
</blockquote>
<blockquote cite=3D"mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<p><span>Yet, It still shows in the web UI?</span></p>
</div>
</blockquote>
Any errors in ovirt-engine.log file ? if the volume is deleted from db idea=
lly it should be deleted from UI too. Can you go to gluster nodes and=
check for the following:<br>
<br>
1) gluster volume list -> should not return anything since you have dele=
ted the volumes.<br>
<br>
2) gluster peer status -> on all the nodes should show that all the peer=
s are in connected state.<br>
<br>
can you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster log and =
capture the error messages when you try deleting the volume from UI?<br>
<br>
Log what you have pasted in the previous mail only gives info and i could n=
ot get any details from that on why volume delete is failing<br>
<br>
<blockquote cite=3D"mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
</div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color=
=3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> Precht, Andrew<br>
<b>Sent:</b> Tuesday, April 11, 2017 2:39:31 PM<br>
<b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik=
; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus=
ter volume</font>
<div> </div>
</div>
<div><style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;=
margin-bottom:0;} --></style>
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<div>The plot thickens=85</div>
<div>I put all hosts in the cluster into maintenance mode, with the Stop Gl=
uster service checkbox checked. I then deleted the /var/lib/glusterd/vols/t=
est1 directory on all hosts. I then took the host that the test1 volume was=
on out of maintenance mode. Then
I tried to remove the test1 volume from within the web UI. With no luck, I=
got the message: Could not delete Gluster Volume test1 on cluster 8000-1.<=
/div>
<div><br>
</div>
<div>I went back and checked all host for the test1 directory, it is not on=
any host. Yet I still can=92t remove it=85</div>
<div><br>
</div>
<div>Any suggestions?</div>
<br>
</div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color=
=3D"#000000" face=3D"Calibri,
sans-serif"><b>From:</b> Precht, Andrew<br>
<b>Sent:</b> Tuesday, April 11, 2017 1:15:22 PM<br>
<b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik=
; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus=
ter volume</font>
<div> </div>
</div>
<div><style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;=
margin-bottom:0;} --></style>
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<div>Here is an update=85</div>
<div><br>
</div>
<div>I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the=
node that had the trouble volume (test1). I didn=92t see any errors. So, I=
ran a tail -f on the log as I tried to remove the volume using the web UI.=
here is what was appended:</div>
<div><br>
</div>
<div>[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:147=
4:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</di=
v>
<div>[2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:153=
7:__glusterd_handle_cli_get_volume] 0-management: Received get vol req</div=
>
<div>The message "I [MSGID: 106487] [glusterd-handler.c:1474:__gluster=
d_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated=
6 times between [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.5965=
36]</div>
<div>The message "I [MSGID: 106488] [glusterd-handler.c:1537:__gluster=
d_handle_cli_get_volume] 0-management: Received get vol req" repeated =
20 times between [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.0821=
79]</div>
<div>[2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:147=
4:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</di=
v>
<div><br>
</div>
<div>I=92m seeing that the timestamps on these log entries do not match the=
time on the node.</div>
<div><br>
</div>
<div>The next steps</div>
<div>I stopped the glusterd service on the node with volume test1</div>
<div>I deleted it with: rm -rf /var/lib/glusterd/vols/test1</div>
<div>I started the glusterd service.</div>
<div><br>
</div>
<div>After starting the gluster service back up, the directory /var/lib/glu=
sterd/vols/test1 reappears.</div>
<div>I=92m guessing syncing with the other nodes?</div>
<div>Is this because I have the Volume Option: auth allow *</div>
<div>Do I need to remove the directory /var/lib/glusterd/vols/test1 on all =
nodes in the cluster individually?</div>
<div><br>
</div>
<div>thanks</div>
<br>
</div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color=
=3D"#000000" face=3D"Calibri,
sans-serif"><b>From:</b> knarra
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-rfc2396E" href=3D"mailto:=
knarra@redhat.com">
<knarra@redhat.com></a><br>
<b>Sent:</b> Tuesday, April 11, 2017 11:51:18 AM<br>
<b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon =
Mureinik; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus=
ter volume</font>
<div> </div>
</div>
<div>
<div class=3D"moz-cite-prefix">On 04/11/2017 11:28 PM, Precht, Andrew wrote=
:<br>
</div>
<blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style>
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<div>Hi all,</div>
<div>The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. </div>
<div>On the node I can not find /var/log/glusterfs/glusterd.log However, th=
ere is a /var/log/glusterfs/glustershd.log
<br>
</div>
</div>
</blockquote>
can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? =
if yes, can you check if there is any error present in that file ?<span sty=
le=3D"color: rgb(84, 84,
84); font-family: arial, sans-serif; font-size: small;
font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: normal;
letter-spacing: normal; orphans: 2; text-align: left;
text-indent: 0px; text-transform: none; white-space:
normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255); display: inline !important; float:
none;"></span>
<blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<div><br>
</div>
<div>What happens if I follow the four steps outlined here to remove the vo=
lume from the node
<u>BUT</u>, I do have another volume present in the cluster. It too is a te=
st volume. Neither one has any data on them. So, data loss is not an issue.=
</div>
</div>
</blockquote>
Running those four steps will remove the volume from your cluster . If the =
volumes what you have are test volumes you could just follow the steps outl=
ined to delete them (since you are not able to delete from UI) and bring ba=
ck the cluster into a normal state.<br>
<blockquote cite=3D"mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB082=
4.namprd09.prod.outlook.com" type=3D"cite">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr">
<br>
</div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" color=
=3D"#000000" face=3D"Calibri, sans-serif"><b>From:</b> knarra
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-rfc2396E" href=3D"mailto:=
knarra@redhat.com">
<knarra@redhat.com></a><br>
<b>Sent:</b> Tuesday, April 11, 2017 10:32:27 AM<br>
<b>To:</b> Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon =
Mureinik; Nir Soffer<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] I=92m having trouble deleting a test glus=
ter volume</font>
<div> </div>
</div>
<div>
<div class=3D"moz-cite-prefix">On 04/11/2017 10:44 PM, Sandro Bonazzola wro=
te:<br>
</div>
<blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA=
bkkQ@mail.gmail.com" type=3D"cite">
<div dir=3D"auto">Adding some people</div>
<div class=3D"gmail_extra"><br>
<div class=3D"gmail_quote">Il 11/Apr/2017 19:06, "Precht, Andrew"=
<<a moz-do-not-send=3D"true" href=3D"mailto:Andrew.Precht@sjlibrary.org=
">Andrew.Precht@sjlibrary.org</a>> ha scritto:<br type=3D"attribution">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x
#ccc solid;padding-left:1ex">
<div dir=3D"ltr">
<div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12=
pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr=
">
<div>Hi Ovirt users,</div>
<div>I=92m a newbie to oVirt and I=92m having trouble deleting a test glust=
er volume. The nodes are 4.1.1 and the engine is 4.1.0</div>
<div><br>
</div>
<div>When I try to remove the test volume, I click Remove, the dialog box p=
rompting to confirm the deletion pops up and after I click OK, the dialog b=
ox changes to show a little spinning wheel and then it disappears. In the e=
nd the volume is still there.</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
with the latest version of glusterfs & ovirt we do not see any issue wi=
th deleting a volume. Can you please check /var/log/glusterfs/glusterd.log =
file if there is any error present?
<br>
<br>
<br>
<blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA=
bkkQ@mail.gmail.com" type=3D"cite">
<div class=3D"gmail_extra">
<div class=3D"gmail_quote">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x
#ccc solid;padding-left:1ex">
<div dir=3D"ltr">
<div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12=
pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr=
">
The test volume was distributed with two host members. One of the hosts I w=
as able to remove from the volume by removing the host form the cluster. Wh=
en I try to remove the remaining host in the volume, even with the =93Force=
Remove=94 box ticked, I get this response:
Cannot remove Host. Server having Gluster volume.
<div><br>
</div>
<div>What to try next?</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
since you have already removed the volume from one host in the cluster and =
you still see it on another host you can do the following to remove the vol=
ume from another host.<br>
<br>
1) Login to the host where the volume is present.<br>
2) cd to /var/lib/glusterd/vols<br>
3) rm -rf <vol_name><br>
4) Restart glusterd on that host.<br>
<br>
And before doing the above make sure that you do not have any other volume =
present in the cluster.<br>
<br>
Above steps should not be run on a production system as you might loose the=
volume and data.
<br>
<br>
Now removing the host from UI should succed.<br>
<br>
<blockquote cite=3D"mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oA=
bkkQ@mail.gmail.com" type=3D"cite">
<div class=3D"gmail_extra">
<div class=3D"gmail_quote">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x
#ccc solid;padding-left:1ex">
<div dir=3D"ltr">
<div id=3D"m_7567123270890452842divtagdefaultwrapper" style=3D"font-size:12=
pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir=3D"ltr=
">
<div><br>
</div>
<div>P.S. I=92ve tried to join this user group several times in the past, w=
ith no response.</div>
<div>Is it possible for me to join this group?</div>
<div><br>
</div>
<div>Regards,</div>
<div>Andrew</div>
<br>
</div>
</div>
</blockquote>
</div>
</div>
<br>
<fieldset class=3D"mimeAttachmentHeader"></fieldset> <br>
<pre wrap=3D"">_______________________________________________
Users mailing list
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"mail=
to:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext" href=3D"http://=
lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/list=
info/users</a>
</pre>
</blockquote>
<p><br>
</p>
</div>
</blockquote>
<p><br>
</p>
</div>
</div>
</div>
</blockquote>
<p><br>
</p>
</div>
</blockquote>
<p><br>
</p>
</div>
</body>
</html>
--_000_BY1PR09MB0824328CE2138DBDA861057EF5030BY1PR09MB0824namp_--
This is a multi-part message in MIME format. --------------7E0D932FD9865E9D45BD24B8 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit On 04/13/2017 02:35 AM, Precht, Andrew wrote:
Hi all, In the end, I ran this on each host node and is what worked: systemctl stop glusterd && rm -rf /var/lib/glusterd/vols/* && rm -rf /var/lib/glusterd/peers/* Thanks so much for your help.
P.S. I work as a sys admin for the San Jose library. Part of my job satisfaction comes from knowing that the work I do here goes directly back into this community. Wer fortunate that you, your coworkers, and Red Hat do so much to give back. I have to imagine you too feel this sense of satisfaction. Thanks again
P.S.S. I never did hear back from users@ovirt.org mailing list. I did fill out the fields on this page: https://lists.ovirt.org/mailman/listinfo/users. Yet, everytime I send them an email I get: Your message to Users awaits moderator approval. Is there a secret handshake, Im not aware of? Ideally you should be able to. Not sure why . May be a separate email
YW :-) thread to this user list should help you get the right answer ?
Regards, Andrew
------------------------------------------------------------------------ *From:* knarra <knarra@redhat.com> *Sent:* Wednesday, April 12, 2017 10:01:33 AM *To:* Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume On 04/12/2017 08:45 PM, Precht, Andrew wrote:
Hi all,
You asked: Any errors in ovirt-engine.log file ?
Yes, In the engine.log this error is repeated about every 3 minutes:
2017-04-12 07:16:12,554-07 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob] (DefaultQuartzScheduler3) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Error updating tasks from CLI: org.ovirt.engine.core.common.errors.EngineException: EngineException: Command execution failederror: Error : Request timed outreturn code: 1 (Failed with error GlusterVolumeStatusAllFailedException and code 4161)error: Error : Request timed out
I am not sure why this says "Request timed out".
1) gluster volume list -> Still shows the deleted volume (test1)
2) gluster peer status -> Shows one of the peers twice with different uuids:
Hostname: 192.168.10.109Uuid: 42fbb7de-8e6f-4159-a601-3f858fa65f6cState: Peer in Cluster (Connected)Hostname: 192.168.10.109Uuid: e058babe-7f9d-49fe-a3ea-ccdc98d7e5b5State: Peer in Cluster (Connected)
How did this happen? Are the hostname same for two hosts ?
I tried a gluster volume stop test1, with this result: volume stop: test1: failed: Another transaction is in progress for test1. Please try again after sometime.
can you restart glusterd and try to stop and delete the volume?
The etc-glusterfs-glusterd.vol.log shows no activity triggered by trying to remove the test1 volume from the UI.
The ovirt-engine.log shows this repeating many times, when trying to remove the test1 volume from the UI:
2017-04-12 07:57:38,049-07 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler9) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[b0e1b909-9a6a-49dc-8e20-3a027218f7e1=<GLUSTER, ACTION_TYPE_FAILED_GLUSTER_OPERATION_INPROGRESS>]', sharedLocks='null'}'
can you restart ovirt-engine service because i see that "failed to acquire lock". Once ovirt-engine is restarted some one who is holding the lock should be release and things should work fine.
Last but not least, if none of the above works:
Login to all your nodes in the cluster. rm -rf /var/lib/glusterd/vols/* rm -rf /var/lib/glusterd/peers/* systemctl restart glusterd on all the nodes.
Login to UI and see if any volumes / hosts are present. If yes, remove them.
This should clear things for you and you can start from basic.
Thanks much,
Andrew
------------------------------------------------------------------------ *From:* knarra <knarra@redhat.com> *Sent:* Tuesday, April 11, 2017 11:10:04 PM *To:* Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume On 04/12/2017 03:35 AM, Precht, Andrew wrote:
I just noticed this in the Alerts tab: Detected deletion of volume test1 on cluster 8000-1, and deleted it from engine DB.
Yet, It still shows in the web UI?
Any errors in ovirt-engine.log file ? if the volume is deleted from db ideally it should be deleted from UI too. Can you go to gluster nodes and check for the following:
1) gluster volume list -> should not return anything since you have deleted the volumes.
2) gluster peer status -> on all the nodes should show that all the peers are in connected state.
can you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster log and capture the error messages when you try deleting the volume from UI?
Log what you have pasted in the previous mail only gives info and i could not get any details from that on why volume delete is failing
------------------------------------------------------------------------ *From:* Precht, Andrew *Sent:* Tuesday, April 11, 2017 2:39:31 PM *To:* knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume The plot thickens I put all hosts in the cluster into maintenance mode, with the Stop Gluster service checkbox checked. I then deleted the /var/lib/glusterd/vols/test1 directory on all hosts. I then took the host that the test1 volume was on out of maintenance mode. Then I tried to remove the test1 volume from within the web UI. With no luck, I got the message: Could not delete Gluster Volume test1 on cluster 8000-1.
I went back and checked all host for the test1 directory, it is not on any host. Yet I still cant remove it
Any suggestions?
------------------------------------------------------------------------ *From:* Precht, Andrew *Sent:* Tuesday, April 11, 2017 1:15:22 PM *To:* knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume Here is an update
I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the node that had the trouble volume (test1). I didnt see any errors. So, I ran a tail -f on the log as I tried to remove the volume using the web UI. here is what was appended:
[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req The message "I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 6 times between [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536] The message "I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 20 times between [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179] [2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
Im seeing that the timestamps on these log entries do not match the time on the node.
The next steps I stopped the glusterd service on the node with volume test1 I deleted it with: rm -rf /var/lib/glusterd/vols/test1 I started the glusterd service.
After starting the gluster service back up, the directory /var/lib/glusterd/vols/test1 reappears. Im guessing syncing with the other nodes? Is this because I have the Volume Option: auth allow * Do I need to remove the directory /var/lib/glusterd/vols/test1 on all nodes in the cluster individually?
thanks
Hi all, The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. On the node I can not find /var/log/glusterfs/glusterd.log However, there is a /var/log/glusterfs/glustershd.log can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? if yes, can you check if there is any error present in that file ?
What happens if I follow the four steps outlined here to remove the volume from the node _BUT_, I do have another volume present in the cluster. It too is a test volume. Neither one has any data on them. So, data loss is not an issue. Running those four steps will remove the volume from your cluster . If the volumes what you have are test volumes you could just follow
------------------------------------------------------------------------ *From:* knarra <knarra@redhat.com> *Sent:* Tuesday, April 11, 2017 11:51:18 AM *To:* Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume On 04/11/2017 11:28 PM, Precht, Andrew wrote: the steps outlined to delete them (since you are not able to delete from UI) and bring back the cluster into a normal state.
------------------------------------------------------------------------ *From:* knarra <knarra@redhat.com> *Sent:* Tuesday, April 11, 2017 10:32:27 AM *To:* Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:
Adding some people
Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht@sjlibrary.org <mailto:Andrew.Precht@sjlibrary.org>> ha scritto:
Hi Ovirt users, Im a newbie to oVirt and Im having trouble deleting a test gluster volume. The nodes are 4.1.1 and the engine is 4.1.0
When I try to remove the test volume, I click Remove, the dialog box prompting to confirm the deletion pops up and after I click OK, the dialog box changes to show a little spinning wheel and then it disappears. In the end the volume is still there.
with the latest version of glusterfs & ovirt we do not see any issue with deleting a volume. Can you please check /var/log/glusterfs/glusterd.log file if there is any error present?
The test volume was distributed with two host members. One of the hosts I was able to remove from the volume by removing the host form the cluster. When I try to remove the remaining host in the volume, even with the Force Remove box ticked, I get this response: Cannot remove Host. Server having Gluster volume.
What to try next?
since you have already removed the volume from one host in the cluster and you still see it on another host you can do the following to remove the volume from another host.
1) Login to the host where the volume is present. 2) cd to /var/lib/glusterd/vols 3) rm -rf <vol_name> 4) Restart glusterd on that host.
And before doing the above make sure that you do not have any other volume present in the cluster.
Above steps should not be run on a production system as you might loose the volume and data.
Now removing the host from UI should succed.
P.S. Ive tried to join this user group several times in the past, with no response. Is it possible for me to join this group?
Regards, Andrew
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--------------7E0D932FD9865E9D45BD24B8 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">On 04/13/2017 02:35 AM, Precht, Andrew wrote:<br> </div> <blockquote cite="mid:BY1PR09MB0824328CE2138DBDA861057EF5030@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <meta http-equiv="Content-Type" content="text/html; charset=windows-1252"> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div>Hi all,</div> <div>In the end, I ran this on each host node and is what worked:</div> <div>systemctl stop glusterd && rm -rf /var/lib/glusterd/vols/* && rm -rf /var/lib/glusterd/peers/*</div> </div> </blockquote> <blockquote cite="mid:BY1PR09MB0824328CE2138DBDA861057EF5030@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div>Thanks so much for your help.</div> <div><br> </div> </div> </blockquote> YW :-)<br> <blockquote cite="mid:BY1PR09MB0824328CE2138DBDA861057EF5030@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div> </div> <div>P.S. I work as a sys admin for the San Jose library. Part of my job satisfaction comes from knowing that the work I do here goes directly back into this community. Wer fortunate that you, your coworkers, and Red Hat do so much to give back. I have to imagine you too feel this sense of satisfaction. Thanks again </div> <div><br> </div> <div>P.S.S. I never did hear back from <a class="moz-txt-link-abbreviated" href="mailto:users@ovirt.org">users@ovirt.org</a> mailing list. I did fill out the fields on this page: <a class="moz-txt-link-freetext" href="https://lists.ovirt.org/mailman/listinfo/users">https://lists.ovirt.org/mailman/listinfo/users</a>. Yet, everytime I send them an email I get: Your message to Users awaits moderator approval. Is there a secret handshake, Im not aware of?</div> </div> </blockquote> Ideally you should be able to. Not sure why . May be a separate email thread to this user list should help you get the right answer ?<br> <blockquote cite="mid:BY1PR09MB0824328CE2138DBDA861057EF5030@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div><br> </div> <div>Regards,</div> <div>Andrew</div> <br> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> knarra <a class="moz-txt-link-rfc2396E" href="mailto:knarra@redhat.com"><knarra@redhat.com></a><br> <b>Sent:</b> Wednesday, April 12, 2017 10:01:33 AM<br> <b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <div class="moz-cite-prefix">On 04/12/2017 08:45 PM, Precht, Andrew wrote:<br> </div> <blockquote cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p>Hi all,</p> <p><span id="docs-internal-guid-44c81590-62b4-32c0-d6d1-82b627328ebb"></span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">You asked: Any errors in ovirt-engine.log file ?</span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent: 36pt;"> <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">Yes, In the engine.log this error is repeated about every 3 minutes:</span></p> <br> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">2017-04-12 07:16:12,554-07 ERROR [org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob] (DefaultQuartzScheduler3) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Error updating tasks from CLI: org.ovirt.engine.core.common.errors.EngineException: EngineException: Command execution failed</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">error: Error : Request timed out</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">return code: 1 (Failed with error GlusterVolumeStatusAllFailedException and code 4161)</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">error: Error : Request timed out</span></p> </div> </blockquote> I am not sure why this says "Request timed out".<br> <blockquote cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent: 36pt;"> <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"></span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">1) gluster volume list -> Still shows the deleted volume (test1)</span></p> </div> </blockquote> <blockquote cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent: 36pt;"> <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"></span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">2) gluster peer status -> Shows one of the peers twice with different uuids:</span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">Hostname: 192.168.10.109</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">Uuid: 42fbb7de-8e6f-4159-a601-3f858fa65f6c</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">State: Peer in Cluster (Connected)</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"></span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">Hostname: 192.168.10.109</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">Uuid: e058babe-7f9d-49fe-a3ea-ccdc98d7e5b5</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">State: Peer in Cluster (Connected)</span></p> </div> </blockquote> How did this happen? Are the hostname same for two hosts ?<br> <blockquote cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"></span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="vertical-align: baseline;"><font color="#212121" face="Arial"><span style="font-size: 11.5pt; white-space: pre-wrap;">I tried a gluster volume stop test1, with this </span><span style="font-size: 15.3333px; white-space: pre-wrap;">result</span><span style="font-size: 11.5pt; white-space: pre-wrap;">: <span>volume stop: test1: failed: Another transaction is in progress for test1. Please try again after sometime.</span></span></font></span></p> </div> </blockquote> <font color="#212121"><font face="Arial">can you restart glusterd and try to stop and delete the volume? </font></font><br> <blockquote cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt; white-space: pre-wrap; text-indent: 36pt;"></span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt; white-space: pre-wrap; text-indent: 36pt;"></span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"><span></span></span><span style="color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt; white-space: pre-wrap; text-indent: 36pt;">The etc-glusterfs-glusterd.vol.log shows no activity triggered by trying to remove the test1 volume from the UI</span><span style="color: rgb(33, 33, 33); font-family: Arial; font-size: 11.5pt; white-space: pre-wrap; text-indent: 36pt;">.</span></p> <br> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">The ovirt-engine.log shows this repeating many times, when <span style="color: rgb(33, 33, 33); font-family: Arial; font-size: 15.3333px; text-indent: 48px; white-space: pre-wrap;"> trying to remove the test1 volume from the UI</span>:</span></p> <br> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;">2017-04-12 07:57:38,049-07 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler9) [ccc8ed0d-8b91-4397-b6b9-ab0f77c5f7b8] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[b0e1b909-9a6a-49dc-8e20-3a027218f7e1=<GLUSTER, ACTION_TYPE_FAILED_GLUSTER_OPERATION_INPROGRESS>]', sharedLocks='null'}'</span><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"> </span></p> </div> </blockquote> can you restart ovirt-engine service because i see that "failed to acquire lock". Once ovirt-engine is restarted some one who is holding the lock should be release and things should work fine.<br> <br> Last but not least, if none of the above works:<br> <br> Login to all your nodes in the cluster.<br> rm -rf /var/lib/glusterd/vols/*<br> rm -rf /var/lib/glusterd/peers/*<br> systemctl restart glusterd on all the nodes.<br> <br> Login to UI and see if any volumes / hosts are present. If yes, remove them.<br> <br> This should clear things for you and you can start from basic.<br> <br> <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"></span><br class="kix-line-break"> <span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"></span> <blockquote cite="mid:BY1PR09MB0824A056B7185A98CDF374C7F5030@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size: 11.5pt; font-family: Arial; color: rgb(33, 33, 33); vertical-align: baseline; white-space: pre-wrap;"></span></p> Thanks much, <p>Andrew</p> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> knarra <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:knarra@redhat.com"><knarra@redhat.com></a><br> <b>Sent:</b> Tuesday, April 11, 2017 11:10:04 PM<br> <b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <div class="moz-cite-prefix">On 04/12/2017 03:35 AM, Precht, Andrew wrote:<br> </div> <blockquote cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p>I just <span>noticed </span>this in the Alerts tab: <span>Detected deletion of volume test1 on cluster 8000-1, and deleted it from engine DB.</span><br> </p> </div> </blockquote> <blockquote cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <p><span>Yet, It still shows in the web UI?</span></p> </div> </blockquote> Any errors in ovirt-engine.log file ? if the volume is deleted from db ideally it should be deleted from UI too. Can you go to gluster nodes and check for the following:<br> <br> 1) gluster volume list -> should not return anything since you have deleted the volumes.<br> <br> 2) gluster peer status -> on all the nodes should show that all the peers are in connected state.<br> <br> can you tail -f /var/log/ovirt-engine/ovirt-engine.log and gluster log and capture the error messages when you try deleting the volume from UI?<br> <br> Log what you have pasted in the previous mail only gives info and i could not get any details from that on why volume delete is failing<br> <br> <blockquote cite="mid:BY1PR09MB0824E7FF50EC2611720F4D14F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Precht, Andrew<br> <b>Sent:</b> Tuesday, April 11, 2017 2:39:31 PM<br> <b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div>The plot thickens </div> <div>I put all hosts in the cluster into maintenance mode, with the Stop Gluster service checkbox checked. I then deleted the /var/lib/glusterd/vols/test1 directory on all hosts. I then took the host that the test1 volume was on out of maintenance mode. Then I tried to remove the test1 volume from within the web UI. With no luck, I got the message: Could not delete Gluster Volume test1 on cluster 8000-1.</div> <div><br> </div> <div>I went back and checked all host for the test1 directory, it is not on any host. Yet I still cant remove it </div> <div><br> </div> <div>Any suggestions?</div> <br> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Precht, Andrew<br> <b>Sent:</b> Tuesday, April 11, 2017 1:15:22 PM<br> <b>To:</b> knarra; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div>Here is an update </div> <div><br> </div> <div>I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the node that had the trouble volume (test1). I didnt see any errors. So, I ran a tail -f on the log as I tried to remove the volume using the web UI. here is what was appended:</div> <div><br> </div> <div>[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</div> <div>[2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req</div> <div>The message "I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 6 times between [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536]</div> <div>The message "I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 20 times between [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179]</div> <div>[2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</div> <div><br> </div> <div>Im seeing that the timestamps on these log entries do not match the time on the node.</div> <div><br> </div> <div>The next steps</div> <div>I stopped the glusterd service on the node with volume test1</div> <div>I deleted it with: rm -rf /var/lib/glusterd/vols/test1</div> <div>I started the glusterd service.</div> <div><br> </div> <div>After starting the gluster service back up, the directory /var/lib/glusterd/vols/test1 reappears.</div> <div>Im guessing syncing with the other nodes?</div> <div>Is this because I have the Volume Option: auth allow *</div> <div>Do I need to remove the directory /var/lib/glusterd/vols/test1 on all nodes in the cluster individually?</div> <div><br> </div> <div>thanks</div> <br> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> knarra <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:knarra@redhat.com"> <knarra@redhat.com></a><br> <b>Sent:</b> Tuesday, April 11, 2017 11:51:18 AM<br> <b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <div class="moz-cite-prefix">On 04/11/2017 11:28 PM, Precht, Andrew wrote:<br> </div> <blockquote cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div>Hi all,</div> <div>The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. </div> <div>On the node I can not find /var/log/glusterfs/glusterd.log However, there is a /var/log/glusterfs/glustershd.log <br> </div> </div> </blockquote> can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? if yes, can you check if there is any error present in that file ?<span style="color: rgb(84, 84, 84); font-family: arial, sans-serif; font-size: small; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); display: inline !important; float: none;"></span> <blockquote cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div><br> </div> <div>What happens if I follow the four steps outlined here to remove the volume from the node <u>BUT</u>, I do have another volume present in the cluster. It too is a test volume. Neither one has any data on them. So, data loss is not an issue.</div> </div> </blockquote> Running those four steps will remove the volume from your cluster . If the volumes what you have are test volumes you could just follow the steps outlined to delete them (since you are not able to delete from UI) and bring back the cluster into a normal state.<br> <blockquote cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <br> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> knarra <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:knarra@redhat.com"> <knarra@redhat.com></a><br> <b>Sent:</b> Tuesday, April 11, 2017 10:32:27 AM<br> <b>To:</b> Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <div class="moz-cite-prefix">On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:<br> </div> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div dir="auto">Adding some people</div> <div class="gmail_extra"><br> <div class="gmail_quote">Il 11/Apr/2017 19:06, "Precht, Andrew" <<a moz-do-not-send="true" href="mailto:Andrew.Precht@sjlibrary.org">Andrew.Precht@sjlibrary.org</a>> ha scritto:<br type="attribution"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> <div>Hi Ovirt users,</div> <div>Im a newbie to oVirt and Im having trouble deleting a test gluster volume. The nodes are 4.1.1 and the engine is 4.1.0</div> <div><br> </div> <div>When I try to remove the test volume, I click Remove, the dialog box prompting to confirm the deletion pops up and after I click OK, the dialog box changes to show a little spinning wheel and then it disappears. In the end the volume is still there.</div> </div> </div> </blockquote> </div> </div> </blockquote> with the latest version of glusterfs & ovirt we do not see any issue with deleting a volume. Can you please check /var/log/glusterfs/glusterd.log file if there is any error present? <br> <br> <br> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> The test volume was distributed with two host members. One of the hosts I was able to remove from the volume by removing the host form the cluster. When I try to remove the remaining host in the volume, even with the Force Remove box ticked, I get this response: Cannot remove Host. Server having Gluster volume. <div><br> </div> <div>What to try next?</div> </div> </div> </blockquote> </div> </div> </blockquote> since you have already removed the volume from one host in the cluster and you still see it on another host you can do the following to remove the volume from another host.<br> <br> 1) Login to the host where the volume is present.<br> 2) cd to /var/lib/glusterd/vols<br> 3) rm -rf <vol_name><br> 4) Restart glusterd on that host.<br> <br> And before doing the above make sure that you do not have any other volume present in the cluster.<br> <br> Above steps should not be run on a production system as you might loose the volume and data. <br> <br> Now removing the host from UI should succed.<br> <br> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> <div><br> </div> <div>P.S. Ive tried to join this user group several times in the past, with no response.</div> <div>Is it possible for me to join this group?</div> <div><br> </div> <div>Regards,</div> <div>Andrew</div> <br> </div> </div> </blockquote> </div> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <p><br> </p> </div> </blockquote> <p><br> </p> </div> </div> </div> </blockquote> <p><br> </p> </div> </blockquote> <p><br> </p> </div> </blockquote> <p><br> </p> </body> </html> --------------7E0D932FD9865E9D45BD24B8--
This is a multi-part message in MIME format. --------------338202019CEB535D6EE10912 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit On 04/12/2017 01:45 AM, Precht, Andrew wrote:
Here is an update
I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the node that had the trouble volume (test1). I didnt see any errors. So, I ran a tail -f on the log as I tried to remove the volume using the web UI. here is what was appended:
[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req The message "I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 6 times between [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536] The message "I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 20 times between [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179] [2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
Im seeing that the timestamps on these log entries do not match the time on the node. gluster logs are in UTC format. That is the reason you might be seeing a different timestamp on your node and in the gluster logs.
The next steps I stopped the glusterd service on the node with volume test1 I deleted it with: rm -rf /var/lib/glusterd/vols/test1 I started the glusterd service.
After starting the gluster service back up, the directory /var/lib/glusterd/vols/test1 reappears. Im guessing syncing with the other nodes? yes, since you deleted it only one one node. Is this because I have the Volume Option: auth allow * Do I need to remove the directory /var/lib/glusterd/vols/test1 on all nodes in the cluster individually? you need to remove the file /var/lib/glusterd/vols/test1 on all nodes and restart glusterd service on all the nodes in the cluster.
thanks
Hi all, The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. On the node I can not find /var/log/glusterfs/glusterd.log However, there is a /var/log/glusterfs/glustershd.log can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? if yes, can you check if there is any error present in that file ?
What happens if I follow the four steps outlined here to remove the volume from the node _BUT_, I do have another volume present in the cluster. It too is a test volume. Neither one has any data on them. So, data loss is not an issue. Running those four steps will remove the volume from your cluster . If
------------------------------------------------------------------------ *From:* knarra <knarra@redhat.com> *Sent:* Tuesday, April 11, 2017 11:51:18 AM *To:* Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume On 04/11/2017 11:28 PM, Precht, Andrew wrote: the volumes what you have are test volumes you could just follow the steps outlined to delete them (since you are not able to delete from UI) and bring back the cluster into a normal state.
------------------------------------------------------------------------ *From:* knarra <knarra@redhat.com> *Sent:* Tuesday, April 11, 2017 10:32:27 AM *To:* Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer *Cc:* users *Subject:* Re: [ovirt-users] Im having trouble deleting a test gluster volume On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:
Adding some people
Il 11/Apr/2017 19:06, "Precht, Andrew" <Andrew.Precht@sjlibrary.org <mailto:Andrew.Precht@sjlibrary.org>> ha scritto:
Hi Ovirt users, Im a newbie to oVirt and Im having trouble deleting a test gluster volume. The nodes are 4.1.1 and the engine is 4.1.0
When I try to remove the test volume, I click Remove, the dialog box prompting to confirm the deletion pops up and after I click OK, the dialog box changes to show a little spinning wheel and then it disappears. In the end the volume is still there.
with the latest version of glusterfs & ovirt we do not see any issue with deleting a volume. Can you please check /var/log/glusterfs/glusterd.log file if there is any error present?
The test volume was distributed with two host members. One of the hosts I was able to remove from the volume by removing the host form the cluster. When I try to remove the remaining host in the volume, even with the Force Remove box ticked, I get this response: Cannot remove Host. Server having Gluster volume.
What to try next?
since you have already removed the volume from one host in the cluster and you still see it on another host you can do the following to remove the volume from another host.
1) Login to the host where the volume is present. 2) cd to /var/lib/glusterd/vols 3) rm -rf <vol_name> 4) Restart glusterd on that host.
And before doing the above make sure that you do not have any other volume present in the cluster.
Above steps should not be run on a production system as you might loose the volume and data.
Now removing the host from UI should succed.
P.S. Ive tried to join this user group several times in the past, with no response. Is it possible for me to join this group?
Regards, Andrew
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--------------338202019CEB535D6EE10912 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">On 04/12/2017 01:45 AM, Precht, Andrew wrote:<br> </div> <blockquote cite="mid:BY1PR09MB0824E15BF3EEC19B5B776EB5F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <meta http-equiv="Content-Type" content="text/html; charset=windows-1252"> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div>Here is an update </div> <div><br> </div> <div>I checked the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log on the node that had the trouble volume (test1). I didnt see any errors. So, I ran a tail -f on the log as I tried to remove the volume using the web UI. here is what was appended:</div> <div><br> </div> <div>[2017-04-11 19:48:40.756360] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</div> <div>[2017-04-11 19:48:42.238840] I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req</div> <div>The message "I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 6 times between [2017-04-11 19:48:40.756360] and [2017-04-11 19:49:32.596536]</div> <div>The message "I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 20 times between [2017-04-11 19:48:42.238840] and [2017-04-11 19:49:34.082179]</div> <div>[2017-04-11 19:51:41.556077] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</div> <div><br> </div> <div>Im seeing that the timestamps on these log entries do not match the time on the node.</div> </div> </blockquote> gluster logs are in UTC format. That is the reason you might be seeing a different timestamp on your node and in the gluster logs.<br> <blockquote cite="mid:BY1PR09MB0824E15BF3EEC19B5B776EB5F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div><br> </div> <div>The next steps</div> <div>I stopped the glusterd service on the node with volume test1</div> <div>I deleted it with: rm -rf /var/lib/glusterd/vols/test1</div> <div>I started the glusterd service.</div> <div><br> </div> <div>After starting the gluster service back up, the directory /var/lib/glusterd/vols/test1 reappears.</div> <div>Im guessing syncing with the other nodes?</div> </div> </blockquote> yes, since you deleted it only one one node.<br> <blockquote cite="mid:BY1PR09MB0824E15BF3EEC19B5B776EB5F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div>Is this because I have the Volume Option: auth allow *</div> <div>Do I need to remove the directory /var/lib/glusterd/vols/test1 on all nodes in the cluster individually?</div> </div> </blockquote> you need to remove the file /var/lib/glusterd/vols/test1 on all nodes and restart glusterd service on all the nodes in the cluster.<br> <blockquote cite="mid:BY1PR09MB0824E15BF3EEC19B5B776EB5F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div><br> </div> <div>thanks</div> <br> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> knarra <a class="moz-txt-link-rfc2396E" href="mailto:knarra@redhat.com"><knarra@redhat.com></a><br> <b>Sent:</b> Tuesday, April 11, 2017 11:51:18 AM<br> <b>To:</b> Precht, Andrew; Sandro Bonazzola; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <div class="moz-cite-prefix">On 04/11/2017 11:28 PM, Precht, Andrew wrote:<br> </div> <blockquote cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div>Hi all,</div> <div>The node is oVirt Node 4.1.1 with glusterfs-3.8.10-1.el7. </div> <div>On the node I can not find /var/log/glusterfs/glusterd.log However, there is a /var/log/glusterfs/glustershd.log <br> </div> </div> </blockquote> can you check if /var/log/glusterfs/etc-glusterfs-glusterd.vol.log exists? if yes, can you check if there is any error present in that file ?<span style="color: rgb(84, 84, 84); font-family: arial, sans-serif; font-size: small; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); display: inline !important; float: none;"></span> <blockquote cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <div><br> </div> <div>What happens if I follow the four steps outlined here to remove the volume from the node <u>BUT</u>, I do have another volume present in the cluster. It too is a test volume. Neither one has any data on them. So, data loss is not an issue.</div> </div> </blockquote> Running those four steps will remove the volume from your cluster . If the volumes what you have are test volumes you could just follow the steps outlined to delete them (since you are not able to delete from UI) and bring back the cluster into a normal state.<br> <blockquote cite="mid:BY1PR09MB08246AC41807C7DFFE7A3308F5000@BY1PR09MB0824.namprd09.prod.outlook.com" type="cite"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr"> <br> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> knarra <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:knarra@redhat.com"><knarra@redhat.com></a><br> <b>Sent:</b> Tuesday, April 11, 2017 10:32:27 AM<br> <b>To:</b> Sandro Bonazzola; Precht, Andrew; Sahina Bose; Tal Nisan; Allon Mureinik; Nir Soffer<br> <b>Cc:</b> users<br> <b>Subject:</b> Re: [ovirt-users] Im having trouble deleting a test gluster volume</font> <div> </div> </div> <div> <div class="moz-cite-prefix">On 04/11/2017 10:44 PM, Sandro Bonazzola wrote:<br> </div> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div dir="auto">Adding some people</div> <div class="gmail_extra"><br> <div class="gmail_quote">Il 11/Apr/2017 19:06, "Precht, Andrew" <<a moz-do-not-send="true" href="mailto:Andrew.Precht@sjlibrary.org">Andrew.Precht@sjlibrary.org</a>> ha scritto:<br type="attribution"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> <div>Hi Ovirt users,</div> <div>Im a newbie to oVirt and Im having trouble deleting a test gluster volume. The nodes are 4.1.1 and the engine is 4.1.0</div> <div><br> </div> <div>When I try to remove the test volume, I click Remove, the dialog box prompting to confirm the deletion pops up and after I click OK, the dialog box changes to show a little spinning wheel and then it disappears. In the end the volume is still there.</div> </div> </div> </blockquote> </div> </div> </blockquote> with the latest version of glusterfs & ovirt we do not see any issue with deleting a volume. Can you please check /var/log/glusterfs/glusterd.log file if there is any error present? <br> <br> <br> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> The test volume was distributed with two host members. One of the hosts I was able to remove from the volume by removing the host form the cluster. When I try to remove the remaining host in the volume, even with the Force Remove box ticked, I get this response: Cannot remove Host. Server having Gluster volume. <div><br> </div> <div>What to try next?</div> </div> </div> </blockquote> </div> </div> </blockquote> since you have already removed the volume from one host in the cluster and you still see it on another host you can do the following to remove the volume from another host.<br> <br> 1) Login to the host where the volume is present.<br> 2) cd to /var/lib/glusterd/vols<br> 3) rm -rf <vol_name><br> 4) Restart glusterd on that host.<br> <br> And before doing the above make sure that you do not have any other volume present in the cluster.<br> <br> Above steps should not be run on a production system as you might loose the volume and data. <br> <br> Now removing the host from UI should succed.<br> <br> <blockquote cite="mid:CAPQRNTk8iieOFrM0GHeCjkozBSxoA3WoLaOr+EAAHgY_oAbkkQ@mail.gmail.com" type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div id="m_7567123270890452842divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr"> <div><br> </div> <div>P.S. Ive tried to join this user group several times in the past, with no response.</div> <div>Is it possible for me to join this group?</div> <div><br> </div> <div>Regards,</div> <div>Andrew</div> <br> </div> </div> </blockquote> </div> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <p><br> </p> </div> </blockquote> <p><br> </p> </div> </blockquote> <p><br> </p> </body> </html> --------------338202019CEB535D6EE10912--
participants (3)
-
knarra -
Precht, Andrew -
Sandro Bonazzola