Volume XXX contains a apparently corrupt brick(s).
---1103591728-816431412-1409604258=:73116 Content-Type: text/plain; charset=us-ascii Dear All; I have GlusterFS storage domain in my oVirt Engine Version: 3.4.3-1.el6. One of the peers get disconnected due to network cable unplugged and after that I failed to add/remove disks from my GlusterFS domain. I have healed the GlusterFS volume and removed the corrupted index file and healed the GlusterFS volume. As workaround, I have stopped all the VM and put the GlusterFS domian in maintenance mode and activated it again and now I can add/remove disks from the GlusterFS domain. But I still have the below warnings in my ovirt-engine.log:- 2014-09-01 23:09:47,550 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-26) Volume XXX contains a apparently corrupt brick(s). Hence will not add it to engine at this point. 2014-09-01 23:09:52,558 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) START, GlusterVolumesListVDSCommand(HostName = host1, HostId = 0de0e7d7-a3dd-4776-9393-c36b8c4b452d), log id: 2b6381b0 2014-09-01 23:09:52,588 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-61) Could not add brick host2:/var/lib/exports/gluster/brick1 to volume fca9d8a4-392d-4e55-a2ac-8e09a75c5cd2 - server uuid d7763283-10b6-4880-a7b3-48d0353205b0 not found in cluster 00000001-0001-0001-0001-00000000033a 2014-09-01 23:09:52,594 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, GlusterVolumesListVDSCommand, return: {fca9d8a4-392d-4e55-a2ac-8e09a75c5cd2=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@107254ac}, log id: 2b6381b0 Best Regards Fawzy Ibrahim ---1103591728-816431412-1409604258=:73116 Content-Type: text/html; charset=us-ascii <html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:8pt"><div style="" class="">Dear All;</div><div style="" class=""><br style="" class=""></div><div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class="">I have GlusterFS storage domain in my oVirt Engine Version: 3.4.3-1.el6.</div><div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class=""><br></div><div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class="">One of the peers get disconnected due to network cable unplugged and after that I failed to add/remove disks from my GlusterFS domain.</div><div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class=""><br></div><div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class="">I have healed the GlusterFS volume and removed the corrupted index file and healed the GlusterFS volume.</div><div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class=""><br></div><div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class="">As workaround, I have stopped all the VM and put the GlusterFS domian in maintenance mode and activated it again and now I can add/remove disks from the GlusterFS domain.</div><div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class=""><br></div><div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class="">But I still have the below warnings in my ovirt-engine.log:-<br class="" style=""></div><div class="" style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;"><br style="" class=""></div><div class="" style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;">2014-09-01 23:09:47,550 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-26) Volume XXX contains a apparently corrupt brick(s). Hence will not add it to engine at this point.<br style="" class="">2014-09-01 23:09:52,558 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) START, GlusterVolumesListVDSCommand(HostName = host1, HostId = 0de0e7d7-a3dd-4776-9393-c36b8c4b452d), log id: 2b6381b0<br style="" class="">2014-09-01 23:09:52,588 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-61) Could not add brick host2:/var/lib/exports/gluster/brick1 to volume fca9d8a4-392d-4e55-a2ac-8e09a75c5cd2 - server uuid d7763283-10b6-4880-a7b3-48d0353205b0 not found in cluster 00000001-0001-0001-0001-00000000033a<br style="" class="">2014-09-01 23:09:52,594 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, GlusterVolumesListVDSCommand, return: {fca9d8a4-392d-4e55-a2ac-8e09a75c5cd2=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@107254ac}, log id: 2b6381b0</div><div class="" style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;"><br></div><div class="" style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;">Best Regards</div><div class="" style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;">Fawzy Ibrahim<br></div></div></body></html> ---1103591728-816431412-1409604258=:73116--
This is a multi-part message in MIME format. --------------030707050409040804000705 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 09/02/2014 02:14 AM, Fawzy Ibrhim wrote:
Dear All;
I have GlusterFS storage domain in my oVirt Engine Version: 3.4.3-1.el6.
One of the peers get disconnected due to network cable unplugged and after that I failed to add/remove disks from my GlusterFS domain.
I have healed the GlusterFS volume and removed the corrupted index file and healed the GlusterFS volume.
As workaround, I have stopped all the VM and put the GlusterFS domian in maintenance mode and activated it again and now I can add/remove disks from the GlusterFS domain.
But I still have the below warnings in my ovirt-engine.log:-
2014-09-01 23:09:47,550 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-26) Volume XXX contains a apparently corrupt brick(s). Hence will not add it to engine at this point. 2014-09-01 23:09:52,558 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) START, GlusterVolumesListVDSCommand(HostName = host1, HostId = 0de0e7d7-a3dd-4776-9393-c36b8c4b452d), log id: 2b6381b0 2014-09-01 23:09:52,588 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-61) Could not add brick host2:/var/lib/exports/gluster/brick1 to volume fca9d8a4-392d-4e55-a2ac-8e09a75c5cd2 - server uuid d7763283-10b6-4880-a7b3-48d0353205b0 not found in cluster 00000001-0001-0001-0001-00000000033a 2014-09-01 23:09:52,594 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, GlusterVolumesListVDSCommand, return: {fca9d8a4-392d-4e55-a2ac-8e09a75c5cd2=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@107254ac}, log id: 2b6381b0
It looks like you're managing your gluster storage using oVirt as well. Since one of your hosts got disconnected (host2 ?), did the host entry get removed from the list of hosts in your cluster? The gluster volume has a brick on one of the removed hosts, and it is unable to synchronize the brick information returned from gluster cli to the engine's database.
Best Regards Fawzy Ibrahim
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--------------030707050409040804000705 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <br> <div class="moz-cite-prefix">On 09/02/2014 02:14 AM, Fawzy Ibrhim wrote:<br> </div> <blockquote cite="mid:1409604258.73116.YahooMailNeo@web124503.mail.ne1.yahoo.com" type="cite"> <div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:8pt"> <div style="" class="">Dear All;</div> <div style="" class=""><br style="" class=""> </div> <div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class="">I have GlusterFS storage domain in my oVirt Engine Version: 3.4.3-1.el6.</div> <div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class=""><br> </div> <div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class="">One of the peers get disconnected due to network cable unplugged and after that I failed to add/remove disks from my GlusterFS domain.</div> <div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class=""><br> </div> <div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class="">I have healed the GlusterFS volume and removed the corrupted index file and healed the GlusterFS volume.</div> <div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class=""><br> </div> <div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class="">As workaround, I have stopped all the VM and put the GlusterFS domian in maintenance mode and activated it again and now I can add/remove disks from the GlusterFS domain.</div> <div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class=""><br> </div> <div style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;" class="">But I still have the below warnings in my ovirt-engine.log:-<br class="" style=""> </div> <div class="" style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;"><br style="" class=""> </div> <div class="" style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;">2014-09-01 23:09:47,550 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Worker-26) Volume XXX contains a apparently corrupt brick(s). Hence will not add it to engine at this point.<br style="" class=""> 2014-09-01 23:09:52,558 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) START, GlusterVolumesListVDSCommand(HostName = host1, HostId = 0de0e7d7-a3dd-4776-9393-c36b8c4b452d), log id: 2b6381b0<br style="" class=""> 2014-09-01 23:09:52,588 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-61) Could not add brick host2:/var/lib/exports/gluster/brick1 to volume fca9d8a4-392d-4e55-a2ac-8e09a75c5cd2 - server uuid d7763283-10b6-4880-a7b3-48d0353205b0 not found in cluster 00000001-0001-0001-0001-00000000033a<br style="" class=""> 2014-09-01 23:09:52,594 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, GlusterVolumesListVDSCommand, return: {fca9d8a4-392d-4e55-a2ac-8e09a75c5cd2=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@107254ac}, log id: 2b6381b0</div> <div class="" style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;"><br> </div> </div> </blockquote> <br> <br> It looks like you're managing your gluster storage using oVirt as well. Since one of your hosts got disconnected (host2 ?), did the host entry get removed from the list of hosts in your cluster?<br> <br> The gluster volume has a brick on one of the removed hosts, and it is unable to synchronize the brick information returned from gluster cli to the engine's database.<br> <br> <br> <br> <blockquote cite="mid:1409604258.73116.YahooMailNeo@web124503.mail.ne1.yahoo.com" type="cite"> <div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:8pt"> <div class="" style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;">Best Regards</div> <div class="" style="color: rgb(0, 0, 0); font-size: 10.6667px; font-family: times new roman,new york,times,serif; background-color: transparent; font-style: normal;">Fawzy Ibrahim<br> </div> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> </body> </html> --------------030707050409040804000705--
participants (2)
-
Fawzy Ibrhim -
Sahina Bose