hosted engine storage does not heal
Hello We have a problem with hosted engine storage after updating one host which serves as a gluster server for the engine (the setup is gluster replica 3 with local disks from 3 hypervisors) Volume heal command shows [root@o5-car0118 engine]# gluster volume heal engine info Brick o5-car0118.gfs-int.uoc.gr:/gluster_bricks/engine/engine /a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/e3a3ef50-56b6-48b0-a9f8-2d6382e2286e/b6973d5b-cc7a-4abd-8d2d-94f551936a97.meta /a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/f5e576d4-eea7-431b-a0f0-f8a557006471/464518c5-f79c-4271-9d97-995981cde2cb.meta Status: Connected Number of entries: 2 Brick o2-car0121.gfs-int.uoc.gr:/gluster_bricks/engine/engine /a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/e3a3ef50-56b6-48b0-a9f8-2d6382e2286e/b6973d5b-cc7a-4abd-8d2d-94f551936a97.meta /a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/f5e576d4-eea7-431b-a0f0-f8a557006471/464518c5-f79c-4271-9d97-995981cde2cb.meta Status: Connected Number of entries: 2 Brick o9-car0114.gfs-int.uoc.gr:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 On all the gluster servers I notice that the affected directories have date in 1970. [root@o5-car0118 images]# ls -al σύνολο 24 drwxr-xr-x. 23 vdsm kvm 8192 Σεπ 24 12:07 . drwxr-xr-x. 6 vdsm kvm 64 Σεπ 19 2018 .. drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 2bac658f-70ce-4adb-ab68-a0f0c205c70c drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 3034a69c-b5b5-46fa-a393-59ea46635142 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 5538ae6b-ccc6-4861-b71b-6b2c7af2e0ab drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 66dbce25-8863-42b5-904a-484f8e9c225a drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 6c049108-28f7-47d9-8d54-4ac2697dcba8 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 72702607-1896-420d-931a-42c9f01d37f1 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 7c617da4-ab6b-4791-80be-541f5be60dd8 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 902a16d3-6494-4840-a528-b49972f9c332 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 96fd6116-7983-4385-bca6-e6ca8edc94ca drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 abd875cd-96b6-47a6-b6a3-ae35300a21cc drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 add7bc92-1a40-474d-9255-53ac861b75ed drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 b7b06df7-465f-4fc7-a214-033b7dca6bc7 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 c0ecacac-26c6-40d9-87da-af17d9de8d21 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 c4d2d5da-2a15-4735-8919-324ae8372064 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 c7e0c784-bb8e-4024-95df-b6f4267b0208 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 d1f1ff5a-387d-442c-9240-1c58e4d6f8a7 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 d3e172cb-b6dd-4867-a9cd-f4fa006648bc drwxr-xr-x. 2 vdsm kvm 8192 Ιαν 1 1970 e3a3ef50-56b6-48b0-a9f8-2d6382e2286e <----- drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 e477ec02-11ab-4d92-b5fd-44e91fbde7f9 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 e839485b-b0be-47f6-9847-b691e02ce9a4 drwxr-xr-x. 2 vdsm kvm 8192 Ιαν 1 1970 f5e576d4-eea7-431b-a0f0-f8a557006471 <----- I think this has something to do with a gluster bug. Is there a way to correct this and heal the volume? Thank you!
On February 13, 2020 5:20:18 PM GMT+02:00, g.vasilopoulos@uoc.gr wrote:
Hello We have a problem with hosted engine storage after updating one host which serves as a gluster server for the engine (the setup is gluster replica 3 with local disks from 3 hypervisors) Volume heal command shows [root@o5-car0118 engine]# gluster volume heal engine info Brick o5-car0118.gfs-int.uoc.gr:/gluster_bricks/engine/engine /a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/e3a3ef50-56b6-48b0-a9f8-2d6382e2286e/b6973d5b-cc7a-4abd-8d2d-94f551936a97.meta
/a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/f5e576d4-eea7-431b-a0f0-f8a557006471/464518c5-f79c-4271-9d97-995981cde2cb.meta
Status: Connected Number of entries: 2
Brick o2-car0121.gfs-int.uoc.gr:/gluster_bricks/engine/engine /a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/e3a3ef50-56b6-48b0-a9f8-2d6382e2286e/b6973d5b-cc7a-4abd-8d2d-94f551936a97.meta
/a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/f5e576d4-eea7-431b-a0f0-f8a557006471/464518c5-f79c-4271-9d97-995981cde2cb.meta
Status: Connected Number of entries: 2
Brick o9-car0114.gfs-int.uoc.gr:/gluster_bricks/engine/engine Status: Connected Number of entries: 0
On all the gluster servers I notice that the affected directories have date in 1970. [root@o5-car0118 images]# ls -al σύνολο 24 drwxr-xr-x. 23 vdsm kvm 8192 Σεπ 24 12:07 . drwxr-xr-x. 6 vdsm kvm 64 Σεπ 19 2018 .. drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 2bac658f-70ce-4adb-ab68-a0f0c205c70c drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 3034a69c-b5b5-46fa-a393-59ea46635142 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 5538ae6b-ccc6-4861-b71b-6b2c7af2e0ab drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 66dbce25-8863-42b5-904a-484f8e9c225a drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 6c049108-28f7-47d9-8d54-4ac2697dcba8 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 72702607-1896-420d-931a-42c9f01d37f1 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 7c617da4-ab6b-4791-80be-541f5be60dd8 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 902a16d3-6494-4840-a528-b49972f9c332 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 96fd6116-7983-4385-bca6-e6ca8edc94ca drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 abd875cd-96b6-47a6-b6a3-ae35300a21cc drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 add7bc92-1a40-474d-9255-53ac861b75ed drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 b7b06df7-465f-4fc7-a214-033b7dca6bc7 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 c0ecacac-26c6-40d9-87da-af17d9de8d21 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 c4d2d5da-2a15-4735-8919-324ae8372064 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 c7e0c784-bb8e-4024-95df-b6f4267b0208 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 d1f1ff5a-387d-442c-9240-1c58e4d6f8a7 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 d3e172cb-b6dd-4867-a9cd-f4fa006648bc drwxr-xr-x. 2 vdsm kvm 8192 Ιαν 1 1970 e3a3ef50-56b6-48b0-a9f8-2d6382e2286e <----- drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 e477ec02-11ab-4d92-b5fd-44e91fbde7f9 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 e839485b-b0be-47f6-9847-b691e02ce9a4 drwxr-xr-x. 2 vdsm kvm 8192 Ιαν 1 1970 f5e576d4-eea7-431b-a0f0-f8a557006471 <-----
I think this has something to do with a gluster bug. Is there a way to correct this and heal the volume? Thank you! _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EHC5TPN6DU5ZV7...
For the meta files - check on the bricks which is newest. Usually only the timestamp inside is different. If data is the same -> rsync the newest file from the brick to the other bricks and use 'gluster volume heal <volname> full' (for me it never worked without 'full'). We are expecting fix for the meta files with different gfid very soon. For the timestamp - write an e-mail to the gluster-users mail list, as they can help better. Best Regards, Strahil Nikolov
it seems that this exactly was the problem Thank you so much! Στις 13/2/20 5:33 μ.μ., ο Strahil Nikolov έγραψε:
On February 13, 2020 5:20:18 PM GMT+02:00, g.vasilopoulos@uoc.gr wrote:
Hello We have a problem with hosted engine storage after updating one host which serves as a gluster server for the engine (the setup is gluster replica 3 with local disks from 3 hypervisors) Volume heal command shows [root@o5-car0118 engine]# gluster volume heal engine info Brick o5-car0118.gfs-int.uoc.gr:/gluster_bricks/engine/engine /a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/e3a3ef50-56b6-48b0-a9f8-2d6382e2286e/b6973d5b-cc7a-4abd-8d2d-94f551936a97.meta
/a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/f5e576d4-eea7-431b-a0f0-f8a557006471/464518c5-f79c-4271-9d97-995981cde2cb.meta
Status: Connected Number of entries: 2
Brick o2-car0121.gfs-int.uoc.gr:/gluster_bricks/engine/engine /a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/e3a3ef50-56b6-48b0-a9f8-2d6382e2286e/b6973d5b-cc7a-4abd-8d2d-94f551936a97.meta
/a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/f5e576d4-eea7-431b-a0f0-f8a557006471/464518c5-f79c-4271-9d97-995981cde2cb.meta
Status: Connected Number of entries: 2
Brick o9-car0114.gfs-int.uoc.gr:/gluster_bricks/engine/engine Status: Connected Number of entries: 0
On all the gluster servers I notice that the affected directories have date in 1970. [root@o5-car0118 images]# ls -al σύνολο 24 drwxr-xr-x. 23 vdsm kvm 8192 Σεπ 24 12:07 . drwxr-xr-x. 6 vdsm kvm 64 Σεπ 19 2018 .. drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 2bac658f-70ce-4adb-ab68-a0f0c205c70c drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 3034a69c-b5b5-46fa-a393-59ea46635142 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 5538ae6b-ccc6-4861-b71b-6b2c7af2e0ab drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 66dbce25-8863-42b5-904a-484f8e9c225a drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 6c049108-28f7-47d9-8d54-4ac2697dcba8 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 72702607-1896-420d-931a-42c9f01d37f1 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 7c617da4-ab6b-4791-80be-541f5be60dd8 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 902a16d3-6494-4840-a528-b49972f9c332 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 96fd6116-7983-4385-bca6-e6ca8edc94ca drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 abd875cd-96b6-47a6-b6a3-ae35300a21cc drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 add7bc92-1a40-474d-9255-53ac861b75ed drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 b7b06df7-465f-4fc7-a214-033b7dca6bc7 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 c0ecacac-26c6-40d9-87da-af17d9de8d21 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 c4d2d5da-2a15-4735-8919-324ae8372064 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 c7e0c784-bb8e-4024-95df-b6f4267b0208 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 d1f1ff5a-387d-442c-9240-1c58e4d6f8a7 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 d3e172cb-b6dd-4867-a9cd-f4fa006648bc drwxr-xr-x. 2 vdsm kvm 8192 Ιαν 1 1970 e3a3ef50-56b6-48b0-a9f8-2d6382e2286e <----- drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 e477ec02-11ab-4d92-b5fd-44e91fbde7f9 drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 e839485b-b0be-47f6-9847-b691e02ce9a4 drwxr-xr-x. 2 vdsm kvm 8192 Ιαν 1 1970 f5e576d4-eea7-431b-a0f0-f8a557006471 <-----
I think this has something to do with a gluster bug. Is there a way to correct this and heal the volume? Thank you! _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EHC5TPN6DU5ZV7... For the meta files - check on the bricks which is newest. Usually only the timestamp inside is different.
If data is the same -> rsync the newest file from the brick to the other bricks and use 'gluster volume heal <volname> full' (for me it never worked without 'full').
We are expecting fix for the meta files with different gfid very soon.
For the timestamp - write an e-mail to the gluster-users mail list, as they can help better.
Best Regards, Strahil Nikolov
-- Βασιλόπουλος Γιώργος Ηλεκτρολόγος Μηχανικός Τ.Ε. Διαχειριστής Υπολ. Συστημάτων Πανεπιστήμιο Κρήτης Κ.Υ.Υ.Τ.Π.Ε. Τμήμα Επικοινωνιών και Δικτύων Βούτες Ηρακλείου 70013 Τηλ : 2810393310 email : g.vasilopoulos@uoc.gr http://www.ucnet.uoc.gr
participants (3)
-
g.vasilopoulos＠uoc.gr -
George Vasilopoulos
-
Strahil Nikolov