On February 13, 2020 5:20:18 PM GMT+02:00, g.vasilopoulos(a)uoc.gr wrote: > Hello > We have a problem with hosted engine storage after updating one host > which serves as a gluster server for the engine (the setup is gluster > replica 3 with local disks from 3 hypervisors) > Volume heal command shows > [root@o5-car0118 engine]# gluster volume heal engine info > Brick o5-car0118.gfs-int.uoc.gr:/gluster_bricks/engine/engine > /a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/e3a3ef50-56b6-48b0-a9f8-2d6382e2286e/b6973d5b-cc7a-4abd-8d2d-94f551936a97.meta > > /a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/f5e576d4-eea7-431b-a0f0-f8a557006471/464518c5-f79c-4271-9d97-995981cde2cb.meta > > Status: Connected > Number of entries: 2 > > Brick o2-car0121.gfs-int.uoc.gr:/gluster_bricks/engine/engine > /a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/e3a3ef50-56b6-48b0-a9f8-2d6382e2286e/b6973d5b-cc7a-4abd-8d2d-94f551936a97.meta > > /a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/f5e576d4-eea7-431b-a0f0-f8a557006471/464518c5-f79c-4271-9d97-995981cde2cb.meta > > Status: Connected > Number of entries: 2 > > Brick o9-car0114.gfs-int.uoc.gr:/gluster_bricks/engine/engine > Status: Connected > Number of entries: 0 > > On all the gluster servers > I notice that the affected directories have date in 1970. > [root@o5-car0118 images]# ls -al > σύνολο 24 > drwxr-xr-x. 23 vdsm kvm 8192 Σεπ 24 12:07 . > drwxr-xr-x. 6 vdsm kvm 64 Σεπ 19 2018 .. > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > 2bac658f-70ce-4adb-ab68-a0f0c205c70c > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > 3034a69c-b5b5-46fa-a393-59ea46635142 > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > 5538ae6b-ccc6-4861-b71b-6b2c7af2e0ab > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > 66dbce25-8863-42b5-904a-484f8e9c225a > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > 6c049108-28f7-47d9-8d54-4ac2697dcba8 > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > 72702607-1896-420d-931a-42c9f01d37f1 > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > 7c617da4-ab6b-4791-80be-541f5be60dd8 > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > 902a16d3-6494-4840-a528-b49972f9c332 > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > 96fd6116-7983-4385-bca6-e6ca8edc94ca > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > abd875cd-96b6-47a6-b6a3-ae35300a21cc > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > add7bc92-1a40-474d-9255-53ac861b75ed > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > b7b06df7-465f-4fc7-a214-033b7dca6bc7 > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > c0ecacac-26c6-40d9-87da-af17d9de8d21 > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > c4d2d5da-2a15-4735-8919-324ae8372064 > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > c7e0c784-bb8e-4024-95df-b6f4267b0208 > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > d1f1ff5a-387d-442c-9240-1c58e4d6f8a7 > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > d3e172cb-b6dd-4867-a9cd-f4fa006648bc > drwxr-xr-x. 2 vdsm kvm 8192 Ιαν 1 1970 > e3a3ef50-56b6-48b0-a9f8-2d6382e2286e <----- > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > e477ec02-11ab-4d92-b5fd-44e91fbde7f9 > drwxr-xr-x. 2 vdsm kvm 149 Αύγ 2 2019 > e839485b-b0be-47f6-9847-b691e02ce9a4 > drwxr-xr-x. 2 vdsm kvm 8192 Ιαν 1 1970 > f5e576d4-eea7-431b-a0f0-f8a557006471 <----- > > I think this has something to do with a gluster bug. > Is there a way to correct this and heal the volume? > Thank you! > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/EHC5TPN6DU5... For the meta files - check on the bricks which is newest. Usually only the timestamp inside is different. If data is the same -> rsync the newest file from the brick to the other bricks and use 'gluster volume heal <volname> full' (for me it never worked without 'full'). We are expecting fix for the meta files with different gfid very soon. For the timestamp - write an e-mail to the gluster-users mail list, as they can help better. Best Regards, Strahil Nikolov