Hello Colleagues,

I want to share my experience and especially how I have recovered after a situation where gluster refuses to heal several files of my oVirt Lab.

Most probably the situation was caused by the fact that I didn't check if all files were healed before I started the upgrade on the next node , which in a 'replica 2 arbiter1' setup has caused multiple files missing/conflicting and heal fails to happen.

Background:

1. I have powered off my HostedEngine VM and made a snapshot of the gluster volume

2. Started the update , but without screen - I messed it up and decided to revert from the snapshot

3. Powered off the volume , restored from snapshot and then started (and again snapshoted the volume)

4. Upgrade of the HostedEngine was successfull

5. Upgraded the arbiter (ovirt3)

6. I forgot to check the heal status on the arbiter and upgraded ovirt1/gluster1 (which maybe was the reason for the issue)

7. After gluster1 was healed I saw that some files are left for healing , but expected it will finish till ovirt2/gluster2 is patched

8. Sadly my assumption was not right and after the reboot of ovirt2/gluster2 I noticed that some files never heal.

Symptoms:

2 files per volume never heal (used 'full' mode) even after I 'stat'-ed every file/dir in the volume.

Ovirt Dashboard reported multiple errors (50+) that it cannot update the OVF metadata for volume/VM.

Here are my notes while I was recovering from the situation. As this is my LAB, I shutdown all VMs (including the HostedEngine) as downtime was not an issue: