Hello, One engine and two hosts all with updated f19 (despite on their names) and ovirt updates-testing repo enabled. So I have 3.3.0.1-1 and vdsm-4.12.1-4 kernel is 3.11.2-201.fc19.x86_64 (problems booting with latest 3.11.4-201.fc19.x86_64) Storage domain configured with gluster as in f19 (3.4.1-1.fc19.x86_64 recompiled binding to port 50152+) and distributed replicated bricks I do this kind of operations: - power off all VMs (to start clean) - put both hosts in maintenance - shutdown both hosts - startup one host - activate one host in webadmin gui after about 2-3 minutes delay it comes up and so it has its own gluster copy active - power on VM and write 3Gb on it [g.cecchi@c6s ~]$ sudo time dd if=/dev/zero bs=1024k count=3096 of=/testfile 3096+0 records in 3096+0 records out 3246391296 bytes (3.2 GB) copied, 42.3414 s, 76.7 MB/s 0.01user 7.99system 0:42.34elapsed 18%CPU (0avgtext+0avgdata 7360maxresident)k 0inputs+6352984outputs (0major+493minor)pagefaults 0swaps Originally the gluster fs had 13Gb used, so now has 16Gb (see /rhev/data-center/mnt/glusterSD/f18ovn01.mydomain:gvdata below): [root@f18ovn01 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/fedora-root 15G 5.6G 8.1G 41% / devtmpfs 24G 0 24G 0% /dev tmpfs 24G 8.0K 24G 1% /dev/shm tmpfs 24G 656K 24G 1% /run tmpfs 24G 0 24G 0% /sys/fs/cgroup tmpfs 24G 0 24G 0% /tmp /dev/mapper/3600508b1001037414d4b3039383300021 477M 103M 345M 23% /boot /dev/mapper/fedora-ISO_GLUSTER 10G 33M 10G 1% /gluster/ISO_GLUSTER /dev/mapper/fedora-DATA_GLUSTER 30G 16G 15G 52% /gluster/DATA_GLUSTER f18ovn01.mydomain:gvdata 30G 16G 15G 52% /rhev/data-center/mnt/glusterSD/f18ovn01.mydomain:gvdata - power on second host [root@f18ovn03 glusterfs]# gluster volume status Status of volume: gviso Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick f18ovn01.mydomain:/gluster/ISO_GLUSTER/bric k1 50153 Y 1314 Brick f18ovn03.mydomain:/gluster/ISO_GLUSTER/bric k1 50153 Y 1275 NFS Server on localhost 2049 Y 1288 Self-heal Daemon on localhost N/A Y 1295 NFS Server on 192.168.3.1 2049 Y 1328 Self-heal Daemon on 192.168.3.1 N/A Y 1335 There are no active volume tasks Status of volume: gvdata Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick f18ovn01.mydomain:/gluster/DATA_GLUSTER/bri ck1 50152 Y 1313 Brick f18ovn03.mydomaint:/gluster/DATA_GLUSTER/bri ck1 50152 Y 1280 NFS Server on localhost 2049 Y 1288 Self-heal Daemon on localhost N/A Y 1295 NFS Server on 192.168.3.1 2049 Y 1328 Self-heal Daemon on 192.168.3.1 N/A Y 1335 There are no active volume tasks But actually I don't see any network sync and fs remains in fact at 13GB... [root@f18ovn03 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/fedora-root 15G 4.7G 8.9G 35% / devtmpfs 16G 0 16G 0% /dev tmpfs 16G 0 16G 0% /dev/shm tmpfs 16G 560K 16G 1% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup tmpfs 16G 0 16G 0% /tmp /dev/mapper/3600508b1001037424d4b3035343800031 477M 103M 345M 23% /boot /dev/mapper/fedora-DATA_GLUSTER 30G 13G 18G 44% /gluster/DATA_GLUSTER /dev/mapper/fedora-ISO_GLUSTER 10G 33M 10G 1% /gluster/ISO_GLUSTER I wait some minutes but nothing changes... - I activate from webadmin gui this second host and it comes up in Up state But actually it is not synced from a storage point of view so in my opinion it should come up... Now I see on it: [root@f18ovn03 bricks]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/fedora-root 15G 4.7G 8.9G 35% / devtmpfs 16G 0 16G 0% /dev tmpfs 16G 0 16G 0% /dev/shm tmpfs 16G 588K 16G 1% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup tmpfs 16G 0 16G 0% /tmp /dev/mapper/3600508b1001037424d4b3035343800031 477M 103M 345M 23% /boot /dev/mapper/fedora-DATA_GLUSTER 30G 13G 18G 44% /gluster/DATA_GLUSTER /dev/mapper/fedora-ISO_GLUSTER 10G 33M 10G 1% /gluster/ISO_GLUSTER f18ovn01.mydomain:gvdata 30G 13G 18G 44% /rhev/data-center/mnt/glusterSD/f18ovn01.mydomain:gvdata So also the /rhev/data-center/mnt/glusterSD/f18ovn01.mydomain:gvdata view is incorrect (13Gb instead of 16Gb) I think that oVirt should have some sort of detection of this and avoid activation, or put a warning because in case of f18ovn01 mainteneance it could not becaome SPM. For example oVirt could check heal info. Normally I see no heal for ok volumes eg on a gluster volume (gviso) on this same cluster, planned to be used for iso and with no data I get: [root@f18ovn03 bricks]# gluster volume heal gviso info Gathering Heal info on volume gviso has been successful Brick f18ovn01.mydomain:/gluster/ISO_GLUSTER/brick1 Number of entries: 0 Brick f18ovn03.mydomain:/gluster/ISO_GLUSTER/brick1 Number of entries: 0 Instead on this gvdata one I do get now: [root@f18ovn03 bricks]# gluster volume heal gvdata info Gathering Heal info on volume gvdata has been successful Brick f18ovn01.mydomain:/gluster/DATA_GLUSTER/brick1 Number of entries: 6 /d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/dom_md/ids /d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/15f9ca1c-c435-4892-9eb7-0c84583b2a7d/a123801a-0a4d-4a47-a426-99d8480d2e49 /d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/a5e4f67b-50b5-4740-9990-39deb8812445/53408cb0-bcd4-40de-bc69-89d59b7b5bc2 /d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/dom_md/leases /d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/dom_md /d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/dom_md/metadata Brick f18ovn03.mydomain:/gluster/DATA_GLUSTER/brick1 Number of entries: 5 /d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/dom_md/ids <gfid:59a4d113-5881-4147-95c4-b5dee9872ad3> /d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/dom_md <gfid:db9985f8-117e-4340-98b9-a62bd963f6bf> /d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/dom_md/metadata What do you think about this? What is recommended gluster command to fix f18ovn03? Let me know which kind of ovrt/gluster logs could help. Thanks in advance, Gianluca