[ovirt-users] Re: HCI Fresh Deploy - Cannot upgrade 4.4. to 4.5 or add more hosts

Their may be aspect of this where HCI engine.. composes bricks of gluster into engine hosted storage... and then is not able to add hosts into cluster. Below is looping error : [root@ovirte01 ~]# tail -f /var/log/ovirt-engine/engine.log 2021-03-08 10:40:19,192-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [] START, GlusterServersListVDSCommand(HostName = medusa.penguinpages.local, VdsIdVDSCommandParametersBase:{hostId='ea26e8ad-e762-4852-bf71-b0a6b2b69853'}), log id: 76bd303b 2021-03-08 10:40:19,395-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [] FINISH, GlusterServersListVDSCommand, return: [172.16.101.103/24:CONNECTED, thorst.penguinpages.local:CONNECTED, odinst.penguinpages.local:CONNECTED], log id: 76bd303b 2021-03-08 10:40:19,400-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [] START, GlusterVolumesListVDSCommand(HostName = medusa.penguinpages.local, GlusterVolumesListVDSParameters:{hostId='ea26e8ad-e762-4852-bf71-b0a6b2b69853'}), log id: 2ebcd203 2021-03-08 10:40:19,507-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [] Could not add brick 'thorst.penguinpages.local:/gluster_bricks/data/data' to volume '78117441-467f-4c9c-ae10-4608de688404' - server uuid '165ecdcd-10c1-4b34-aefa-9a0a6d3d8751' not found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:19,508-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [] Could not add brick 'odinst.penguinpages.local:/gluster_bricks/data/data' to volume '78117441-467f-4c9c-ae10-4608de688404' - server uuid '835293d4-38b8-4e1b-80ff-9e654effa3c3' not found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:19,511-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [] Could not associate brick 'medusast.penguinpages.local:/gluster_bricks/data/data' of volume '78117441-467f-4c9c-ae10-4608de688404' with correct network as no gluster network found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:19,511-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [] Could not add brick 'thorst.penguinpages.local:/gluster_bricks/engine/engine' to volume '40f9f0b8-1861-45d6-b63b-d59ec264422d' - server uuid '165ecdcd-10c1-4b34-aefa-9a0a6d3d8751' not found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:19,512-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [] Could not add brick 'odinst.penguinpages.local:/gluster_bricks/engine/engine' to volume '40f9f0b8-1861-45d6-b63b-d59ec264422d' - server uuid '835293d4-38b8-4e1b-80ff-9e654effa3c3' not found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:19,514-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [] Could not associate brick 'medusast.penguinpages.local:/gluster_bricks/engine/engine' of volume '40f9f0b8-1861-45d6-b63b-d59ec264422d' with correct network as no gluster network found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:19,515-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [] Could not add brick 'thorst.penguinpages.local:/gluster_bricks/vmstore/vmstore' to volume '59cb23d6-325b-48ea-a03c-6fb43809f714' - server uuid '165ecdcd-10c1-4b34-aefa-9a0a6d3d8751' not found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:19,515-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [] Could not add brick 'odinst.penguinpages.local:/gluster_bricks/vmstore/vmstore' to volume '59cb23d6-325b-48ea-a03c-6fb43809f714' - server uuid '835293d4-38b8-4e1b-80ff-9e654effa3c3' not found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:19,517-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [] Could not associate brick 'medusast.penguinpages.local:/gluster_bricks/vmstore/vmstore' of volume '59cb23d6-325b-48ea-a03c-6fb43809f714' with correct network as no gluster network found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:19,518-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [] FINISH, GlusterVolumesListVDSCommand, return: {78117441-467f-4c9c-ae10-4608de688404=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b562e3bc, 59cb23d6-325b-48ea-a03c-6fb43809f714=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@8eb20b93, 40f9f0b8-1861-45d6-b63b-d59ec264422d=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d1f4bf10}, log id: 2ebcd203 2021-03-08 10:40:34,543-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [] START, GlusterServersListVDSCommand(HostName = medusa.penguinpages.local, VdsIdVDSCommandParametersBase:{hostId='ea26e8ad-e762-4852-bf71-b0a6b2b69853'}), log id: 2b2e9659 2021-03-08 10:40:34,747-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [] FINISH, GlusterServersListVDSCommand, return: [172.16.101.103/24:CONNECTED, thorst.penguinpages.local:CONNECTED, odinst.penguinpages.local:CONNECTED], log id: 2b2e9659 2021-03-08 10:40:34,750-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [] START, GlusterVolumesListVDSCommand(HostName = medusa.penguinpages.local, GlusterVolumesListVDSParameters:{hostId='ea26e8ad-e762-4852-bf71-b0a6b2b69853'}), log id: 2c39f795 2021-03-08 10:40:34,858-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [] Could not add brick 'thorst.penguinpages.local:/gluster_bricks/data/data' to volume '78117441-467f-4c9c-ae10-4608de688404' - server uuid '165ecdcd-10c1-4b34-aefa-9a0a6d3d8751' not found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:34,858-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [] Could not add brick 'odinst.penguinpages.local:/gluster_bricks/data/data' to volume '78117441-467f-4c9c-ae10-4608de688404' - server uuid '835293d4-38b8-4e1b-80ff-9e654effa3c3' not found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:34,861-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [] Could not associate brick 'medusast.penguinpages.local:/gluster_bricks/data/data' of volume '78117441-467f-4c9c-ae10-4608de688404' with correct network as no gluster network found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:34,862-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [] Could not add brick 'thorst.penguinpages.local:/gluster_bricks/engine/engine' to volume '40f9f0b8-1861-45d6-b63b-d59ec264422d' - server uuid '165ecdcd-10c1-4b34-aefa-9a0a6d3d8751' not found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:34,863-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [] Could not add brick 'odinst.penguinpages.local:/gluster_bricks/engine/engine' to volume '40f9f0b8-1861-45d6-b63b-d59ec264422d' - server uuid '835293d4-38b8-4e1b-80ff-9e654effa3c3' not found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:34,866-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [] Could not associate brick 'medusast.penguinpages.local:/gluster_bricks/engine/engine' of volume '40f9f0b8-1861-45d6-b63b-d59ec264422d' with correct network as no gluster network found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:34,867-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [] Could not add brick 'thorst.penguinpages.local:/gluster_bricks/vmstore/vmstore' to volume '59cb23d6-325b-48ea-a03c-6fb43809f714' - server uuid '165ecdcd-10c1-4b34-aefa-9a0a6d3d8751' not found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:34,867-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [] Could not add brick 'odinst.penguinpages.local:/gluster_bricks/vmstore/vmstore' to volume '59cb23d6-325b-48ea-a03c-6fb43809f714' - server uuid '835293d4-38b8-4e1b-80ff-9e654effa3c3' not found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:34,871-05 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [] Could not associate brick 'medusast.penguinpages.local:/gluster_bricks/vmstore/vmstore' of volume '59cb23d6-325b-48ea-a03c-6fb43809f714' with correct network as no gluster network found in cluster '91fde852-8013-11eb-8054-00163e7e2056' 2021-03-08 10:40:34,871-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-87) [] FINISH, GlusterVolumesListVDSCommand, return: {78117441-467f-4c9c-ae10-4608de688404=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b562e3bc, 59cb23d6-325b-48ea-a03c-6fb43809f714=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@8eb20b93, 40f9f0b8-1861-45d6-b63b-d59ec264422d=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d1f4bf10}, log id: 2c39f795