What do I mean by 'Hosts in Cluster B crashed': I have (3) events (events during which I twinked with the Data Center A network) during which I accumulated symptoms: Event 1: A handful of the several dozen VMs in Cluster B Paused due to Storage Issues. Restarted the VMs to restore service. Event 2: Same Event 3: One of the (3) Hosts in Cluster B rebooted (that's what I mean by 'crash'). gluster was unhappy also (two of the three Hosts also function as Gluster Bricks) ... but that could be a byproduct. At the start of Event 3, I put hosted-engine into global maintenance mode: hosted-engine --set-maintenance --mode=global Why? Because I was imagining that hosted-engine might perform some sort of connectivity checks with its local IP gateway ... and if it couldn't reach it, then emit some sort of 'shutdown' commands to *all* the KVM hosts it knows about (yes, I'm waiving my hands a lot right here ... ergo my interest in reading about what kind of checks ovirt-engine performs and what kind of remedial action it might take based on the results of those checks). You are suggesting that Cluster B depends, storage-wise, on Cluster A (or, more precisely, on Storage located at Cluster A's site). That's where my thoughts turned immediately ... but thus far, I don't see it in the pcaps I've gathering -- lots of ovirt-engine traffic, but nothing else. More poking needed. ovirt 3.5 glusterfs 3.7.6 I want to do more homework, to demonstrate that Cluster B has no storage dependency on Data Center A. But back to my original question: where might I go to better understand what kind of checks ovirt-engine performs on KVM hosts and what kind of remedial action it might take, based on the results of those checks? --sk

On Thu, Aug 2, 2018 at 1:47 PM, <stuartk@alleninstitute.org> wrote:

OK, I've spent time capturing traffic from the Hosts in Cluster B back to Data Center A. I don't believe most of the traffic matters: syslog, snmp, icmp, influxd (grafana), ssh, cfengine

After filtering out all that, I'm left with TCP 54321 -- netstat tells me that the Python interpreter owns this port -- I'm guessing that this daemon is talking with ovirt-engine down in Data Center A.

No sign of gluster-oriented traffic, e.g. TCP/UDP ports 24007/24008 nor 49152+ (that's what I expected to see ... some sort of storage dependency between the two)

So I'm back to wondering under what happens when the conversation between ovirt-engine and KVM instances is disrupted? Does it sound plausible that bad things happen? Or would you say that this seems unlikely ... that management functions may be disrupted, but operational functions would be unaffected?

--sk

https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/htm... port 54321 used by both manager and hosts to inter-communicate each one with the others: VDSM communications with the Manager and other virtualization hosts. I think that in case engine in site A is not able to connect to vdsmd of a host in site B (did this happen? you only talk about disruption inside site A but it is not clear the kind of disruption...), I think it should mark it as not responsive and eventually fence it so that it can release VM resources (if VMs running on it) and storage (if SPM) it is carrying on and start on other hosts. But if all cluster B hosts becomes unresponsive from the engine point of view I don't know the default action what would be: perhaps freeze all until something comes back? Did you configure fencing in your clusters? If so, when you stop communication inside site A, could it affect your fencing configuration towards hosts in site B? Gianluca

For future readers, here is supporting material: An admin's view of the feature: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualizat... A developer’s view of the feature: https://www.ovirt.org/develop/developer-guide/engine/automatic-fencing/ A discussion of how this works in practice (similar failure scenario): http://users.ovirt.narkive.com/XWYhDO6R/ovirt-users-strange-fencing-behaviou...