Re: [ovirt-users] What network test validates a host?
On Wed, Jun 1, 2016 at 2:27 PM, Nicolas Ecarnot <nicolas(a)ecarnot.net> wrote:
Hello,
Last week, one of our DC went through a network crash, and surprisingly,
most of our hosts did resist.
Some of them lost there connectivity, and were stonithed.
I'd like to be sure to understand what tests are made to declare a host
valid :
- On the storage part, I guess EVERY[1] host is doing a read+write test
(using "dd") towards the storage domain(s), every... say 5 seconds (?)
We do:
- every 10 seconds (irs:sd_health_check_delay)
- read first block from the metadata volume
- check if vg is partial (block storage)
- perform statvfs call (file storage)
- validate master domain mount
- every 5 minutes (irs:repo_stats_cache_refresh_timeout):
- run vgck (block storage)
We do not check writes to the storage, I guess we should add this, or monitor
sanlock status, which does write to all storage domains every 20 seconds.
In case of failure, I guess a countdown is triggered until this host
is
shot.
In case of failure, domain status is reported as invalid with an error code.
On the engine side, we start a 5 minutes timer (configurable). If the domain did
not recover from the invalid state before the timer expire, we consider the
domain as failing.
If the domain is failing only on one host, this host will become
non-operational.
If the domain is failing on all hosts it will be deactivated. I think
we also try
to recover the domain, but I don't know the details.
But the network failure we faced was not on the dedicated storage network,
but purely on the "LAN" network (5 virtual networks).
- What kind of test is done on each host to declare the connectivity is OK
on every virtual network?
I ask that because oVirt has no knowledge of any gateway it could ping, and
in some cases, some virtual networks don't even have a gateway.
Is it a ping towards the SPM?
Engine checks the spm host status regularly, and if it fails it will
try to stop it and
start the spm on another host.
Towards the engine?
Is it a ping?
I ask that because I found out that some host restarted nicely, ran some
VMs, which had their NICs OK, but inside those guests, we find evidences
that they were not able to communicate with very simple networks usually
provided but the host.
So I'm trying to figure out if a host could come back to life, but partially
sound.
[1] Thus, I don't clearly see the benefit of the SPM concept...
The spm is the only host that can do metadata operations on shared storage.
Without it, your data will be corrupted, so there is a benefit.
However there are
many issues with the spm, and we are working on removing it and master
domain, and replacing it with more fault tolerant, efficient and
easier to maintain
solution.
Nir