6:27 a.m.
Hello,
Last week, one of our DC went through a network crash, and surprisingly,
most of our hosts did resist.
Some of them lost there connectivity, and were stonithed.
I'd like to be sure to understand what tests are made to declare a host
valid :
- On the storage part, I guess EVERY[1] host is doing a read+write test
(using "dd") towards the storage domain(s), every... say 5 seconds (?)
In case of failure, I guess a countdown is triggered until this host is
shot.
But the network failure we faced was not on the dedicated storage
network, but purely on the "LAN" network (5 virtual networks).
- What kind of test is done on each host to declare the connectivity is
OK on every virtual network?
I ask that because oVirt has no knowledge of any gateway it could ping,
and in some cases, some virtual networks don't even have a gateway.
Is it a ping towards the SPM?
Towards the engine?
Is it a ping?
I ask that because I found out that some host restarted nicely, ran some
VMs, which had their NICs OK, but inside those guests, we find evidences
that they were not able to communicate with very simple networks usually
provided but the host.
So I'm trying to figure out if a host could come back to life, but
partially sound.
[1] Thus, I don't clearly see the benefit of the SPM concept...
--
Nicolas ECARNOT
4:50 a.m.
On Wed, Jun 1, 2016 at 2:27 PM, Nicolas Ecarnot <nicolas(a)ecarnot.net> wrote:
Hello,
Last week, one of our DC went through a network crash, and surprisingly,
most of our hosts did resist.
Some of them lost there connectivity, and were stonithed.
I'd like to be sure to understand what tests are made to declare a host
valid :
- On the storage part, I guess EVERY[1] host is doing a read+write test
(using "dd") towards the storage domain(s), every... say 5 seconds (?)
In case of failure, I guess a countdown is triggered until this host is
shot.
But the network failure we faced was not on the dedicated storage network,
but purely on the "LAN" network (5 virtual networks).
- What kind of test is done on each host to declare the connectivity is OK
on every virtual network?
I ask that because oVirt has no knowledge of any gateway it could ping,
and in some cases, some virtual networks don't even have a gateway.
Is it a ping towards the SPM?
Towards the engine?
Is it a ping?
I ask that because I found out that some host restarted nicely, ran some
VMs, which had their NICs OK, but inside those guests, we find evidences
that they were not able to communicate with very simple networks usually
provided but the host.
So I'm trying to figure out if a host could come back to life, but
partially sound.
[1] Thus, I don't clearly see the benefit of the SPM concept...
--
Nicolas ECARNOT
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
Hello Nicolas,
In general, oVirt Engine checks frequently the host state by asking it to
send a stats report.
As part of that report, nic state is reported.
Engine will move the host to non-operational in case a 'required' network
nic link is down, or if it cannot reach the host through the management
network.
One can also use a VDSM hook to check against a reference IP for
connectivity and fake the nic state.
In case storage domain connectivity fails (attempts to read fails), it will
report back to engine through the stats report and Engine will move the
host to non-operational after a few minutes.
Thanks,
Edy.
5:49 a.m.
On Wed, Jun 1, 2016 at 2:27 PM, Nicolas Ecarnot <nicolas(a)ecarnot.net> wrote:
Hello,
Last week, one of our DC went through a network crash, and surprisingly,
most of our hosts did resist.
Some of them lost there connectivity, and were stonithed.
I'd like to be sure to understand what tests are made to declare a host
valid :
- On the storage part, I guess EVERY[1] host is doing a read+write test
(using "dd") towards the storage domain(s), every... say 5 seconds (?)
We do:
- every 10 seconds (irs:sd_health_check_delay)
- read first block from the metadata volume
- check if vg is partial (block storage)
- perform statvfs call (file storage)
- validate master domain mount
- every 5 minutes (irs:repo_stats_cache_refresh_timeout):
- run vgck (block storage)
We do not check writes to the storage, I guess we should add this, or monitor
sanlock status, which does write to all storage domains every 20 seconds.
In case of failure, I guess a countdown is triggered until this host
is
shot.
In case of failure, domain status is reported as invalid with an error code.
On the engine side, we start a 5 minutes timer (configurable). If the domain did
not recover from the invalid state before the timer expire, we consider the
domain as failing.
If the domain is failing only on one host, this host will become
non-operational.
If the domain is failing on all hosts it will be deactivated. I think
we also try
to recover the domain, but I don't know the details.
But the network failure we faced was not on the dedicated storage network,
but purely on the "LAN" network (5 virtual networks).
- What kind of test is done on each host to declare the connectivity is OK
on every virtual network?
I ask that because oVirt has no knowledge of any gateway it could ping, and
in some cases, some virtual networks don't even have a gateway.
Is it a ping towards the SPM?
Engine checks the spm host status regularly, and if it fails it will
try to stop it and
start the spm on another host.
Towards the engine?
Is it a ping?
I ask that because I found out that some host restarted nicely, ran some
VMs, which had their NICs OK, but inside those guests, we find evidences
that they were not able to communicate with very simple networks usually
provided but the host.
So I'm trying to figure out if a host could come back to life, but partially
sound.
[1] Thus, I don't clearly see the benefit of the SPM concept...
The spm is the only host that can do metadata operations on shared storage.
Without it, your data will be corrupted, so there is a benefit.
However there are
many issues with the spm, and we are working on removing it and master
domain, and replacing it with more fault tolerant, efficient and
easier to maintain
solution.
Nir
2878
days inactive
2879
days old
3 comments
3 participants
participants (3)
-
Edward Haas -
Nicolas Ecarnot -
Nir Soffer