[Engine-devel] vdsm checkConnectivity pitfals
Hi all, VDSM network provisioning api exposes a validity check to know the newly applied changes works. The actual check is comparing the /var/run/vdsm/client.log modified time to the time when the check began and repeats that check after sleeping 1 seconds for X time (where X is connectivityTimeOut) def clientSeen(timeout): start = time.time() while timeout >= 0: if os.stat(constants.P_VDSM_CLIENT_LOG).st_mtime > start: return True time.sleep(1) timeout -= 1 return False Main issues I spot are: 1. In case the host is in maintenance, the caller of the API must generate traffic, concurrently to running api call, and then must join and sync threads to realize when all is done. see http://gerrit.ovirt.org/#change,584 2. locally calling vdsClient also modifies the client.log - we can't rely on that no one will do that during the call. 3. Failure writing to the client log will fail network provisioning! All of the above makes it not very reliable as a check and harder to call, without posing races, as client. Possible alternate solution: 1. We can try to reach the api caller socket in return, maybe use http code 100 ? 2. pass in the API a URL which the VDSM will call. could be a health-check servlet or something similar. Any toughs?
----- Original Message -----
Hi all,
VDSM network provisioning api exposes a validity check to know the newly applied changes works. The actual check is comparing the /var/run/vdsm/client.log modified time to the time when the check began and repeats that check after sleeping 1 seconds for X time (where X is connectivityTimeOut)
def clientSeen(timeout): start = time.time() while timeout >= 0: if os.stat(constants.P_VDSM_CLIENT_LOG).st_mtime > start: return True time.sleep(1) timeout -= 1 return False
Main issues I spot are: 1. In case the host is in maintenance, the caller of the API must generate traffic, concurrently to running api call, and then must join and sync threads to realize when all is done. see http://gerrit.ovirt.org/#change,584 2. locally calling vdsClient also modifies the client.log - we can't rely on that no one will do that during the call. 3. Failure writing to the client log will fail network provisioning!
All of the above makes it not very reliable as a check and harder to call, without posing races, as client.
Possible alternate solution: 1. We can try to reach the api caller socket in return, maybe use http code 100 ? 2. pass in the API a URL which the VDSM will call. could be a health-check servlet or something similar.
Any toughs?
introduce bi-directional communications over qmf.
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
On 12/22/2011 01:05 PM, Roy Golan wrote:
Hi all,
VDSM network provisioning api exposes a validity check to know the newly applied changes works. The actual check is comparing the /var/run/vdsm/client.log modified time to the time when the check began and repeats that check after sleeping 1 seconds for X time (where X is connectivityTimeOut)
def clientSeen(timeout): start = time.time() while timeout >= 0: if os.stat(constants.P_VDSM_CLIENT_LOG).st_mtime > start: return True time.sleep(1) timeout -= 1 return False
Main issues I spot are: 1. In case the host is in maintenance, the caller of the API must generate traffic, concurrently to running api call, and then must join and sync threads to realize when all is done. see http://gerrit.ovirt.org/#change,584 2. locally calling vdsClient also modifies the client.log - we can't rely on that no one will do that during the call. 3. Failure writing to the client log will fail network provisioning!
All of the above makes it not very reliable as a check and harder to call, without posing races, as client.
Possible alternate solution: 1. We can try to reach the api caller socket in return, maybe use http code 100 ? 2. pass in the API a URL which the VDSM will call. could be a health-check servlet or something similar.
VDSM calling voirt engine is not a good enough health check. it doesn't mean engine can call back. so you would need VDSM to call an engine URL asking engine to generate traffic to VDSM. replying to Ayal - I think even with QMF you would need to ask to get a reply - so mechanism is probably same wrt API, just "URL" may change.
On Thu, Dec 22, 2011 at 01:05:22PM +0200, Roy Golan wrote:
Hi all,
VDSM network provisioning api exposes a validity check to know the newly applied changes works. The actual check is comparing the /var/run/vdsm/client.log modified time to the time when the check began and repeats that check after sleeping 1 seconds for X time (where X is connectivityTimeOut)
def clientSeen(timeout): start = time.time() while timeout >= 0: if os.stat(constants.P_VDSM_CLIENT_LOG).st_mtime > start: return True time.sleep(1) timeout -= 1 return False
Main issues I spot are: 1. In case the host is in maintenance, the caller of the API must generate traffic, concurrently to running api call, and then must join and sync threads to realize when all is done. see http://gerrit.ovirt.org/#change,584 2. locally calling vdsClient also modifies the client.log - we can't rely on that no one will do that during the call.
TODO: when verifying connectivity, vdsm should ack only if the caller of setupNetworks is logged; communication from other IPs should be ignored for that sense. Until then, customers should remember that vdsClient is unsupported, and can cause much worse stuff than fooling Vdsm to believe it has ovirt-engine connectivity.
3. Failure writing to the client log will fail network provisioning!
If vdsm cannot touch a zero-length file, it has more serious problems. 4. we verify connectivity only of the management network. It could be much nicer if we had an independent verification for other networks. avahi protocol could give this + auto discovery.
All of the above makes it not very reliable as a check and harder to call, without posing races, as client.
Possible alternate solution: 1. We can try to reach the api caller socket in return, maybe use http code 100 ? 2. pass in the API a URL which the VDSM will call. could be a health-check servlet or something similar.
Any toughs?
On Sun 25 Dec 2011 02:41:08 PM IST, Dan Kenigsberg wrote:
On Thu, Dec 22, 2011 at 01:05:22PM +0200, Roy Golan wrote:
Hi all,
VDSM network provisioning api exposes a validity check to know the newly applied changes works. The actual check is comparing the /var/run/vdsm/client.log modified time to the time when the check began and repeats that check after sleeping 1 seconds for X time (where X is connectivityTimeOut)
def clientSeen(timeout): start = time.time() while timeout>= 0: if os.stat(constants.P_VDSM_CLIENT_LOG).st_mtime> start: return True time.sleep(1) timeout -= 1 return False
Main issues I spot are: 1. In case the host is in maintenance, the caller of the API must generate traffic, concurrently to running api call, and then must join and sync threads to realize when all is done. see http://gerrit.ovirt.org/#change,584 2. locally calling vdsClient also modifies the client.log - we can't rely on that no one will do that during the call.
TODO: when verifying connectivity, vdsm should ack only if the caller of setupNetworks is logged; communication from other IPs should be ignored for that sense.
Until then, customers should remember that vdsClient is unsupported, and can cause much worse stuff than fooling Vdsm to believe it has ovirt-engine connectivity.
3. Failure writing to the client log will fail network provisioning!
If vdsm cannot touch a zero-length file, it has more serious problems.
4. we verify connectivity only of the management network. It could be much nicer if we had an independent verification for other networks. avahi protocol could give this + auto discovery.
its actually a different check per network. For mgmt network we need to know the engine can reach the vdsm on storage network, though, the VDSM is the client of the storage server. Display network couldn't determine who's its clients - can avahi solve that?
All of the above makes it not very reliable as a check and harder to call, without posing races, as client.
Possible alternate solution: 1. We can try to reach the api caller socket in return, maybe use http code 100 ? 2. pass in the API a URL which the VDSM will call. could be a health-check servlet or something similar.
Any toughs?
participants (4)
-
Ayal Baron -
Dan Kenigsberg -
Itamar Heim -
Roy Golan