[ovirt-users] Dealing with time drifts due to migrated/paused VMs

[Christian Affolter]

Hi everyone,

recently, I observed some time drifts within a couple of virtual CentOS
7 VMs running on oVirt 3.6.

All the VMs have NTP and the qemu guest agent running and use the
kvm-clock as their clock source.

The drifts vary over the VMs, a few of them were drifting even so high,
that NTP would panic and terminate itself.

After some log analyzing, I found out, that some of the VMs experienced
some I/O issues lately and were paused automatically (and resumed later
on), others were migrated a couple of time.

The root cause seems to be the pause/resume cycle which the guest
doesn't notice and hence must rely solely on NTP to correct the time.
Which, of course is the purpose of NTP but can take relatively long,
even if it detects a spike and performs a clock_step.

vdsm provides the _syncGuestTime() method [1], which should help in this
case and set the time using the qemu guest agent. However, it seems that
this method only gets called during a restore [2] and not after a
migration [3].
Some peeking at the master branch, reveals that the idea of setting the
time after resuming a VM won't be implemented anymore [4].

Although, the first argument within the above referenced comment might
be valid for some systems, most of it probably still prefer to get the
time corrected as quickly as possible - but of course this depends on
the use case and should ideally be configurable by the user.

I couldn't find a reference or comment about the situation during the
continuation of a VM [5]. As some simple vdsClient pause/continue cycles
have shown, the clock won't be set on continuation and it took NTP
around 20 minutes before it performed a hard clock_step.

This finally leads me to my questions:

Is there a simple way to let ovirt/vdsm/libvirt set the time after a
migration or the continuation of paused VM?

How are others dealing with this situation?


Many thanks,
Chris


[1]
https://github.com/oVirt/vdsm/blob/ovirt-3.6.3/vdsm/virt/vm.py#L1199-L1208

[2] https://github.com/oVirt/vdsm/blob/ovirt-3.6.3/vdsm/virt/vm.py#L2838

[3]
https://github.com/oVirt/vdsm/blob/ovirt-3.6.3/vdsm/virt/vm.py#L2862-L2866

[4] https://github.com/oVirt/vdsm/blob/master/vdsm/virt/vm.py#L3106-L3116

[5] https://github.com/oVirt/vdsm/blob/master/vdsm/virt/vm.py#L1011