[Users] Time keeping in Windows VM
Hi all, We're currently experiencing an issue in our production oVirt 3.2.2 environment (on CentOS 6.4) with time keeping on our Windows guests. It seems to have appeared around the time of the recent DST change. It's taken us a while to get to the bottom of this as some machines were created in the wrong timezone, which muddied the waters a bit. It appears that when Windows automatically updated time in the VMs, this new offset from UTC was not stored correctly, and the next time the host was shutdown and started back up, the clock was set incorrectly. We eventually managed to get this to be consistently reproducible: Set the clock an hour ahead; power off VM; power it back on; set the clock an hour back (ie return it to the original time), power off and power back on; observe that the clock has now shifted to an hour before the original time. This can be observed in the vm_dynamic table on the database. To be honest, I don't think that this an oVirt problem, as I've done some limited testing on another host using virt-manager/libvirt. If I edit the xml to set the clock offset to variable, using UTC as the basis and setting the adjustment to 3600 (mimicking how it would have been before the DST change), when I change the time in the VM back by an hour (as Windows would do automatically at DST change), the xml shows a new offset of -3600, so it seems when the clock is changed the offset it's putting in the XML is the offset based on the time from when the VM was started, not the offset from UTC. Does anyone have any suggestions? At the moment, the only things I can think of doing are either a) shutting down each VM and setting their offset to 0 in the vm_dynamic table before starting the back up again or b) setting the time forward and back an appropriate amount of time so that the offset becomes 0, shutting the VM down and powering it back on again. Thanks, Martin -- Martin Goldstone IT Systems Administrator - Finance & IT Keele University, Keele, Staffordshire, United Kingdom, ST5 5BG Telephone: +44 1782 734457 G+: http://google.com/+MartinGoldstoneKeele
On Tue, Nov 19, 2013 at 04:36:24PM +0000, Martin Goldstone wrote:
Hi all,
We're currently experiencing an issue in our production oVirt 3.2.2 environment (on CentOS 6.4) with time keeping on our Windows guests. It seems to have appeared around the time of the recent DST change. It's taken us a while to get to the bottom of this as some machines were created in the wrong timezone, which muddied the waters a bit.
It appears that when Windows automatically updated time in the VMs, this new offset from UTC was not stored correctly, and the next time the host was shutdown and started back up, the clock was set incorrectly. We eventually managed to get this to be consistently reproducible: Set the clock an hour ahead; power off VM; power it back on; set the clock an hour back (ie return it to the original time), power off and power back on; observe that the clock has now shifted to an hour before the original time. This can be observed in the vm_dynamic table on the database.
To be honest, I don't think that this an oVirt problem, as I've done some limited testing on another host using virt-manager/libvirt. If I edit the xml to set the clock offset to variable, using UTC as the basis and setting the adjustment to 3600 (mimicking how it would have been before the DST change), when I change the time in the VM back by an hour (as Windows would do automatically at DST change), the xml shows a new offset of -3600, so it seems when the clock is changed the offset it's putting in the XML is the offset based on the time from when the VM was started, not the offset from UTC.
I believe that you are seeing "Bug 956741 - When RHEL VMs are powered off/on time is off by as much as 3 hrs when system comes back up". http://libvirt.org/html/libvirt-libvirt.html#virConnectDomainEventRTCChangeC... says that utcoffset is relative to UTC. If it's not so, it's a bug (either in the doc or in the code) so I'm copying libvir-list. Which libvirt and qemu versions do you have?
Does anyone have any suggestions? At the moment, the only things I can think of doing are either a) shutting down each VM and setting their offset to 0 in the vm_dynamic table before starting the back up again or b) setting the time forward and back an appropriate amount of time so that the offset becomes 0, shutting the VM down and powering it back on again.
On Nov 19, 2013, at 23:20 , Dan Kenigsberg <danken@redhat.com> wrote:
On Tue, Nov 19, 2013 at 04:36:24PM +0000, Martin Goldstone wrote:
Hi all,
We're currently experiencing an issue in our production oVirt 3.2.2 environment (on CentOS 6.4) with time keeping on our Windows guests. It seems to have appeared around the time of the recent DST change. It's taken us a while to get to the bottom of this as some machines were created in the wrong timezone, which muddied the waters a bit.
It appears that when Windows automatically updated time in the VMs, this new offset from UTC was not stored correctly, and the next time the host was shutdown and started back up, the clock was set incorrectly. We eventually managed to get this to be consistently reproducible: Set the clock an hour ahead; power off VM; power it back on; set the clock an hour back (ie return it to the original time), power off and power back on; observe that the clock has now shifted to an hour before the original time. This can be observed in the vm_dynamic table on the database.
To be honest, I don't think that this an oVirt problem, as I've done some limited testing on another host using virt-manager/libvirt. If I edit the xml to set the clock offset to variable, using UTC as the basis and setting the adjustment to 3600 (mimicking how it would have been before the DST change), when I change the time in the VM back by an hour (as Windows would do automatically at DST change), the xml shows a new offset of -3600, so it seems when the clock is changed the offset it's putting in the XML is the offset based on the time from when the VM was started, not the offset from UTC.
I believe that you are seeing "Bug 956741 - When RHEL VMs are powered off/on time is off by as much as 3 hrs when system comes back up".
http://libvirt.org/html/libvirt-libvirt.html#virConnectDomainEventRTCChangeC... says that utcoffset is relative to UTC. If it's not so, it's a bug (either in the doc or in the code) so I'm copying libvir-list.
it is this [1] bug. AFAIK the intention is to fix doc and not the behavior because it's with us for really long time... [1] https://bugzilla.redhat.com/show_bug.cgi?id=964177
Which libvirt and qemu versions do you have?
Does anyone have any suggestions? At the moment, the only things I can think of doing are either a) shutting down each VM and setting their offset to 0 in the vm_dynamic table before starting the back up again or b) setting the time forward and back an appropriate amount of time so that the offset becomes 0, shutting the VM down and powering it back on again.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
We've got a couple of versions of libvirt and qemu installed across our boxes: qemu-kvm-0.12.1.2-2.355.0.1.el6.centos.6.x86_64 libvirt-0.10.2-18.el6_4.9.x86_64 qemu-kvm-0.12.1.2-2.355.0.1.el6_4.9.x86_64 libvirt-0.10.2-18.el6_4.14.x86_64 qemu-kvm-0.12.1.2-2.355.0.1.el6.centos.7.x86_64 libvirt-0.10.2-18.el6_4.9.x86_64 I guess it probably is that bug. I'm guessing that once the doc is updated someone will have to fix libvirt so that it uses the offset it gets back from qemu to work out what the offset from utc it should store is. On 20 November 2013 15:40, Michal Skrivanek <michal.skrivanek@redhat.com>wrote:
On Nov 19, 2013, at 23:20 , Dan Kenigsberg <danken@redhat.com> wrote:
Hi all,
We're currently experiencing an issue in our production oVirt 3.2.2 environment (on CentOS 6.4) with time keeping on our Windows guests. It seems to have appeared around the time of the recent DST change. It's taken us a while to get to the bottom of this as some machines were created in the wrong timezone, which muddied the waters a bit.
It appears that when Windows automatically updated time in the VMs, this new offset from UTC was not stored correctly, and the next time the host was shutdown and started back up, the clock was set incorrectly. We eventually managed to get this to be consistently reproducible: Set the clock an hour ahead; power off VM; power it back on; set the clock an hour back (ie return it to the original time), power off and power back on; observe that the clock has now shifted to an hour before the original time. This can be observed in the vm_dynamic table on the database.
To be honest, I don't think that this an oVirt problem, as I've done some limited testing on another host using virt-manager/libvirt. If I edit
xml to set the clock offset to variable, using UTC as the basis and setting the adjustment to 3600 (mimicking how it would have been before the DST change), when I change the time in the VM back by an hour (as Windows would do automatically at DST change), the xml shows a new offset of -3600, so it seems when the clock is changed the offset it's putting in the XML is
On Tue, Nov 19, 2013 at 04:36:24PM +0000, Martin Goldstone wrote: the the
offset based on the time from when the VM was started, not the offset from UTC.
I believe that you are seeing "Bug 956741 - When RHEL VMs are powered off/on time is off by as much as 3 hrs when system comes back up".
http://libvirt.org/html/libvirt-libvirt.html#virConnectDomainEventRTCChangeC...
says that utcoffset is relative to UTC. If it's not so, it's a bug (either in the doc or in the code) so I'm copying libvir-list.
it is this [1] bug. AFAIK the intention is to fix doc and not the behavior because it's with us for really long time...
[1] https://bugzilla.redhat.com/show_bug.cgi?id=964177
Which libvirt and qemu versions do you have?
Does anyone have any suggestions? At the moment, the only things I can think of doing are either a) shutting down each VM and setting their
to 0 in the vm_dynamic table before starting the back up again or b) setting the time forward and back an appropriate amount of time so that
offset the
offset becomes 0, shutting the VM down and powering it back on again.
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Martin Goldstone IT Systems Administrator - Finance & IT Keele University, Keele, Staffordshire, United Kingdom, ST5 5BG Telephone: +44 1782 734457 G+: http://google.com/+MartinGoldstoneKeele
About this bug, Will a windows machine will "set" the "hardware" clock or will it set the "showed"\"Calculated" time? Eliezer On 20/11/13 18:46, Martin Goldstone wrote:
We've got a couple of versions of libvirt and qemu installed across our boxes:
qemu-kvm-0.12.1.2-2.355.0.1.el6.centos.6.x86_64 libvirt-0.10.2-18.el6_4.9.x86_64
qemu-kvm-0.12.1.2-2.355.0.1.el6_4.9.x86_64 libvirt-0.10.2-18.el6_4.14.x86_64
qemu-kvm-0.12.1.2-2.355.0.1.el6.centos.7.x86_64 libvirt-0.10.2-18.el6_4.9.x86_64
I guess it probably is that bug. I'm guessing that once the doc is updated someone will have to fix libvirt so that it uses the offset it gets back from qemu to work out what the offset from utc it should store is.
participants (4)
-
Dan Kenigsberg -
Eliezer Croitoru -
Martin Goldstone -
Michal Skrivanek