Dealing with time drifts due to migrated/paused VMs
by Christian Affolter
Hi everyone,
recently, I observed some time drifts within a couple of virtual CentOS
7 VMs running on oVirt 3.6.
All the VMs have NTP and the qemu guest agent running and use the
kvm-clock as their clock source.
The drifts vary over the VMs, a few of them were drifting even so high,
that NTP would panic and terminate itself.
After some log analyzing, I found out, that some of the VMs experienced
some I/O issues lately and were paused automatically (and resumed later
on), others were migrated a couple of time.
The root cause seems to be the pause/resume cycle which the guest
doesn't notice and hence must rely solely on NTP to correct the time.
Which, of course is the purpose of NTP but can take relatively long,
even if it detects a spike and performs a clock_step.
vdsm provides the _syncGuestTime() method [1], which should help in this
case and set the time using the qemu guest agent. However, it seems that
this method only gets called during a restore [2] and not after a
migration [3].
Some peeking at the master branch, reveals that the idea of setting the
time after resuming a VM won't be implemented anymore [4].
Although, the first argument within the above referenced comment might
be valid for some systems, most of it probably still prefer to get the
time corrected as quickly as possible - but of course this depends on
the use case and should ideally be configurable by the user.
I couldn't find a reference or comment about the situation during the
continuation of a VM [5]. As some simple vdsClient pause/continue cycles
have shown, the clock won't be set on continuation and it took NTP
around 20 minutes before it performed a hard clock_step.
This finally leads me to my questions:
Is there a simple way to let ovirt/vdsm/libvirt set the time after a
migration or the continuation of paused VM?
How are others dealing with this situation?
Many thanks,
Chris
[1]
https://github.com/oVirt/vdsm/blob/ovirt-3.6.3/vdsm/virt/vm.py#L1199-L1208
[2] https://github.com/oVirt/vdsm/blob/ovirt-3.6.3/vdsm/virt/vm.py#L2838
[3]
https://github.com/oVirt/vdsm/blob/ovirt-3.6.3/vdsm/virt/vm.py#L2862-L2866
[4] https://github.com/oVirt/vdsm/blob/master/vdsm/virt/vm.py#L3106-L3116
[5] https://github.com/oVirt/vdsm/blob/master/vdsm/virt/vm.py#L1011
7 years, 11 months
new qemu-kvm-ev available
by Gianluca Cecchi
This morning I noticed the message about updates availalble for my CentOS
7.3 host.
A yum update on it proposed qemu-kvm-ev 2.6.0-28.el7_3.3.1 and related
packages
The changelog from the current one (2.6.0-27.1.el7) seems quit big (see
below)...
I applied it and I'm going to test.
After working on the first host and activatinh inside the cluster, do you
think I can live migrate a VM from a 2.6.0-27.1.el7 host to
a 2.6.0-28.el7_3.3.1 in your opinion?
Gianluca
* Fri Jan 20 2017 Sandro Bonazzola <sbonazzo(a)redhat.com> -
ev-2.6.0-28.el7_3.3.1
- Removing RH branding from package name
* Wed Jan 04 2017 Miroslav Rezanina <mrezanin(a)redhat.com> -
rhev-2.6.0-28.el7_3.3
- kvm-pc_piix-fix-compat-props-typo-for-RHEL6-machine-type.patch
[bz#1408122]
- kvm-net-don-t-poke-at-chardev-internal-QemuOpts.patch [bz#1410200]
- Resolves: bz#1408122
(Opteron_G4 CPU model broken in QEMU 2.6 with RHEL 6 machine type)
- Resolves: bz#1410200
(qemu gets SIGSEGV when hot-plug a vhostuser network)
* Fri Dec 09 2016 Miroslav Rezanina <mrezanin(a)redhat.com> -
rhev-2.6.0-28.el7_3.2
- kvm-numa-do-not-leak-NumaOptions.patch [bz#1397745]
- kvm-char-free-the-tcp-connection-data-when-closing.patch [bz#1397745]
- kvm-char-free-MuxDriver-when-closing.patch [bz#1397745]
- kvm-ahci-free-irqs-array.patch [bz#1397745]
- kvm-virtio-input-free-config-list.patch [bz#1397745]
- kvm-usb-free-USBDevice.strings.patch [bz#1397745]
- kvm-usb-free-leaking-path.patch [bz#1397745]
- kvm-ahci-fix-sglist-leak-on-retry.patch [bz#1397745]
- kvm-virtio-add-virtqueue_rewind.patch [bz#1402509]
- kvm-virtio-balloon-fix-stats-vq-migration.patch [bz#1402509]
- kvm-virtio-blk-Release-s-rq-queue-at-system_reset.patch [bz#1393041]
- kvm-virtio-blk-Remove-stale-comment-about-draining.patch [bz#1393041]
- Resolves: bz#1393041
(system_reset should clear pending request for error (virtio-blk))
- Resolves: bz#1397745
(Backport memory leak fixes from QEMU 2.7)
- Resolves: bz#1402509
(virtio-balloon stats virtqueue does not migrate properly)
* Wed Nov 30 2016 Miroslav Rezanina <mrezanin(a)redhat.com> -
rhev-2.6.0-28.el7_3.1
- kvm-ide-fix-halted-IO-segfault-at-reset.patch [bz#1393043]
- kvm-atapi-fix-halted-DMA-reset.patch [bz#1393043]
- kvm-ahci-clear-aiocb-in-ncq_cb.patch [bz#1393736]
- kvm-Workaround-rhel6-ctrl_guest_offloads-machine-type-mi.patch
[bz#1392876]
- kvm-Postcopy-vs-xbzrle-Don-t-send-xbzrle-pages-once-in-p.patch
[bz#1395360]
- kvm-ui-fix-refresh-of-VNC-server-surface.patch [bz#1392881]
- Resolves: bz#1392876
(windows guests migration from rhel6.8-z to rhel7.3 with virtio-net-pci
fail)
- Resolves: bz#1392881
(Graphic can't be showed out quickly if guest graphic mode is vnc)
- Resolves: bz#1393043
(system_reset should clear pending request for error (IDE))
- Resolves: bz#1393736
(qemu core dump when there is an I/O error on AHCI)
- Resolves: bz#1395360
(Post-copy migration fails with XBZRLE compression)
* Tue Sep 27 2016 Miroslav Rezanina <mrezanin(a)redhat.com> -
rhev-2.6.0-28.el7
- kvm-ARM-ACPI-fix-the-AML-ID-format-for-CPU-devices.patch [bz#1373733]
- Resolves: bz#1373733
(failed to run a guest VM with >= 12 vcpu under ACPI mode)
7 years, 11 months
Ovirt FQDN
by rightkicktech.gmail.com
------PXLRQ3P0G6SY9KXSTS097IKHZKWG31
Content-Transfer-Encoding: 8bit
Content-Type: text/plain;
charset=UTF-8
Hi all,
Is there any way to bypass the FQDN access requirement on ovirt 4? On previous versions I was able to access ovirt engine using IP. It is impractical to access with FQDN when doing remote port forwarding.
Thanx,
Alex
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
------PXLRQ3P0G6SY9KXSTS097IKHZKWG31
Content-Type: text/html;
charset=utf-8
Content-Transfer-Encoding: 8bit
Hi all,<br>
<br>
Is there any way to bypass the FQDN access requirement on ovirt 4? On previous versions I was able to access ovirt engine using IP. It is impractical to access with FQDN when doing remote port forwarding.<br>
<br>
Thanx,<br>
Alex<br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.
------PXLRQ3P0G6SY9KXSTS097IKHZKWG31--
7 years, 11 months
Ovirt dashboard values
by rightkicktech.gmail.com
------ENLJO7GAMXWYZM97AQNB5AQA5DYU0F
Content-Transfer-Encoding: 8bit
Content-Type: text/plain;
charset=UTF-8
Hi all,
I'm trying to understand what is the meaning of the overcommit and usage percentage values shown in dashboard. For examlle, for the cpu, documentation simply states that indicates the ratio of virtual/physical cores available to "running" VMs.
I have the following cpu values:
86% available of 100%
Overcommit: 38% (allocated 63%).
Bug 1402007 mentions that overcommit (although misleading term) indicates the physical resources allocated to running VMs. While "allocated" is the total physical resource allocated to provisioned VMs, running or not. Still why 86% available? What is the math behind it?
Thanx,
Alex
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
------ENLJO7GAMXWYZM97AQNB5AQA5DYU0F
Content-Type: text/html;
charset=utf-8
Content-Transfer-Encoding: 8bit
Hi all,<br>
<br>
I'm trying to understand what is the meaning of the overcommit and usage percentage values shown in dashboard. For examlle, for the cpu, documentation simply states that indicates the ratio of virtual/physical cores available to "running" VMs.<br>
<br>
I have the following cpu values:<br>
86% available of 100%<br>
Overcommit: 38% (allocated 63%).<br>
<br>
Bug 1402007 mentions that overcommit (although misleading term) indicates the physical resources allocated to running VMs. While "allocated" is the total physical resource allocated to provisioned VMs, running or not. Still why 86% available? What is the math behind it?<br>
<br>
Thanx,<br>
Alex<br>
<br>
<br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.
------ENLJO7GAMXWYZM97AQNB5AQA5DYU0F--
7 years, 11 months
Adding Host Issue
by Fernando Fuentes
I am adding a host to my cluster and is failing with:
Jan 25, 2017 10:18:02 AM
Failed to connect Host gigantes to Storage Pool HyperVirtDC
Jan 25, 2017 10:18:01 AM
Host gigantes cannot access the Storage Domain(s) <UNKNOWN> attached to
the Data Center HyperVirtDC. Setting Host state to Non-Operational.
Jan 25, 2017 10:18:01 AM
VDSM gigantes command failed: Cannot find master domain:
u'spUUID=e13d5091-a60e-433e-86ed-bd36deb7ca50,
msdUUID=0e4ca0da-1721-4ea7-92af-25233a8679e0'
Not sure why this is happening.
Any ideas?
Regards,
--
Fernando Fuentes
ffuentes(a)txweather.org
http://www.txweather.org
7 years, 11 months
ovirt-engine-rename after restore
by Sergei Genchev
Hi,
I am trying to upgrade ovirt from 3.6 to 4.0.
I have backed up configuration on current CenTOS6 oVirt 3.6 manager
with engine-backup, successfully restored it to a new CenTOS7 running
4.0, and trying to run ovirt-engine-rename. It always fails, and I am
not sure where to look:
/usr/share/ovirt-engine/setup/bin/ovirt-engine-rename
[ INFO ] Stage: Initializing
Welcome to the ovirt-engine-rename utility
More details about the operation and possible implications
of running this utility can be found here:
http://www.ovirt.org/Changing_Engine_Hostname
[ INFO ] Stage: Environment setup
Configuration files:
['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf',
'/etc/ovirt-engine-setup.conf.d/10-packaging.conf',
'/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf']
Log file:
/var/log/ovirt-engine/setup/ovirt-engine-rename-20170127124705-a1y8is.log
Version: otopi-1.5.2 (otopi-1.5.2-1.el7.centos)
[ INFO ] Stage: Environment packages setup
[ INFO ] Stage: Programs detection
[ INFO ] Stage: Environment setup
[ INFO ] Stage: Environment customization
New fully qualified server name: []: newname.domain.com
--== PRODUCT OPTIONS ==--
[ INFO ] Stage: Setup validation
[ INFO ] Engine machine hosting Storage Domains
The following Storage Domains use the engine machine as an NFS server:
ISO_DOMAIN
They will be modified to use the new name.
[ INFO ] The following files will be updated:
/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf
/etc/ovirt-engine/imageuploader.conf.d/10-engine-setup.conf
/etc/ovirt-engine/isouploader.conf.d/10-engine-setup.conf
/etc/ovirt-engine/logcollector.conf.d/10-engine-setup.conf
/etc/ovirt-engine/ovirt-vmconsole-proxy-helper.conf.d/10-setup.conf
/etc/pki/ovirt-engine/cert.conf
/etc/pki/ovirt-engine/cert.template
/etc/pki/ovirt-engine/certs/apache.cer
/etc/pki/ovirt-engine/keys/apache.key.nopass
/etc/pki/ovirt-engine/keys/apache.p12
[ INFO ] Stage: Transaction setup
[ INFO ] Stopping engine service
[ INFO ] Stage: Misc configuration
[ INFO ] Stage: Package installation
[ INFO ] Stage: Misc configuration
[ ERROR ] Failed to execute stage 'Misc configuration': 'NoneType'
object is not iterable
[ INFO ] Stage: Clean up
Log file is located at
/var/log/ovirt-engine/setup/ovirt-engine-rename-20170127124705-a1y8is.log
[ INFO ] Generating answer file
'/var/lib/ovirt-engine/setup/answers/20170127124714-rename.conf'
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ ERROR ] Execution of rename failed
The only error log file has is not very helpful:
2017-01-27 12:47:14 ERROR
otopi.plugins.ovirt_engine_common.base.core.misc misc._terminate:148
Execution of rename failed
After that, engine actually gets renamed, but certificates are not
generated for a new name, and this is, I think, is a reason why new
server cannot communicate with vdsm on ovirt nodes.
I do use custom Apache certificates. CA files are restored fine from backup.
Where can I start looking next?
Thank you!
7 years, 11 months
Translate hint
by Gianluca Cecchi
Hello,
I'm checking Italian Translation in 4.0.6 and completing it for 4.1 (now at
92%).
Suppose I found an untranslated / bad translated word: what is the best and
quick way to go to Zanata and find the reference for it and correct?
For example in 4.0.6 I'm in dashboard and at bottom I have the three labels
Alert, Events, Tasks
I click ALerts and then I want to dismiss an Alert because it happened
while I was configuring Power Management: I faked the password to verify
that the test failed and I got the alert
Power Management test failed for Host .....
I rght click on the alert line and I see three options and the first and
third one are untranslated
they are
Dismiss Alert
Display All
(I'm going to send a separate e-mail to ask about alerts and events in
general).
If I go to Zanata, select oVirt 4.0 and Italian language line and use the
search function I don't find anything....
I see there is an option to
Download All for Offline Translation
and
Export Italian Documents to TMX
I have not tried them yet, but I would prefer to directly correct online
while I find some things to correct.
Gianluca
7 years, 11 months
Migrate hosted engine to a new storage domain
by Logan Kuhn
Hi
We are planning on moving to a different storage solution and I'm curious,
is there a way to migrate the hosted engine's storage domain to the new
solution? It's NFS currently and can be NFS on the new storage as well.
>From what I've read it looks like it should be possible to
1. Take a full backup of the engine VM
2. Deploy another hosted engine VM with hosted-engine --deploy
3. Install/configure CentOS 7.3
4. Deploy new engine with engine-setup
5. Then restore the backup into the new VM.
What I'm not sure of is if that backup will contain enough of it's data to
restore to a completely different storage domain?
Also, the engine database is on a remote server, the data warehouse service
and all other aspects of the hosted engine reside on the VM.
Thanks,
Logan
7 years, 11 months
Distribute collection of VMs
by Fredrik Olofsson
We currently use an OVA-file for Virtualbox to distribute a collection of
VMs to our customers. We would like to also support customers using oVirt.
Our VMs work if we convert the disk images to qcow2 and manually configure
the VMs. Specifically the network interface connections needs to be
configured. Is it possible to automatically import the OVA file (from the
many threads here I guess not..)?
What is the preferred method to distribute a collection of VMs for use in
oVirt?
Best regards
/Fredrik Olofsson
7 years, 11 months
