ovirt-imagio-proxy upload speed slow
by Dev Ops
I am working on integrating a backup solution for our ovirt environment and having issues with the time it takes to backup the VM's. This backup solution is simply taking a snapshot and making a clone and backing the clone up to a backup server.
A VM that is 100 gig takes 52 minutes to back up. The same VM doing a file backup using the same product, and bypassing their rhv plugin, takes 14 minutes. So the throughput is there but the ovirt imageio-proxy process seems to be what manages how images are uploaded and is officially my bottle neck. Load is not high on the engine or kvm hosts.
I had bumped up the Upload image size from 100MB to 10gig weeks ago and that didn't seem to help.
[root@blah-lab-engine ~]# engine-config -a |grep Upload
UploadImageChunkSizeKB: 10240000 version: general
[root@bgl-vms-engine ~]# rpm -qa |grep ovirt-image
ovirt-imageio-proxy-1.4.6-1.el7.noarch
ovirt-imageio-common-1.4.6-1.el7.x86_64
ovirt-imageio-proxy-setup-1.4.6-1.el7.noarch
I have seen bugs reported to redhat about this but I am running above the affected releases.
engine software is 4.2.8.2-1.el7
Any idea what we can tweak to open up this bottleneck?
5 years, 3 months
Does cluster upgrade wait for heal before proceeding to next host?
by Jayme
I've yet to have cluster upgrade finish updating my three host HCI
cluster. The most recent try was today moving from oVirt 4.3.3 to
4.3.5.5. The first host updates normally, but when it moves on to the
second host it fails to put it in maintenance and the cluster upgrade
stops.
I suspect this is due to that fact that after my hosts are updated it takes
10 minutes or more for all volumes to sync/heal. I have 2Tb SSDs.
Does the cluster upgrade process take heal time in to account before
attempting to place the next host in maintenance to upgrade it? Or is there
something else that may be at fault here, or perhaps a reason why the heal
process takes 10 minutes after reboot to complete?
5 years, 3 months
Re: VMs inaccessible from Ubuntu/Debian-based OS
by Strahil
Hi, I have just tested ovirt VNC console (.vv file) with remote-viewer and it works.
Ensure that you have acees to hosts' ports 5900-5999 , as this is the VNC port.
Otherwise you can configure noVNC, but it requires port 6100 (based on memory).
Best Regards,
Strahil NikolovOn Aug 30, 2019 20:22, Souvalioti Maria <souvaliotimaria(a)mail.com> wrote:
>
> Hello Strahil and thank you for your reply.
>
> Virt-viewer is installed but mozilla is trying to open the console.vv by remote-viewer. In my understanding virt-viewer and remote-viewer are somewhat the same? (Remote-viewer usin virt-viewer) Should I access the VM's console from the ternimal? I have tried changing the VM's graphics type from spice to spice+vnc but still nothing. All I get is a blank window and a message stating that it cannot access the graphic server.
>
> Thanks again for your help.
>
> Maria Souvalioti
>
>
> Sent from BlueMail
> On 30 Aug 2019, at 7:48 pm, Strahil <hunter86_bg(a)yahoo.com> wrote:
>>
>> You need virt-viewer installed on your ubuntu or change console type to spice.
>>
>>
>> Best Regards,
>> Strahil NikolovOn Aug 30, 2019 15:35, souvaliotimaria(a)mail.com wrote:
>>>
>>>
>>> Hello all,
>>>
>>> I'm having an issue the past couple of days. I have tried anything I could find to solve this but with no success.
>>>
>>> I have a three node ovirt installation, glustered, hyperconverged and there i have a few VMs. The installation is for experimental reasons before we merge it in our DC. Anyway, though I can connect to the VMs' console from Fedora and Windows, I can't from Ubuntu.
>>>
>>> I have tried installing the browser-spice-plugin that's the corresponding package to spice-xpi and I got no results.
>>>
>>> I purged the browser-spice-plugin and then I installed the spice-client package, downloaded the spice-xpi from Fedora (FC19) (as instructed in https://www.ovirt.org/develop/infra/testing/spice.html), copied the libnsISpicec.so to /usr/lib/mozilla/plugins and made sure that xserver-xorg-video-qxl and spice-vdagent were installed and in their latest version, and still nothing.
>>>
>>> No matter what I have tried, I can't gain access to the console. No matter the browser I use, the message I get is "Unable to connect to the graphic server /tmp/mozilla_msouval0/console-1.vv".
>>>
>>> I have checked the logs and couldn't find anything useful, maybe I'm not checking the right logs?
>>> I run tcpdump on both the node the VM is being hosted and the Ubuntu machine I'm using and though on the UBuntu I capture a few packets (6) on both sides but on the Ubuntu side there were 15 packets received by the filter.
>>>
>>> Could you please guide towards the right way to solve this? Could this be an ntp problem?
>>>
>>> ovirt node version: 4.2.3.1
>>> Workstation: Jessie
>>>
>>> (I thought maybe Jessie was too old, and I run the same steps from Ubuntu 14.04 and Ubuntu 16.04, but the problem still remains)
>>>
>>> Thank you in advance for any help.
>>> Maria
>>> ________________________________
>>>
>>> Users mailing list -- users(a)ovirt.org
>>> To unsubscribe send an email to users-leave(a)ovirt.org
>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/4VMMYEJNSHG...
5 years, 3 months
Health Warning: Cinnamon toxic when adding EL7 nodes as oVirt hosts
by thomas＠hoberg.net
Of course a desktop on a virtualization host isn't really recommended, but actually, managing highly or even high-available supports services as VMs on a set of fat workstations under the control of oVirt can be rather useful...
And I prefer Cinnamon so much over the other desktops, it gets installed almost first after setup... Which turns out to be a mistake with oVirt.
Anyhow, to make a long journey of hair-pulling root-cause searching short:
Installing Cinnamon from EPEL has Otopi not find rpmUtils for one reason or another, even if I can see it on both sides, the engine and the target host.
Remove Cinnamon, it works again and you can re-install Cinnamon after.
I didn't test re-deploys, but I'd expect them to fail.
Don't know if this should be a bug report, since Cinnamon very unfortunately doesn't seem official just yet.
This is what you may expect to see in the engine logs:
2019-08-22 09:49:24,152+0200 DEBUG otopi.context context._executeMethod:145 method exception
Traceback (most recent call last):
File "/tmp/ovirt-BAJG2FYLMu/pythonlib/otopi/context.py", line 132, in _executeMethod
method['method']()
File "/tmp/ovirt-BAJG2FYLMu/otopi-plugins/ovirt-host-deploy/kdump/packages.py", line 216, in _customization
self._kexec_tools_version_supported()
File "/tmp/ovirt-BAJG2FYLMu/otopi-plugins/ovirt-host-deploy/kdump/packages.py", line 143, in _kexec_tools_version_supported
from rpmUtils.miscutils import compareEVR
ModuleNotFoundError: No module named 'rpmUtils'
2019-08-22 09:49:24,152+0200 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Environment customization': No module named 'rpmUtils'
5 years, 3 months
ovirt with glusterfs data domain - very slow writing speed on Windows server virtual machine
by lyubomir.grancharov＠bottleshipvfx.com
Hi folks,
I have been experimenting with oVirt cluster based on glusterfs for the past few days. (first-timer). The cluster is up and running and it consists of 4 nodes and has 4 replicas. When I try to deploy Windows Server VM I encounter the following issue: The disk of the VM has ok reading speed ( close to bare metal) but the writing speed is very slow. ( about 10 times slower than it is supposed to be). Can anyone give me any suggestion, please? Thanks in advance! Here are the settings of the glusterfs volume:
Volume Name: bottle-volume
Type: Replicate
Volume ID: 869b8d1e-1266-4820-8dcd-4fea92346b90
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 4 = 4
Transport-type: tcp
Bricks:
Brick1: cnode01.bottleship.local:/gluster_bricks/brick-cnode01/brick-cnode01
Brick2: cnode02.bottleship.local:/gluster_bricks/brick-cnode02/brick-cnode02
Brick3: cnode03.bottleship.local:/gluster_bricks/brick-cnode03/brick-cnode03
Brick4: cnode04.bottleship.local:/gluster_bricks/brick-cnode04/brick-cnode04
Options Reconfigured:
network.ping-timeout: 30
cluster.granular-entry-heal: enable
performance.strict-o-direct: on
storage.owner-gid: 36
storage.owner-uid: 36
server.event-threads: 4
client.event-threads: 4
cluster.choose-local: off
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.server-quorum-type: server
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: off
performance.low-prio-threads: 32
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
auth.allow: *
user.cifs: off
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
server.allow-insecure: on
Please let me know if you need any more configuration info or hardware specs.
best,
Lyubo
5 years, 3 months
Procedure to replace out out of three hyperconverged nodes
by thomas＠hoberg.net
On my silent Atom based three node Hyperconverged journey I hit upon a snag: Evidently they are too slow for Ansible.
The Gluster storage part went all great and perfect on fresh oVirt node images that I had configured to leave an empty partition instead of the standard /dev/sdb, but the HostedEngine setup part would then fail without any log-visible error while the transient VM HostedEngineLocal was supposed to be launched and the Wizard would just show "deployment failed" and go ahead and delete the VM.
I then moved the SSD to a more powerful Xeon-D 1541 CPU and after some fiddling with the network (I miss good old eth0!), this also failed the deployment, but also failed to delete the temporary VM image, because that actually turned out to be running: I could even connect to its console and investigate the logs for any clues as to what might have gone wrong (nothing visible): Evidently Ansible was running out of patience just a tiny bit too early.
And then I kicked it into high-gear with an i7-7700K again using the same SSD with a working three node Gluster all in sync, which still took what felt like an hour to creep through every step, but got it done, primary node on i7, secondary nodes on Atoms, with full migration capabilities etc.
I then had to do some fiddling, because the HostedEngine had configued the Cluster CPU architecture to Skylake-Spectre, but after that I migrated it to an Atom node and was now ready to move the primary to the intended Atom hardware target.
But at that point the overlay network has already been configured and evidently it's tied to the device name of the 10Gbit NIC on the i7 workstation and I haven't been able to make it work with the Atom. The Gluster runs fine, but the CPU node is reported "non-operational" and re-installation fails, because the ovirtmgmt network isn't properly configured.
That specific issue may be seem way out of what oVirt should support, yet a HA-embedded edge platform may very well see nodes having to be replaced or renewed with as little interruption or downtime as possible, which is why I am asking the larger question:
How can you replace a) a failed "burned" node or b) upgrade nodes while maintaining fault tolerance?
The distinction in b) would be that it's a planned maneuver during normal operations without downtime.
I'd want to do it pretty much like I have been playing with compute nodes, creating new ones, pushing VMs on them, pushing them out to other hosts, removing and replacing them seamlessly... Except that the Gluster nodes are special and much harder to replace, than a pure Gluster storage brick... from what I see
I welcome any help
- for fixing the network config in my limiping Atom 1:3 cluster
- eliminating the need to fiddle with an i7 because of Ansible timing
- ensuring long-term operability of a software defined datacenter with changing hardware
5 years, 3 months
Any real experiences with nested virtualisation?
by tonyppe＠gmail.com
I have ovirt 4.3.5 and Openstack hosts. There are 2 x openstack compute and 2 x ovirt hosts with hosted manager. I think i can simplify the management of this if I were to virtualise the openstack compute nodes as VMs on ovirt.
I wanted to reach out here to gauge any other users experiences with it if at all. From the research I have done it looks like from the VM perspective, migration must not be allowed.
Any info / comments / feedback appreciated.
Regards,
5 years, 3 months
Re: Cannot Increase Hosted Engine VM Memory
by Douglas Duckworth
Yes, I do. Gold crown indeed.
It's the "HostedEngine" as seen attached!
Thanks,
Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit<https://scu.med.cornell.edu>
Weill Cornell Medicine
1300 York Avenue
New York, NY 10065
E: doug(a)med.cornell.edu<mailto:doug@med.cornell.edu>
O: 212-746-6305
F: 212-746-8690
On Wed, Jan 23, 2019 at 12:02 PM Simone Tiraboschi <stirabos(a)redhat.com<mailto:stirabos@redhat.com>> wrote:
On Wed, Jan 23, 2019 at 5:51 PM Douglas Duckworth <dod2014(a)med.cornell.edu<mailto:dod2014@med.cornell.edu>> wrote:
Hi Simone
Can I get help with this issue? Still cannot increase memory for Hosted Engine.
From the logs it seams that the engine is trying to hotplug memory to the engine VM which is something it should not happen.
The engine should simply update engine VM configuration in the OVF_STORE and require a reboot of the engine VM.
Quick question, in the VM panel do you see a gold crown symbol on the Engine VM?
Thanks,
Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit<https://scu.med.cornell.edu>
Weill Cornell Medicine
1300 York Avenue
New York, NY 10065
E: doug(a)med.cornell.edu<mailto:doug@med.cornell.edu>
O: 212-746-6305
F: 212-746-8690
On Thu, Jan 17, 2019 at 8:08 AM Douglas Duckworth <dod2014(a)med.cornell.edu<mailto:dod2014@med.cornell.edu>> wrote:
Sure, they're attached. In "first attempt" the error seems to be:
2019-01-17 07:49:24,795-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-29) [680f82b3-7612-4d91-afdc-43937aa298a2] EVENT_ID: FAILED_HOT_SET_MEMORY_NOT_DIVIDABLE(2,048), Failed to hot plug memory to VM HostedEngine. Amount of added memory (4000MiB) is not dividable by 256MiB.
Followed by:
2019-01-17 07:49:24,814-05 WARN [org.ovirt.engine.core.bll.UpdateRngDeviceCommand] (default task-29) [26f5f3ed] Validation of action 'UpdateRngDevice' failed for user admin@internal-authz. Reasons: ACTION_TYPE_FAILED_VM_IS_RUNNING
2019-01-17 07:49:24,815-05 ERROR [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-29) [26f5f3ed] Updating RNG device of VM HostedEngine (adf14389-1563-4b1a-9af6-4b40370a825b) failed. Old RNG device = VmRngDevice:{id='VmDeviceId:{deviceId='6435b2b5-163c-4f0c-934e-7994da60dc89', vmId='adf14389-1563-4b1a-9af6-4b40370a825b'}', device='virtio', type='RNG', specParams='[source=urandom]', address='', managed='true', plugged='true', readOnly='false', deviceAlias='', customProperties='null', snapshotId='null', logicalName='null', hostDevice='null'}. New RNG device = VmRngDevice:{id='VmDeviceId:{deviceId='6435b2b5-163c-4f0c-934e-7994da60dc89', vmId='adf14389-1563-4b1a-9af6-4b40370a825b'}', device='virtio', type='RNG', specParams='[source=urandom]', address='', managed='true', plugged='true', readOnly='false', deviceAlias='', customProperties='null', snapshotId='null', logicalName='null', hostDevice='null'}.
In "second attempt" I used values that are dividable by 256 MiB so that's no longer present. Though same error:
2019-01-17 07:56:59,795-05 INFO [org.ovirt.engine.core.vdsbroker.SetAmountOfMemoryVDSCommand] (default task-22) [7059a48f] START, SetAmountOfMemoryVDSCommand(HostName = ovirt-hv1.med.cornell.edu<http://ovirt-hv1.med.cornell.edu>, Params:{hostId='cdd5ffda-95c7-4ffa-ae40-be66f1d15c30', vmId='adf14389-1563-4b1a-9af6-4b40370a825b', memoryDevice='VmDevice:{id='VmDeviceId:{deviceId='7f7d97cc-c273-4033-af53-bc9033ea3abe', vmId='adf14389-1563-4b1a-9af6-4b40370a825b'}', device='memory', type='MEMORY', specParams='[node=0, size=2048]', address='', managed='true', plugged='true', readOnly='false', deviceAlias='', customProperties='null', snapshotId='null', logicalName='null', hostDevice='null'}', minAllocatedMem='6144'}), log id: 50873daa
2019-01-17 07:56:59,855-05 INFO [org.ovirt.engine.core.vdsbroker.SetAmountOfMemoryVDSCommand] (default task-22) [7059a48f] FINISH, SetAmountOfMemoryVDSCommand, log id: 50873daa
2019-01-17 07:56:59,862-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-22) [7059a48f] EVENT_ID: HOT_SET_MEMORY(2,039), Hotset memory: changed the amount of memory on VM HostedEngine from 4096 to 4096
2019-01-17 07:56:59,881-05 WARN [org.ovirt.engine.core.bll.UpdateRngDeviceCommand] (default task-22) [28fd4c82] Validation of action 'UpdateRngDevice' failed for user admin@internal-authz. Reasons: ACTION_TYPE_FAILED_VM_IS_RUNNING
2019-01-17 07:56:59,882-05 ERROR [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-22) [28fd4c82] Updating RNG device of VM HostedEngine (adf14389-1563-4b1a-9af6-4b40370a825b) failed. Old RNG device = VmRngDevice:{id='VmDeviceId:{deviceId='6435b2b5-163c-4f0c-934e-7994da60dc89', vmId='adf14389-1563-4b1a-9af6-4b40370a825b'}', device='virtio', type='RNG', specParams='[source=urandom]', address='', managed='true', plugged='true', readOnly='false', deviceAlias='', customProperties='null', snapshotId='null', logicalName='null', hostDevice='null'}. New RNG device = VmRngDevice:{id='VmDeviceId:{deviceId='6435b2b5-163c-4f0c-934e-7994da60dc89', vmId='adf14389-1563-4b1a-9af6-4b40370a825b'}', device='virtio', type='RNG', specParams='[source=urandom]', address='', managed='true', plugged='true', readOnly='false', deviceAlias='', customProperties='null', snapshotId='null', logicalName='null', hostDevice='null'}.
This message repeats throughout engine.log:
2019-01-17 07:55:43,270-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-89) [] EVENT_ID: VM_MEMORY_UNDER_GUARANTEED_VALUE(148), VM HostedEngine on host ovirt-hv1.med.cornell.edu<http://ovirt-hv1.med.cornell.edu> was guaranteed 8192 MB but currently has 4224 MB
As you can see attached the host has plenty of memory.
Thank you Simone!
Thanks,
Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit<https://scu.med.cornell.edu>
Weill Cornell Medicine
1300 York Avenue
New York, NY 10065
E: doug(a)med.cornell.edu<mailto:doug@med.cornell.edu>
O: 212-746-6305
F: 212-746-8690
On Thu, Jan 17, 2019 at 5:09 AM Simone Tiraboschi <stirabos(a)redhat.com<mailto:stirabos@redhat.com>> wrote:
On Wed, Jan 16, 2019 at 8:22 PM Douglas Duckworth <dod2014(a)med.cornell.edu<mailto:dod2014@med.cornell.edu>> wrote:
Sorry for accidental send.
Anyway I try to increase physical memory however it won't go above 4096MB. The hypervisor has 64GB.
Do I need to modify this value with Hosted Engine offline?
No, it's not required.
Can you please attach your engine.log for the relevant time frame?
Thanks,
Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit<https://scu.med.cornell.edu>
Weill Cornell Medicine
1300 York Avenue
New York, NY 10065
E: doug(a)med.cornell.edu<mailto:doug@med.cornell.edu>
O: 212-746-6305
F: 212-746-8690
On Wed, Jan 16, 2019 at 1:58 PM Douglas Duckworth <dod2014(a)med.cornell.edu<mailto:dod2014@med.cornell.edu>> wrote:
Hello
I am trying to increase Hosted Engine physical memory above 4GB
Thanks,
Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit<https://scu.med.cornell.edu>
Weill Cornell Medicine
1300 York Avenue
New York, NY 10065
E: doug(a)med.cornell.edu<mailto:doug@med.cornell.edu>
O: 212-746-6305
F: 212-746-8690
_______________________________________________
Users mailing list -- users(a)ovirt.org<mailto:users@ovirt.org>
To unsubscribe send an email to users-leave(a)ovirt.org<mailto:users-leave@ovirt.org>
Privacy Statement: https://www.ovirt.org/site/privacy-policy/<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_site_p...>
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_commun...>
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WGSXQVVPJJ2...<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ovirt.org_arch...>
5 years, 3 months
engine.log flooded with "Field 'foo' can not be updated when status is 'Up'"
by Matthias Leopold
Hi,
I updated my production oVirt environment from 4.3.3 to 4.3.5 today.
Everything went fine so far, but there's one annoying phenomenon:
When I log into the "Administration Portal" and request the VM list
("/ovirt-engine/webadmin/?locale=en_US#vms") engine.log is flooded with
lines like
WARN [org.ovirt.engine.core.utils.ObjectIdentityChecker] (default
task-10618) [54d8c375-aa72-42f8-876e-8777d9d1a08a] Field
'balloonEnabled' can not be updated when status is 'Up'
"Field", task and UUID vary and the flood stops after a while. Also
listing or trying to edit other entities seems to trigger this "storm"
or loop over and over again to a point that log file size is becoming an
issue and interface is becoming sluggish. I can also see that CPU usage
of engine java process goes up. When I log out everything is quiet and
"VM Portal" is not affected at all.
I have seen lines like that before and know that they are usually OK
(when changing VM properties), but these logs used to be linked to
singular events. I suspect that the present behaviour might be linked to
VMs that have "Pending Virtual Machine changes", which are in most cases
"Custom Compatibility Version" changes that still stem from the upgrade
to Cluster Version 4.3. I can't be sure and I can't resolve all these
pending changes now, but these should not be causing such annoying
behaviour in the first place.
I resorted to setting engine log level to "ERROR" right now to at least
stop the log file from growing, but this not a solution. I can still see
CPU load going up when using the interface. I very much hope that
someone can explain whats happening and tell me how to resolve this.
thanks a lot
Matthias
5 years, 3 months
Intel Xeon X5460
by Marcello Gentile
Is this procesor supported?
https://ark.intel.com/content/www/us/en/ark/products/33087/intel-xeon-pro...
Seems to meet all the requirements but when I install the self hosted engine it fails with the error message below:
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, deployment errors: code 156: Host Ovirt-Node1.adsdata.ca<http://Ovirt-Node1.adsdata.ca> moved to Non-Operational state as host CPU type is not supported in this cluster compatibility version or is not supported at all, code 9000: Failed to verify Power Management configuration for Host Ovirt-Node1.adsdata.ca<http://Ovirt-Node1.adsdata.ca>., fix accordingly and re-deploy."}
Thanks for any help
5 years, 3 months
