VM hanging at sustained high throughput
by David Johnson
Hi ovirt gurus,
This is an interesting issue, one I never expected to have.
When I push high volumes of writes to my NAS, I will cause VM's to go into
a paused state. I'm looking at this from a number of angles, including
upgrades on the NAS appliance.
I can reproduce this problem at will running a centos 7.9 VM on Ovirt 4.5.
*Questions:*
1. Is my analysis of the failure (below) reasonable/correct?
2. What am I looking for to validate this?
3. Is there a configuration that I can set to make it a little more robust
while I acquire the hardware to improve the NAS?
*Reproduction:*
Standard test of file write speed:
[root@cen-79-pgsql-01 ~]# dd if=/dev/zero of=./test bs=512k count=4096
oflag=direct
4096+0 records in
4096+0 records out
2147483648 bytes (2.1 GB) copied, 1.68431 s, 1.3 GB/s
Give it more data
[root@cen-79-pgsql-01 ~]# dd if=/dev/zero of=./test bs=512k count=12228
oflag=direct
12228+0 records in
12228+0 records out
6410993664 bytes (6.4 GB) copied, 7.22078 s, 888 MB/s
The odds are about 50/50 that 6 GB will kill the VM, but 100% when I hit 8
GB.
*Analysis:*
What I think appears to be happening is that the intent cache on the NAS is
on an SSD, and my VM's are pushing data about three times as fast as the
SSD can handle. When the SSD gets queued up beyond a certain point, the NAS
(which places reliability over speed) says "Whoah Nellie!", and the VM
chokes.
*David Johnson*
2 years, 8 months
ovirt-4.4 morrors failing
by Ayansh Rocks
Hi All,
Maximum mirrors are failing of ovirt-4.4 dependencies....what can be done
here ?
Error: Failed to download metadata for repo 'ovirt-4.4-centos-gluster8':
Cannot prepare internal mirrorlist: No URLs in mirrorlist
Errors during downloading metadata for repository
'ovirt-4.4-openstack-victoria':
- Status code: 404 for
http://mirror.centos.org/centos/8/cloud/x86_64/openstack-victoria/repodat...
(IP: 54.169.224.98)
Error: Failed to download metadata for repo 'ovirt-4.4-openstack-victoria':
Cannot download repomd.xml: Cannot download repodata/repomd.xml: All
mirrors were tried
Errors during downloading metadata for repository
'ovirt-4.4-centos-nfv-openvswitch':
- Status code: 404 for
http://mirror.centos.org/centos/8/nfv/x86_64/openvswitch-2/repodata/repom...
(IP: 13.231.175.254)
Error: Failed to download metadata for repo
'ovirt-4.4-centos-nfv-openvswitch': Cannot download repomd.xml: Cannot
download repodata/repomd.xml: All mirrors were tried
[root@iondelsvr12 yum.repos.d]# dnf install ovirt-hosted-engine-setup -y
Ceph packages for x86_64
0.0 B/s | 0 B 00:00
Errors during downloading metadata for repository
'ovirt-4.4-centos-ceph-pacific':
- Curl error (7): Couldn't connect to server for
http://mirror.centos.org/centos/8/storage/x86_64/ceph-pacific/repodata/re...
[Failed to connect to mirror.centos.org port 80: Connection refused]
Error: Failed to download metadata for repo
'ovirt-4.4-centos-ceph-pacific': Cannot download repomd.xml: Cannot
download repodata/repomd.xml: All mirrors were tried
2 years, 8 months
Unable to migrate VMs to a newly upgraded Ovirt node host
by Giulio Casella
Hi guys,
I just faced a problem after updating a host. I cannot migrate VM to
updated host.
Here's the error I see trying to migrate a VM to that host.
Dec 16 10:13:11 host01.ovn.di.unimi.it systemd[1]: Starting Network
Manager Script Dispatcher Service...
Dec 16 10:13:11 host01.ovn.di.unimi.it libvirtd[5667]: Unable to read
from monitor: Connection reset by peer
Dec 16 10:13:11 host01.ovn.di.unimi.it libvirtd[5667]: internal error:
qemu unexpectedly closed the monitor: 2021-12-16T10:13:00.447480Z
qemu-kvm: -numa node,nodeid=0,cpus=0-15,mem=8192: warning: Parameter
-numa node,mem is deprecated, use -numa node,memdev instead
2021-12-16T10:13:11.158057Z qemu-kvm: Failed to load pckbd:kbd
2021-12-16T10:13:11.158114Z qemu-kvm: error while loading state for
instance 0x0 of device 'pckbd'
2021-12-16T10:13:11.158744Z qemu-kvm: load of migration failed: No such
file or directory
Dec 16 10:13:11 host01.ovn.xx.xxxxx.it kvm[35663]: 0 guests now active
Instead I can start VM on that host, and migrate away VM from that host.
Rolling back to ovirt-node-ng-4.4.9.1-0.20211207.0+1 via host console
restores full functionality.
The affected version is ovirt-node-ng-4.4.9.3-0.20211215.0+1 (and also
previous one, I don't remember precisely, it was another async release).
Any ideas?
TIA,
gc
2 years, 9 months
Install hosted engine using fcoe
by kyles＠aasteel.com
We're building a new cluster, but it utilizes fcoe for our storage. Is there a way to install the hosted engine using fcoe? I am unable to see our LUNs in the ovirt node currently as fcoe isn't enabled.
2 years, 9 months
Deploy ovirt-csi in the kubernetes cluster
by ssarang520＠gmail.com
Hi,
I want to deploy ovirt-csi in the kubernetes cluster. But the guide only has how to deploy to openshift.
How can I deploy the ovirt-csi in the kubernetes cluster? Is there any way to do that?
2 years, 9 months
how to search event not matching a user
by Gianluca Cecchi
Hello,
every event in Advanced view has a field "User".
I'm trying to compose a search in web admin of events with user different
from myuser@internal
It seems I'm not able to get what I want.
I also tried to base attempts on an old 2019 thread (on 4.3.6) where this
queries worked:
Disks: name=engine* or name=host*
Disks: alias=engine* or alias=host*
but now on 4.4.8 gives nothing even if matched.
Any hint and also documentation reference about the correct syntax to use
in 4.4.x?
Thanks,
Gianluca
2 years, 9 months
What happened to oVirt engine-setup?
by Richard W.M. Jones
A while back I had oVirt 4.4.7 installed which I used for testing.
For some reason that installation has died in some way, so I'm trying
to install a fresh new oVirt 4.4.10.
Last time I installed ovirt, it was very easy - I provisioned a couple
of machines, ran engine-setup in one, answered a few questions and
after a few minutes the engine was installed.
Somehow this has changed and now it's really far more complicated,
involving some ansible things and wanting to create VMs and ssh
everywhere.
Can I go back to the old/easy way of installing oVirt engine? And if
so, what happened to the instructions for that?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
2 years, 9 months
Host needs to be reinstalled after configuring power management
by Andrew DeMaria
Hi,
I am running ovirt 4.3 and have found the following action item immediately
after configuring power management for a host:
Host needs to be reinstalled as important configuration changes were
applied on it.
The thing is - I've just freshly installed this host and it seems strange
that I need to reinstall it.
Is there a better way to install a host and configure power management
without having to reinstall it after?
Thanks,
Andrew
2 years, 9 months
Cinderlib RBD ceph template issues
by Sketch
This is on oVirt 4.4.8, engine on CS8, hosts on C8, cluster and DC are
both set to 4.6.
With a newly configured cinderlib/ceph RBD setup. I can create new VM
images, and copy existing VM images, but I can't copy existing template
images to RBD. When I do, I try, I get this error in cinderlib.log (see
below), which sounds like the disk already exists there, but it definitely
does not. This leaves me unable to create new VMs on RBD, only migrate
existing VM disks.
2021-09-01 04:31:05,881 - cinder.volume.driver - INFO - Driver hasn't implemented _init_vendor_properties()
2021-09-01 04:31:05,882 - cinderlib-client - INFO - Creating volume '0e8b9aca-1eb1-4837-ac9e-cb3d8f4c1676', with size '500' GB [5c5d0a6b]
2021-09-01 04:31:05,943 - cinderlib-client - ERROR - Failure occurred when trying to run command 'create_volume': Entity '<class 'cinder.db.sqlalchemy.models.Volume'>' has no property 'glance_metadata' [5c5d0a6b]
2021-09-01 04:31:05,944 - cinder - CRITICAL - Unhandled error
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/cinderlib/objects.py", line 455, in create
self._raise_with_resource()
File "/usr/lib/python3.6/site-packages/cinderlib/objects.py", line 222, in _raise_with_resource
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3.6/site-packages/cinderlib/objects.py", line 448, in create
model_update = self.backend.driver.create_volume(self._ovo)
File "/usr/lib/python3.6/site-packages/cinder/volume/drivers/rbd.py", line 986, in create_volume
features=client.features)
File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
six.reraise(c, e, tb)
File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
rv = meth(*args, **kwargs)
File "rbd.pyx", line 629, in rbd.RBD.create
rbd.ImageExists: [errno 17] RBD image already exists (error creating image)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/base.py", line 399, in _entity_descriptor
return getattr(entity, key)
AttributeError: type object 'Volume' has no attribute 'glance_metadata'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./cinderlib-client.py", line 170, in main
args.command(args)
File "./cinderlib-client.py", line 208, in create_volume
backend.create_volume(int(args.size), id=args.volume_id)
File "/usr/lib/python3.6/site-packages/cinderlib/cinderlib.py", line 175, in create_volume
vol.create()
File "/usr/lib/python3.6/site-packages/cinderlib/objects.py", line 457, in create
self.save()
File "/usr/lib/python3.6/site-packages/cinderlib/objects.py", line 628, in save
self.persistence.set_volume(self)
File "/usr/lib/python3.6/site-packages/cinderlib/persistence/dbms.py", line 254, in set_volume
self.db.volume_update(objects.CONTEXT, volume.id, changed)
File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 236, in wrapper
return f(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 184, in wrapper
return f(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 2570, in volume_update
result = query.filter_by(id=volume_id).update(values)
File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3818, in update
update_op.exec_()
File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1670, in exec_
self._do_pre_synchronize()
File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1743, in _do_pre_synchronize
self._additional_evaluators(evaluator_compiler)
File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1912, in _additional_evaluators
values = self._resolved_values_keys_as_propnames
File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1831, in _resolved_values_keys_as_propnames
for k, v in self._resolved_values:
File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1818, in _resolved_values
desc = _entity_descriptor(self.mapper, k)
File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/base.py", line 402, in _entity_descriptor
"Entity '%s' has no property '%s'" % (description, key)
sqlalchemy.exc.InvalidRequestError: Entity '<class 'cinder.db.sqlalchemy.models.Volume'>' has no property 'glance_metadata'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./cinderlib-client.py", line 390, in <module>
sys.exit(main(sys.argv[1:]))
File "./cinderlib-client.py", line 176, in main
sys.stderr.write(traceback.format_exc(e))
File "/usr/lib64/python3.6/traceback.py", line 167, in format_exc
return "".join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
File "/usr/lib64/python3.6/traceback.py", line 121, in format_exception
type(value), value, tb, limit=limit).format(chain=chain))
File "/usr/lib64/python3.6/traceback.py", line 498, in __init__
_seen=_seen)
File "/usr/lib64/python3.6/traceback.py", line 498, in __init__
_seen=_seen)
File "/usr/lib64/python3.6/traceback.py", line 509, in __init__
capture_locals=capture_locals)
File "/usr/lib64/python3.6/traceback.py", line 338, in extract
if limit >= 0:
TypeError: '>=' not supported between instances of 'InvalidRequestError' and 'int'
2 years, 9 months
gerrit.ovirt.org upgrade
by Denis Volkov
Hello
Gerrit.ovirt.org will be upgraded later today. Approximate start time is
4PM UTC.
During the upgrade service will be shut down and so will not be available.
--
Denis Volkov
2 years, 9 months
