Seeking best performance on oVirt cluster
by David Johnson
Good morning all,
I am trying to get the best performance out of my cluster possible,
Here are the details of what I have now:
Ovirt version: 4.4.10.7-1.el8
Bare metal for the ovirt engine
two hosts
TrueNAS cluster storage
1 NFS share
3 vdevs, 6 drives in raidz2 in each vdev
2 nvme drives for silog
Storage network is 10 GBit all static IP addresses
Tonight, I built a new VM from a template. It had 5 attached disks
totalling 100 GB. It took 30 minutes to deploy the new VM from the
template.
Global utilization was 9%.
The SPM has 50% of its memory free and never showed more than 12% network
utilization
62 out of 65 TB are available on the newly created NFS backing store (no
fragmentation). The TureNAS system is probably overprovisioned for our use.
There were peak throughputs of up to 4 GBytes/second (on a 10 GBit
network), but overall throughput on the NAS and the network were low.
ARC hits were 95 to 100%
L2 hits were 0 to 70%
Here's the NFS usage stats:
[image: image.png]
I believe the first peak is where the silog buffered the initial burst of
instructions, followed by sustained IO as the VM volumes were built in
parallel, and then finally tapering off to the one 50 GB volume that took
40 minutes to copy.
The indications of the NFS stats graph are that the network performance is
just fine.
Here are the disk IO stats covering the same time frame, plus a bit before
to show a spike IO:
[image: image.png]
The spike at 2250 (10 minutes before I started building my VM) shows that
the spinners actually hit write speed of almost 20 MBytes per second
briefly, then settled in at a sustained 3 to 4 MBytes per second. The
silog absorbs several spikes, but remains mostly idle, with activity
measured in kilobytes per second.
The HGST HUS726060AL5210 drives boast a spike throughput of 12 GB/S, and
sustained throughput of 227 Mbps.
------
Now to the questions:
1. Am I asking the on the right list? Does this look like something where
tuning ovirt might make a difference, or is this more likely a
configuration issue with my storage appliances?
2. Am I expecting too much? Is this well within the bounds of acceptable
(expected) performance?
3. How would I go about identifying the bottleneck, should I need to dig
deeper?
Thanks,
David Johnson
2 years, 5 months
unable to create iso domain
by Moritz Baumann
Hi
I have removed the iso domain of an existing data center, and now I am
unable to create a new iso domain
/var/log/ovirt-engine/engine.log shows:
2022-07-14 08:04:40,684+02 INFO
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand]
(default task-34) [8db814e3-43ab-4921-ad35-2b3acd51c385] Lock Acquired
to object
'EngineLock:{exclusiveLocks='[ovirt.storage.inf.ethz.ch:/export/ovirt/iso=STORAGE_CONNECTION]',
sharedLocks=''}'
2022-07-14 08:04:40,689+02 WARN
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand]
(default task-34) [8db814e3-43ab-4921-ad35-2b3acd51c385] Validation of
action 'AddStorageServerConnection' failed for user
xxx@ethz.ch(a)ethz.ch-authz. Reasons:
VAR__ACTION__ADD,VAR__TYPE__STORAGE__CONNECTION,$connectionId
c39c64ef-fb8b-4e87-9803-420c7fb2dd4a,$storageDomainName
,ACTION_TYPE_FAILED_STORAGE_CONNECTION_ALREADY_EXISTS
2022-07-14 08:04:40,690+02 INFO
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand]
(default task-34) [8db814e3-43ab-4921-ad35-2b3acd51c385] Lock freed to
object
'EngineLock:{exclusiveLocks='[ovirt.scratch.inf.ethz.ch:/export/ovirt/iso=STORAGE_CONNECTION]',
sharedLocks=''}'
2022-07-14 08:04:40,756+02 INFO
[org.ovirt.engine.core.bll.storage.connection.DisconnectStorageServerConnectionCommand]
(default task-34) [4148e0fd-58ae-4375-8dc8-a08f47402ed6] Running
command: DisconnectStorageServerConnectionCommand internal: false.
Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type:
SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2022-07-14 08:04:40,756+02 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand]
(default task-34) [4148e0fd-58ae-4375-8dc8-a08f47402ed6] START,
DisconnectStorageServerVDSCommand(HostName = ovirt-node01,
StorageServerConnectionManagementVDSParameters:{hostId='d942c8fe-9a0c-4761-9be2-2f88b622070b',
storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS',
connectionList='[StorageServerConnections:{id='null',
connection='ovirt.storage.inf.ethz.ch:/export/ovirt/iso', iqn='null',
vfsType='null', mountOptions='null', nfsVersion='null',
nfsRetrans='null', nfsTimeo='null', iface='null',
netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 3043bbfd
2022-07-14 08:04:43,017+02 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand]
(default task-34) [4148e0fd-58ae-4375-8dc8-a08f47402ed6] FINISH,
DisconnectStorageServerVDSCommand, return:
{00000000-0000-0000-0000-000000000000=100}, log id: 3043bbfd
[root@ovirt-engine ovirt-engine]# showmount -e ovirt.storage.inf.ethz.ch
| grep ovirt
Export list for ovirt.scratch.inf.ethz.ch:
/export/ovirt/export @ovirt-storage
/export/ovirt/data @ovirt-storage
/export/ovirt/iso @ovirt-storage
the other two domains still work just fine and the netgroup contains all
ovirt-nodes.
storage-node1[0]:/export/ovirt/iso# ls -la
total 2
drwx------. 2 vdsm kvm 2 Jul 14 07:58 .
drwxr-xr-x. 5 root root 5 Aug 19 2020 ..
storage-node1[0]:/export/ovirt/iso# df .
Filesystem 1K-blocks Used Available Use% Mounted on
fs1/ovirt/iso 524288000 256 524287744 1% /export/ovirt/iso
storage-node1[0]:/export/ovirt/iso#
storage-node1[0]:/export/ovirt/iso# exportfs -v | grep ovirt/ -A1
/export/ovirt/iso
@ovirt-storage(sync,wdelay,hide,no_subtree_check,fsid=215812,sec=sys:krb5:krb5i:krb5p,rw,secure,root_squash,no_all_squash)
/export/ovirt/data
@ovirt-storage(sync,wdelay,hide,no_subtree_check,fsid=215811,sec=sys:krb5:krb5i:krb5p,rw,secure,root_squash,no_all_squash)
--
/export/ovirt/export
@ovirt-storage(sync,wdelay,hide,no_subtree_check,fsid=215813,sec=sys:krb5:krb5i:krb5p,rw,secure,root_squash,no_all_squash)
It appears that there is stille some reference to an iso domain
(c39c64ef-fb8b-4e87-9803-420c7fb2dd4a ??) in the database. How can I get
rid of it ?
Best
Moritz
2 years, 5 months
4.3.10 cannot start VMs because of an error with USB
by Pascal D
I am still running 4.3.10 and suddenly I get this error everytime I restart a VM, any Vm on any host (I have 13 hosts in 2 different clusters)
VM FLEETGEN-PCC-001 is down with error. Exit message: XML error: there is no hub at port 1 in USB address bus: 0 port: 1.1.
Any idea what could be different. Template hasn't changed
2 years, 5 months
Import KVM VMs on individual iSCSI luns
by spierce＠cts1.com
Greetings,
Is it possible with oVirt to import existing VMs where the underlying storage is on raw iSCSI luns and to keep them on those luns?
The historical scenario is that we have Virtual farms in multiple sites managed by an ancient Orchestration tool that does not support modern OS's as the hypervisor.
- In each site, there are clusters of hypervisors/Hosts that have visibility to the same iSCSI luns.
- Each VM has it's own set of iscsi luns that are totally dedicated to that VM
- Each VM is using LVM to manage the disk
- Each Host has LVM filtering configured to NOT manage the VM's iscsi luns
- The VMs can be live migrated from any Hypervisor within the cluster to any other Hypervisor in that same cluster
We are attempting to bring this existing environment into oVirt without replacing the storage model.
Is there any documentation that will serve as a guide for this scenario?
In a lab environment, we have successfully
- Added 2 hypervisors (hosts) and oVirt can see their VMs as external-ovtest1 and external-ovtest2
- Removed the LVM filtering on the hosts
- Created a storage domain that is able to see the iscsi luns, but we have not yet done the 'add' of each lun
Is it possible to import these luns as raw block devices without LVM being layered on top of them?
Is it required to actually import the luns into a storage domain, or can the VM's still be imported if all luns are visible on all hosts in the cluster?
In the grand scheme of things, are we trying to do something that is not possible with oVirt?
If it is possible, we would greatly appreciate tips, pointers, links to docs etc that will help us migrate this environment to oVirt.
Thanks in Advance
- S
2 years, 5 months
Q: Instaling Ovirt Engine 4.4.10 on Clean CentOS 8 Stream
by Andrei Verovski
Hi,
Since I run into a lot of problems upgrading 4.4.7 to 4.4.10, is it OK to install clean
CentOS-Stream-8-x86_64-20220712-dvd1.iso
and restore from backup?
4.4.10 release long before Stream 20220712, may I run again into similar problems, or I need to use earlier snapshot of Stream 8 ?
I use dedicated engine PC, not hosted engine.
Thanks in advance for any suggestion(s)
Andrei
2 years, 5 months
Keycloak - the default OpenID/SSO provider for oVirt Engine
by Artur Socha
Hi,
With Ovirt 4.5.1 release [1], the Keycloak based authentication is enabled
by default for fresh/new installations.
Here [2] you can find some usage scenarios describing when/how it is
enabled.
In short - if you just want to login to oVirt Admin / VM / Monitoring
portal, please use 'admin@ovirt' user and the password provided during
engine-setup.
There is ongoing work to make it more explicit [3] and it will be addressed
soon.
For Rest API access, the full user with profile name is required as
username: admin@ovirt@internalsso
Here is a sample 'curl' illustrating the flow:
$ curl -k -H "Accept: application/json" '
https://ENGINE_FQDN/ovirt-engine/sso/oauth/token?grant_type=password&user...
'
And the token response:
{"access_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJEdS10MlVQd0JaZ0gtRU1JUkRTRHFxNFZIOUhZbnc4Nkk5QUlGOERxZ1l3In0.eyJleHAiOjE2NTcyMTY5MzAsImlhdCI6MTY1NzE5OTY1MCwianRpIjoiNTAwOWVkMmItMjc3ZS00YjVmLWEwOTItMjI4MDhkMWFhMWJjIiwiaXNzIjoiaHR0cHM6Ly9kZXYzLmRvbS9vdmlydC1lbmdpbmUtYXV0aC9yZWFsbXMvb3ZpcnQtaW50ZXJuYWwiLCJhdWQiOiJhY2NvdW50Iiwic3ViIjoiMjczMTlkODMtYjdkYy00MzU2LTllMmQtYjJmNzg5NWI3YjczIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoib3ZpcnQtZW5naW5lLWludGVybmFsIiwic2Vzc2lvbl9zdGF0ZSI6IjNmODM5NjY2LTQyNzUtNDRhNC1hNDRhLTM3Njc5MzgxNDRiOCIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiaHR0cHM6Ly9kZXYzLmRvbSJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiZGVmYXVsdC1yb2xlcy1vdmlydC1pbnRlcm5hbCIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im92aXJ0LWV4dD10b2tlbi1pbmZvOnB1YmxpYy1hdXRoei1zZWFyY2ggb3ZpcnQtYXBwLWFwaSBvdmlydC1leHQ9dG9rZW46cGFzc3dvcmQtYWNjZXNzIG92aXJ0LWV4dD10b2tlbi1pbmZvOmF1dGh6LXNlYXJjaCBvdmlydC1leHQ9dG9rZW4taW5mbzp2YWxpZGF0ZSBlbWFpbCBwcm9maWxlIiwic2lkIjoiM2Y4Mzk2NjYtNDI3NS00NGE0LWE0NGEtMzc2NzkzODE0NGI4IiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJncm91cHMiOlsiL292aXJ0LWFkbWluaXN0cmF0b3IiXSwicHJlZmVycmVkX3VzZXJuYW1lIjoiYWRtaW5Ab3ZpcnQiLCJlbWFpbCI6ImFkbWluQGxvY2FsaG9zdCJ9.Ov2IJ-ghtXSB6eb7osWZgT_yeb4prBgVzUU9vAY_VMoDr-ie5bMYBUyinYvNHWpBbYaFGNjg6bC7PHz3-s5H1rxXN1wH13wtIlO4obUbPt8wEb58Slrr42kXBoLLLDrXE3Af9LlabtNjJ0z-a5reSUZmOdVYiJl9sEF4YwG9177mwUSJz7VLQAI1hKN1pg6Ox1sJj2fBwdBqjIiRXsw-KBwoMQx9JmuMk9wCr5-gI5f8I-9Vqizb8Lf5ZJ4SMf35Wy3R8dwQeXXau_7t5zDe9wO9wnc9RfOMCuDCc359-oLDFmtrahgrMjmDx5YrQHol6jC43S_7gQ_2IPLE_TlqiQ","scope":"ovirt-app-api
ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search
ovirt-ext=token-info:validate
ovirt-ext=token:password-access","exp":"9223372036854775807","token_type":"bearer"}%
Now lets use access token to authenticate and fetch hosts:
$ curl -k -H "Accept: application/json" -H "Authorization: Bearer
eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJEdS10MlVQd0JaZ0gtRU1JUkRTRHFxNFZIOUhZbnc4Nkk5QUlGOERxZ1l3In0.eyJleHAiOjE2NTcyMTY5MzAsImlhdCI6MTY1NzE5OTY1MCwianRpIjoiNTAwOWVkMmItMjc3ZS00YjVmLWEwOTItMjI4MDhkMWFhMWJjIiwiaXNzIjoiaHR0cHM6Ly9kZXYzLmRvbS9vdmlydC1lbmdpbmUtYXV0aC9yZWFsbXMvb3ZpcnQtaW50ZXJuYWwiLCJhdWQiOiJhY2NvdW50Iiwic3ViIjoiMjczMTlkODMtYjdkYy00MzU2LTllMmQtYjJmNzg5NWI3YjczIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoib3ZpcnQtZW5naW5lLWludGVybmFsIiwic2Vzc2lvbl9zdGF0ZSI6IjNmODM5NjY2LTQyNzUtNDRhNC1hNDRhLTM3Njc5MzgxNDRiOCIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiaHR0cHM6Ly9kZXYzLmRvbSJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiZGVmYXVsdC1yb2xlcy1vdmlydC1pbnRlcm5hbCIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im92aXJ0LWV4dD10b2tlbi1pbmZvOnB1YmxpYy1hdXRoei1zZWFyY2ggb3ZpcnQtYXBwLWFwaSBvdmlydC1leHQ9dG9rZW46cGFzc3dvcmQtYWNjZXNzIG92aXJ0LWV4dD10b2tlbi1pbmZvOmF1dGh6LXNlYXJjaCBvdmlydC1leHQ9dG9rZW4taW5mbzp2YWxpZGF0ZSBlbWFpbCBwcm9maWxlIiwic2lkIjoiM2Y4Mzk2NjYtNDI3NS00NGE0LWE0NGEtMzc2NzkzODE0NGI4IiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJncm91cHMiOlsiL292aXJ0LWFkbWluaXN0cmF0b3IiXSwicHJlZmVycmVkX3VzZXJuYW1lIjoiYWRtaW5Ab3ZpcnQiLCJlbWFpbCI6ImFkbWluQGxvY2FsaG9zdCJ9.Ov2IJ-ghtXSB6eb7osWZgT_yeb4prBgVzUU9vAY_VMoDr-ie5bMYBUyinYvNHWpBbYaFGNjg6bC7PHz3-s5H1rxXN1wH13wtIlO4obUbPt8wEb58Slrr42kXBoLLLDrXE3Af9LlabtNjJ0z-a5reSUZmOdVYiJl9sEF4YwG9177mwUSJz7VLQAI1hKN1pg6Ox1sJj2fBwdBqjIiRXsw-KBwoMQx9JmuMk9wCr5-gI5f8I-9Vqizb8Lf5ZJ4SMf35Wy3R8dwQeXXau_7t5zDe9wO9wnc9RfOMCuDCc359-oLDFmtrahgrMjmDx5YrQHol6jC43S_7gQ_2IPLE_TlqiQ"
'https://ENGINE_FQDN/ovirt-engine/api/hosts'
In order to change default Keycloak configuration or set up any additional
identity providers you need to access the Keycloak Administration Panel (
https://YOUR_ENGINE_FQDN/ovirt-engine-auth/admin).
By default, on a fresh installation, you can login using 'admin' and the
password provided during engine-setup.
Keycloak allows to easily use all the features that were previously
supported by oVirt in-house authentication implementation plus many more
almost for free - multi factor authentication, 3rd party identity
providers (ie. github, google, facebook etc.) just to name a few.
For more information please see the Keycloak's documentation [4].
[1] https://www.ovirt.org/release/4.5.1/#keycloak-sso-setup-for-ovirt-engine
[2]
https://github.com/oVirt/ovirt-engine-keycloak/blob/master/keycloak_usage.md
[3] https://bugzilla.redhat.com/show_bug.cgi?id=2101474
[4] https://www.keycloak.org/archive/documentation-15.0.html
Please, let us know if you have any questions/concerns.
Last, but not least, any contributions or bug reports are more than
welcomed!
thanks!
Artur
--
Artur Socha
Senior Software Engineer, RHV
Red Hat
2 years, 5 months
Q: oVirt 4.4.7 -> 4.4.10 Upgrade Woes
by Andrei Verovski
Hi,
I’m running dedicated oVirt Engine (separate PC, not hosted engine) and trying to upgrade 4.4.7 -> 4.4.10.
Quite cumbersome process, since CentOS 8.x was switched to Stream.
OK, then, DNF upgrade went successfully, and after sudo dnf install https://resources.ovirt.org/pub/yum-repo/ovirt-release44.rpm and engine-upgrade I run setup.
Here are the problems.
For whatever reason pki keys get auto-deleted during upgrade:
[WARNING] Unable to ensure permissions on /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf
[ ERROR ] Failed to execute stage 'Misc configuration': [Errno 2] No such file or directory: '/etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass'
[ ERROR ] Failed to execute stage 'Misc configuration': [Errno 2] No such file or directory: '/etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass’
Restored these keys from backup (entire “keys” directory).
Still no luck.
[WARNING] Unable to ensure permissions on /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf
[ INFO ] Upgrading CA
[ INFO ] Renewing engine certificate
[ ERROR ] Failed to execute stage 'Misc configuration': Command '/usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh' failed to execute
No matter if I choose renew keys or not, this failure still present.
Logs attached (its quite big).
How to solve this problem?
Thanks in advance.
2 years, 5 months
Reminder: oVirt Survey Summer 2022
by Sandro Bonazzola
Reminder: oVirt Survey Summer 2022
As we continue to develop oVirt 4.5, the oVirt community would value
insights on your experience with the oVirt project.
If you haven't already done, please help us to hit the mark by completing
this short survey: https://forms.gle/2LxoDKb7njEQwr4f7 .
The survey will close in two weeks on July 29th 2022.
Please note the answers to this survey will be publicly accessible.
This survey is under oVirt Privacy Policy available at
https://ovirt.org/privacy-policy.html .
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>
*Red Hat respects your work life balance. Therefore there is no need to
answer this email out of your office hours.*
2 years, 5 months
Upgrade from 4.2 directly to 4.5
by Vinícius Ferrão
Hello, I would like to know if I can do an oVirt upgrade directly from 4.2 to 4.5.
I don’t have a free host to upgrade the oVirt Node, so I was hoping that would be possible to fire up a new engine (with restore-backup) on the old oVirt Nodes and later on upgrade the hosts.
Is this possible?
Thank you.
2 years, 5 months
