Engine storage Domain path change
by michael＠wiswm.com
Hello
I installed oVirt on 3 servers(hv1,hv2,hv3) with Self Hosted Engine couple years ago. Gluster is used as storage for VMs. The engine has its own storage volume.
Then I added 3 more servers(hv4,hv5,hv6).
Now I would like to replace the first 3 servers. I added 3 more servers(hv7,hv8,hv9). I created new gluster volumes on hv7-hv9, moved disks from old volumes.
Now the question is how to migrate engine storage? I decided to do it on Gluster level by replacing each of the brick from the old servers(hv1-hv3) with new ones (hv7-hv9). I successfully replaced bricks from hv1 and hv2 to hv7 and hv8. InoVirt engine storage Domain is created with the path hv3:/engine. I am afraid to replace hv3 brick with hv9, to not break the HostedEngine VM. To change storage Domain path I need to move it to Maintenance, but it will unmount it from all the hosts and HostedEngine will stop working. Is there any otherway to change a storage Domain path?
I already changed storage value in /etc/ovirt-hosted-engine/hosted-engine.conf. I did something like this:
# on hv7
hosted-engine --vm-shutdown
# on all hosts
systemctl stop ovirt-ha-agent
systemctl stop ovirt-ha-broker
hosted-engine --disconnect-storage
sed -i 's/hv3/hv7/g' /etc/ovirt-hosted-engine/hosted-engine.conf
hosted-engine --connect-storage
systemctl restart ovirt-ha-broker
systemctl status ovirt-ha-broker
systemctl restart ovirt-ha-agent
# on hv7
hosted-engine --vm-start
and now the Hosted Engine is up and running and Gluster engine volume is mounted like hv7:/engine
So the main question remains - how to change the engine storage Domain path?
1 year, 9 months
Stuck in Manager upgrade. Can't set Cluster to maintenance mode.
by johannes.lutz＠diamontech.de
Hi Folks,
like many others, the ovirt hosted engine certificates expired on my installation. We tried to follow this knowledge base article: https://access.redhat.com/solutions/6865861
I set the host which runs the hosted engine via the "hosted-engine --set-maintenance --mode=global" command into the global maintenance mode.
Then i try to execute the "engine-setup --offline" command. There we anwser all questions and the script recognizes the expired certificates. But when we try to execute the last step it aborts with following error message.
Output of engine-setup --offline:
[WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/apache.p12'
Perhaps it was changed since last Setup.
Error was:
Mac verify error: invalid password?
One or more of the certificates should be renewed, because they expire soon, or include an invalid expiry date, or they were created with validity period longer than 398 days, or do not include the subjectAltName extension, which can cause them to be rejected by recent browsers and up to date hosts.
See https://www.ovirt.org/develop/release-management/features/infra/pki-renew/ for more details.
Renew certificates? (Yes, No) [No]: Yes
--== APACHE CONFIGURATION ==--
--== SYSTEM CONFIGURATION ==--
--== MISC CONFIGURATION ==--
--== END OF CONFIGURATION ==--
[ INFO ] Stage: Setup validation
During execution engine service will be stopped (OK, Cancel) [OK]: Ok
[ ERROR ] It seems that you are running your engine inside of the hosted-engine VM and are not in "Global Maintenance" mode.
In that case you should put the system into the "Global Maintenance" mode before running engine-setup, or the hosted-engine HA agent might kill the machine, which might corrupt your data.
[ ERROR ] Failed to execute stage 'Setup validation': Hosted Engine setup detected, but Global Maintenance is not set.
[ INFO ] Stage: Clean up
Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20220701205812-yu1osl.log
[ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20220701205843-setup.conf'
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ ERROR ] Execution of setup failed
Any ideas how to get the hosted engine into global maintenance mode?
Thanks for your help in advance!
Best Regard
J. Lutz
1 year, 9 months
Not able to login as admin after successfull deployment of hosted engine (OVirt 4.5.1)
by Ralf Schenk
Hello List,
I successfully deployed a fresh hosted-engine, but I'm not able to login
to Administration-Portal. I'm perferctly sure about the password I had
to type multiple times....
I'm running ovirt-node-ng-4.5.1-0.20220622.0 and deployed engine via
cli-based ovirt-hosted-engine-setup.
Neither "admin" nor "admin@internal" are working (A profile cannot be
choosen as in earlier versions).
I can login to the monitoring part (grafana !) and also Cockpit but not
Administration-Portal nor VM-Portal.
I can ssh into the engine and lookup the user-database which has the user.
root@engine02 ~]# ovirt-aaa-jdbc-tool query --what=user
Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false
-- User admin(2be16cf0-5eb7-4b0e-923e-7bdc7bc2aa6f) --
Namespace: *
Name: admin
ID: 2be16cf0-5eb7-4b0e-923e-7bdc7bc2aa6f
Display Name:
Email: root@localhost
First Name: admin
Last Name:
Department:
Title:
Description:
Account Disabled: false
Account Locked: false
Account Unlocked At: 1970-01-01 00:00:00Z
Account Valid From: 2022-07-15 18:23:47Z
Account Valid To: 2222-07-15 18:23:47Z
Account Without Password: false
Last successful Login At: 1970-01-01 00:00:00Z
Last unsuccessful Login At: 1970-01-01 00:00:00Z
Password Valid To: 2222-05-28 18:23:49Z
However no groups by default ???
[root@engine02 ~]# ovirt-aaa-jdbc-tool query --what=group
Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false
Any solution ? I don't want to repeat the hosted-engine deployment a
fourth time after I mastered all problems with NFS permissions, GUI
deployment not accepting my Bond which is perfectly ok called "bond0"
etc....
Bye
--
Databay AG Logo
*Ralf Schenk
*
fon: 02405 / 40 83 70
mail: rs(a)databay.de
web: www.databay.de <https://www.databay.de>
Databay AG
Jens-Otto-Krag-Str. 11
52146 Würselen
Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202
Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.Kfm.
Philipp Hermanns
Aufsichtsratsvorsitzender: Dr. Jan Scholzen
1 year, 9 months
Q: oVirt 4.4.10 and vdsm-jsonrpc-java (Internal server error 500 after engine-setup)
by Andrei Verovski
Hi,
Finally I managed to migrate 4.4.7 to fresh installation of 4.4.10.
However, after successful engine-setup I’ve got 500 - Internal Server Error
I found this:
https://bugzilla.redhat.com/show_bug.cgi?id=1918022
Bug 1918022 - oVirt Manager is not loading after engine-setup
Article suggested to downgrade vdsm-jsonrpc-java to 1.5.x.
However, this is not possible:
dnf --showduplicates list vdsm-jsonrpc-java
dnf install vdsm-jsonrpc-java-1.5.7-1.el8
Last metadata expiration check: 0:39:52 ago on Fri 15 Jul 2022 02:32:36 PM EEST.
Error:
Problem: problem with installed package ovirt-engine-backend-4.4.10.7-1.el8.noarch
- package ovirt-engine-backend-4.4.10.7-1.el8.noarch requires vdsm-jsonrpc-java >= 1.6.0, but none of the providers can be installed
- cannot install both vdsm-jsonrpc-java-1.5.7-1.el8.noarch and vdsm-jsonrpc-java-1.6.0-1.el8.noarch
- cannot install both vdsm-jsonrpc-java-1.6.0-1.el8.noarch and vdsm-jsonrpc-java-1.5.7-1.el8.noarch
How to fix this?
Thanks in advance.
***************** SERVER LOG *********************
2022-07-15 14:45:44,969+03 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "engine.ear")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.Backend.START" => "java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
Caused by: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
Caused by: javax.ejb.EJBException: org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke protected void org.ovirt.engine.core.bll.TagsDirector.init() on org.ovirt.engine.core.bll.TagsDirector@648487d3
Caused by: org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke protected void org.ovirt.engine.core.bll.TagsDirector.init() on org.ovirt.engine.core.bll.TagsDirector@648487d3
Caused by: java.lang.reflect.InvocationTargetException
Caused by: org.springframework.dao.InvalidDataAccessApiUsageException: Unable to determine the correct call signature - no procedure/function/signature for 'gettagsbyparent_id'"}}
2022-07-15 14:45:44,981+03 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "restapi.war" (runtime-name : "restapi.war")
2022-07-15 14:45:44,982+03 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "engine.ear" (runtime-name : "engine.ear")
2022-07-15 14:45:44,982+03 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "apidoc.war" (runtime-name : "apidoc.war")
2022-07-15 14:45:44,982+03 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "ovirt-web-ui.war" (runtime-name : "ovirt-web-ui.war")
2022-07-15 14:45:45,015+03 INFO [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
WFLYCTL0186: Services which failed to start: service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
WFLYCTL0448: 2 additional services are down due to their dependencies being missing or failed
1 year, 9 months
VM Migration Failed
by KSNull Zero
Running oVirt 4.4.5
VM cannot migrate between hosts.
vdsm.log contains the following error:
libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tls://ovhost01.local/system: authentication failed: Failed to verify peer's certificate
Certificates on hosts was renewed some time ago. How this issue can be fixed ?
Thank you.
1 year, 9 months
Seeking best performance on oVirt cluster
by David Johnson
Good morning all,
I am trying to get the best performance out of my cluster possible,
Here are the details of what I have now:
Ovirt version: 4.4.10.7-1.el8
Bare metal for the ovirt engine
two hosts
TrueNAS cluster storage
1 NFS share
3 vdevs, 6 drives in raidz2 in each vdev
2 nvme drives for silog
Storage network is 10 GBit all static IP addresses
Tonight, I built a new VM from a template. It had 5 attached disks
totalling 100 GB. It took 30 minutes to deploy the new VM from the
template.
Global utilization was 9%.
The SPM has 50% of its memory free and never showed more than 12% network
utilization
62 out of 65 TB are available on the newly created NFS backing store (no
fragmentation). The TureNAS system is probably overprovisioned for our use.
There were peak throughputs of up to 4 GBytes/second (on a 10 GBit
network), but overall throughput on the NAS and the network were low.
ARC hits were 95 to 100%
L2 hits were 0 to 70%
Here's the NFS usage stats:
[image: image.png]
I believe the first peak is where the silog buffered the initial burst of
instructions, followed by sustained IO as the VM volumes were built in
parallel, and then finally tapering off to the one 50 GB volume that took
40 minutes to copy.
The indications of the NFS stats graph are that the network performance is
just fine.
Here are the disk IO stats covering the same time frame, plus a bit before
to show a spike IO:
[image: image.png]
The spike at 2250 (10 minutes before I started building my VM) shows that
the spinners actually hit write speed of almost 20 MBytes per second
briefly, then settled in at a sustained 3 to 4 MBytes per second. The
silog absorbs several spikes, but remains mostly idle, with activity
measured in kilobytes per second.
The HGST HUS726060AL5210 drives boast a spike throughput of 12 GB/S, and
sustained throughput of 227 Mbps.
------
Now to the questions:
1. Am I asking the on the right list? Does this look like something where
tuning ovirt might make a difference, or is this more likely a
configuration issue with my storage appliances?
2. Am I expecting too much? Is this well within the bounds of acceptable
(expected) performance?
3. How would I go about identifying the bottleneck, should I need to dig
deeper?
Thanks,
David Johnson
1 year, 9 months
unable to create iso domain
by Moritz Baumann
Hi
I have removed the iso domain of an existing data center, and now I am
unable to create a new iso domain
/var/log/ovirt-engine/engine.log shows:
2022-07-14 08:04:40,684+02 INFO
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand]
(default task-34) [8db814e3-43ab-4921-ad35-2b3acd51c385] Lock Acquired
to object
'EngineLock:{exclusiveLocks='[ovirt.storage.inf.ethz.ch:/export/ovirt/iso=STORAGE_CONNECTION]',
sharedLocks=''}'
2022-07-14 08:04:40,689+02 WARN
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand]
(default task-34) [8db814e3-43ab-4921-ad35-2b3acd51c385] Validation of
action 'AddStorageServerConnection' failed for user
xxx@ethz.ch(a)ethz.ch-authz. Reasons:
VAR__ACTION__ADD,VAR__TYPE__STORAGE__CONNECTION,$connectionId
c39c64ef-fb8b-4e87-9803-420c7fb2dd4a,$storageDomainName
,ACTION_TYPE_FAILED_STORAGE_CONNECTION_ALREADY_EXISTS
2022-07-14 08:04:40,690+02 INFO
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand]
(default task-34) [8db814e3-43ab-4921-ad35-2b3acd51c385] Lock freed to
object
'EngineLock:{exclusiveLocks='[ovirt.scratch.inf.ethz.ch:/export/ovirt/iso=STORAGE_CONNECTION]',
sharedLocks=''}'
2022-07-14 08:04:40,756+02 INFO
[org.ovirt.engine.core.bll.storage.connection.DisconnectStorageServerConnectionCommand]
(default task-34) [4148e0fd-58ae-4375-8dc8-a08f47402ed6] Running
command: DisconnectStorageServerConnectionCommand internal: false.
Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type:
SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2022-07-14 08:04:40,756+02 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand]
(default task-34) [4148e0fd-58ae-4375-8dc8-a08f47402ed6] START,
DisconnectStorageServerVDSCommand(HostName = ovirt-node01,
StorageServerConnectionManagementVDSParameters:{hostId='d942c8fe-9a0c-4761-9be2-2f88b622070b',
storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS',
connectionList='[StorageServerConnections:{id='null',
connection='ovirt.storage.inf.ethz.ch:/export/ovirt/iso', iqn='null',
vfsType='null', mountOptions='null', nfsVersion='null',
nfsRetrans='null', nfsTimeo='null', iface='null',
netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 3043bbfd
2022-07-14 08:04:43,017+02 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand]
(default task-34) [4148e0fd-58ae-4375-8dc8-a08f47402ed6] FINISH,
DisconnectStorageServerVDSCommand, return:
{00000000-0000-0000-0000-000000000000=100}, log id: 3043bbfd
[root@ovirt-engine ovirt-engine]# showmount -e ovirt.storage.inf.ethz.ch
| grep ovirt
Export list for ovirt.scratch.inf.ethz.ch:
/export/ovirt/export @ovirt-storage
/export/ovirt/data @ovirt-storage
/export/ovirt/iso @ovirt-storage
the other two domains still work just fine and the netgroup contains all
ovirt-nodes.
storage-node1[0]:/export/ovirt/iso# ls -la
total 2
drwx------. 2 vdsm kvm 2 Jul 14 07:58 .
drwxr-xr-x. 5 root root 5 Aug 19 2020 ..
storage-node1[0]:/export/ovirt/iso# df .
Filesystem 1K-blocks Used Available Use% Mounted on
fs1/ovirt/iso 524288000 256 524287744 1% /export/ovirt/iso
storage-node1[0]:/export/ovirt/iso#
storage-node1[0]:/export/ovirt/iso# exportfs -v | grep ovirt/ -A1
/export/ovirt/iso
@ovirt-storage(sync,wdelay,hide,no_subtree_check,fsid=215812,sec=sys:krb5:krb5i:krb5p,rw,secure,root_squash,no_all_squash)
/export/ovirt/data
@ovirt-storage(sync,wdelay,hide,no_subtree_check,fsid=215811,sec=sys:krb5:krb5i:krb5p,rw,secure,root_squash,no_all_squash)
--
/export/ovirt/export
@ovirt-storage(sync,wdelay,hide,no_subtree_check,fsid=215813,sec=sys:krb5:krb5i:krb5p,rw,secure,root_squash,no_all_squash)
It appears that there is stille some reference to an iso domain
(c39c64ef-fb8b-4e87-9803-420c7fb2dd4a ??) in the database. How can I get
rid of it ?
Best
Moritz
1 year, 9 months
4.3.10 cannot start VMs because of an error with USB
by Pascal D
I am still running 4.3.10 and suddenly I get this error everytime I restart a VM, any Vm on any host (I have 13 hosts in 2 different clusters)
VM FLEETGEN-PCC-001 is down with error. Exit message: XML error: there is no hub at port 1 in USB address bus: 0 port: 1.1.
Any idea what could be different. Template hasn't changed
1 year, 9 months
Import KVM VMs on individual iSCSI luns
by spierce＠cts1.com
Greetings,
Is it possible with oVirt to import existing VMs where the underlying storage is on raw iSCSI luns and to keep them on those luns?
The historical scenario is that we have Virtual farms in multiple sites managed by an ancient Orchestration tool that does not support modern OS's as the hypervisor.
- In each site, there are clusters of hypervisors/Hosts that have visibility to the same iSCSI luns.
- Each VM has it's own set of iscsi luns that are totally dedicated to that VM
- Each VM is using LVM to manage the disk
- Each Host has LVM filtering configured to NOT manage the VM's iscsi luns
- The VMs can be live migrated from any Hypervisor within the cluster to any other Hypervisor in that same cluster
We are attempting to bring this existing environment into oVirt without replacing the storage model.
Is there any documentation that will serve as a guide for this scenario?
In a lab environment, we have successfully
- Added 2 hypervisors (hosts) and oVirt can see their VMs as external-ovtest1 and external-ovtest2
- Removed the LVM filtering on the hosts
- Created a storage domain that is able to see the iscsi luns, but we have not yet done the 'add' of each lun
Is it possible to import these luns as raw block devices without LVM being layered on top of them?
Is it required to actually import the luns into a storage domain, or can the VM's still be imported if all luns are visible on all hosts in the cluster?
In the grand scheme of things, are we trying to do something that is not possible with oVirt?
If it is possible, we would greatly appreciate tips, pointers, links to docs etc that will help us migrate this environment to oVirt.
Thanks in Advance
- S
1 year, 9 months
Q: Instaling Ovirt Engine 4.4.10 on Clean CentOS 8 Stream
by Andrei Verovski
Hi,
Since I run into a lot of problems upgrading 4.4.7 to 4.4.10, is it OK to install clean
CentOS-Stream-8-x86_64-20220712-dvd1.iso
and restore from backup?
4.4.10 release long before Stream 20220712, may I run again into similar problems, or I need to use earlier snapshot of Stream 8 ?
I use dedicated engine PC, not hosted engine.
Thanks in advance for any suggestion(s)
Andrei
1 year, 9 months
