Single SPM Host orphaned from ovirt-engine, cannot confirm host rebooted or maintenance host.
by Terry
We have number of clusters connected to ovirt-engine. Some of these are single host clusters (running ovirt-release43-4.3.5.2-1 on CentOS7) with local storage. Recently, ovirt-engine started reporting one of these hosts NonResponsive, VMs were still running on the host but ovirt seems unable to communicate with it, testing shows no issues connecting engine -> host:vdsm and likewise the host can communicate with the engine on port 80 and 443.
The host in question cannot be managed via IPMI for power management but we are able to perform an SSH reboot via the engine interface. We opted to login to the running virtual machines, shut them down and issue the SSH reboot from the engine. The server changes to rebooting status for some time and then reports NonResponsive state.
We are unable to maintenance the host or confirm host has been rebooted manually as we are presented with the following
"Error while executing action: Cannot perform confirm 'Host has been rebooted'. Another power management action is already in progress."
The VDSM logs on the host in question are continually showing:
2020-04-16 08:23:51,478+0000 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:711)
2020-04-16 08:23:52,332+0000 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo error=Unknown pool id, pool not connected: (u'6baea5dc-b049-47c2-a94f-5229c37c62d0',) from=::ffff:10.10.1.252,33680, task_id=420249a4-55c0-436d-92c7-ea1286a0e287 (api:52)
2020-04-16 08:23:52,332+0000 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='420249a4-55c0-436d-92c7-ea1286a0e287') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run
return fn(*args, **kargs)
File "<string>", line 2, in getStoragePoolInfo
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2550, in getStoragePoolInfo
pool = self.getPool(spUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 351, in getPool
raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: (u'6baea5dc-b049-47c2-a94f-5229c37c62d0',)
2020-04-16 08:23:52,333+0000 INFO (jsonrpc/7) [storage.TaskManager.Task] (Task='420249a4-55c0-436d-92c7-ea1286a0e287') aborting: Task is aborted: "Unknown pool id, pool not connected: (u'6baea5dc-b049-47c2-a94f-5229c37c62d0',)" - code 309 (task:1181)
During this period, the following is observed in the engine logs:
2020-04-16 08:23:52,307Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM compute01.ovirt.local command SpmStatusVDS failed: Message timeout which can be caused by communication issues
2020-04-16 08:23:52,307Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [] Command 'SpmStatusVDSCommand(HostName = compute01.ovirt.local, SpmStatusVDSCommandParameters:{hostId='67dc53da-d5ee-461e-87de-2ca6dd78637f', storagePoolId='6baea5dc-b049-47c2-a94f-5229c37c62d0'})' execution failed: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues
2020-04-16 08:23:52,346Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [] Failed in 'GetStoragePoolInfoVDS' method
2020-04-16 08:23:52,355Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command GetStoragePoolInfoVDS failed: Unknown pool id, pool not connected: (u'6baea5dc-b049-47c2-a94f-5229c37c62d0',)
2020-04-16 08:23:52,356Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: Failed to GetStoragePoolInfoVDS, error = Unknown pool id, pool not connected: (u'6baea5dc-b049-47c2-a94f-5229c37c62d0',), code = 309
The metadata file for the local storage domain looks fine?
ALIGNMENT=1048576
BLOCK_SIZE=512
CLASS=Data
DESCRIPTION=compute01_local_storage
IOOPTIMEOUTSEC=10
LEASERETRIES=3
LEASETIMESEC=60
LOCKPOLICY=
LOCKRENEWALINTERVALSEC=5
MASTER_VERSION=1
POOL_DESCRIPTION=compute01_local
POOL_DOMAINS=1cc26dea-688c-40cc-bda6-38b00054001e:Active
POOL_SPM_ID=-1
POOL_SPM_LVER=-1
POOL_UUID=6baea5dc-b049-47c2-a94f-5229c37c62d0
REMOTE_PATH=/mnt/ovirt_datastore
ROLE=Master
SDUUID=1cc26dea-688c-40cc-bda6-38b00054001e
TYPE=LOCALFS
VERSION=5
_SHA_CKSUM=24c85256b889d0b3384e7975c660f4a5cbb58d33
I would assume this has happened because ovirt was unable to power cycle the machine and now can't confirm the SPM state? Normally in a case like this we would confirm the host has been manually rebooted but we're unable to do that.
How can I clear the power management action that ovirt-engine thinks is in progress?
4 years
disk snapshot in illegal status
by David David
hello
after removing "auto generated" snapshot, the vm snapshot became to illegal
status and in the "snapshot" tab also marked in illegal status
now my vm is not booting
but "vdsm-client Volume getInfo" shows the snapshot status is legal
vdsm-client Volume getInfo \
storagepoolID=00000000-0000-0000-0000-000000000000 \
storagedomainID=d17aa967-f7f7-4d9f-98e4-7022854da4f5 \
imageID=1844873e-45ef-4a2b-a991-2df6e92a3e2d \
volumeID=f447cd4f-9ab3-489d-8c2a-b2e69456496f
{
"status": "OK",
"lease": {
"path": "/dev/d17aa967-f7f7-4d9f-98e4-7022854da4f5/leases",
"owners": [],
"version": null,
"offset": 132120576
},
"domain": "d17aa967-f7f7-4d9f-98e4-7022854da4f5",
"capacity": "21474836480",
"voltype": "LEAF",
"description": "None",
"parent": "00000000-0000-0000-0000-000000000000",
"format": "RAW",
"generation": 1,
"image": "1844873e-45ef-4a2b-a991-2df6e92a3e2d",
"uuid": "f447cd4f-9ab3-489d-8c2a-b2e69456496f",
"disktype": "DATA",
"legality": "LEGAL",
"mtime": "0",
"apparentsize": "21474836480",
"truesize": "21474836480",
"type": "PREALLOCATED",
"children": [],
"pool": "",
"ctime": "1530472945"
}
how to fix the problem?
4 years
ovirt-engine unresponsive - how to rescue?
by Shareef Jalloq
So my engine has gone down and I can't ssh into it either. If I try to log
into the web-ui of the node it is running on, I get redirected because the
node can't reach the engine.
What are my next steps?
Shareef.
4 years
yum mirrors 403 forbidden
by Shareef Jalloq
I seem to be taking 2 steps back and 1 forward at the moment. What could I
have done on a node to start getting 403 forbidden errors when using yum?
[root@ovirt-node-00 ~]# yum repolist
Loaded plugins: enabled_repos_upload, fastestmirror, imgbased-persist,
package_upload, product-id, search-disabled-repos, vdsmupgrade, versionlock
Loading mirror speeds from cached hostfile
ovirt-4.3-epel/x86_64/metalink
| 13 kB 00:00:00
* ovirt-4.3-epel: mirror.de.leaseweb.net
http://mirror.centos.org/centos/7/sclo/x86_64/rh/repodata/repomd.xml:
[Errno 14] HTTP Error 403 - Forbidden
Trying other mirror.
To address this issue please refer to the below wiki article
https://wiki.centos.org/yum-errors
If above article doesn't help to resolve this issue please use
https://bugs.centos.org/.
http://mirror.centos.org/centos/7/sclo/x86_64/rh/repodata/repomd.xml:
[Errno 14] HTTP Error 403 - Forbidden
Trying other mirror.
ovirt-4.3/7
| 3.0 kB 00:00:00
http://mirror.centos.org/centos/7/storage/x86_64/gluster-6/repodata/repom...:
[Errno 14] HTTP Error 403 - Forbidden
Trying other mirror.
http://mirror.centos.org/centos/7/opstools/x86_64/repodata/repomd.xml:
[Errno 14] HTTP Error 403 - Forbidden
Trying other mirror.
http://mirror.centos.org/centos/7/virt/x86_64/ovirt-4.3/repodata/repomd.xml:
[Errno 14] HTTP Error 403 - Forbidden
Trying other mirror.
http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/repodata/repomd.xml:
[Errno 14] HTTP Error 403 - Forbidden
Trying other mirror.
http://fedorapeople.org/groups/virt/virtio-win/repo/latest/repodata/repom...:
[Errno 14] HTTP Error 403 - Forbidden
Trying other mirror.
sac-gluster-ansible/x86_64
| 3.3 kB 00:00:00
repo id
repo name
status
centos-sclo-rh-release/x86_64
CentOS-7 - SCLo rh
0
ovirt-4.3/7
Latest oVirt 4.3 Release
2,831
ovirt-4.3-centos-gluster6/x86_64
CentOS-7 - Gluster 6
0
ovirt-4.3-centos-opstools/x86_64
CentOS-7 - OpsTools - release
0
ovirt-4.3-centos-ovirt43/x86_64
CentOS-7 - oVirt 4.3
0
ovirt-4.3-centos-qemu-ev/x86_64
CentOS-7 - QEMU EV
0
ovirt-4.3-epel/x86_64
Extra Packages for Enterprise Linux 7 - x86_64
13,244
ovirt-4.3-virtio-win-latest
virtio-win builds roughly matching what will be shipped in upcoming RHEL
0
sac-gluster-ansible/x86_64
Copr repo for gluster-ansible owned by sac
18
repolist: 16,093
Uploading Enabled Repositories Report
Cannot upload enabled repos report, is this client registered?
4 years
cloud-init ovirt
by PunkGeek
Hello,
How can I use ovirt tools that configure cloud-init vms by python?
https://red.ht/2K1gBkn
Actually, I want just to use this tool in my python libvirt script.
Best regards
4 years
Can't migrate over migration network
by briandumont＠gmail.com
Hello,
I am trying to configure a dedicated migration network and am unable to get it to work. It is failing with a no route to host error in the cockpit logs, specifically a libvirtd error virNetSocketNewConnectTCP:591 : unable to connect to server at '192.168.1.50:49152': No route to host. Trying to migrate from Host 1 to Host 2. 192.168.1.50 is the ip address on the source host for the migration.
If I try to migrate in the other direction I get the same error with tthat hosts migration network ip.
My migration network is setup on a different VLAN than my ovirtmgmt network.
Here is my configuration:
- 2 Hosts
- Host 1
- nic 1 - ovirtmgmt - 192.168.1.24/24 - VLAN 1
- nic 2 - migration network - 192.168.1.50/24 VLAN 25
- Host 2
- nic 1 - ovirtmgmnt - 192.168.1.24/24 - VLAN 1
- nic 2 - migration network - 192.168.1.51/24 - VLAN 25
Migration network is setup as follows:
- Assign - Yes
- Required - Yes
- VM Network - No (tried it with VM Nework = Yes with same results)
- Migration Network - Yes
I think I've got something fundamentally wrong with my host networking but I can't figure it out.
Appreciate the help.
4 years
NUMA Pinning bug with Hugepages?
by Alan G
Hi,
I seem to have found an issue when trying to setup a high performance VM utilising hugepages and NUMA pinning.
The VM is configured for 32GB RAM and uses hugepages of size 1G.
The host has two NUMA nodes each having 64GB RAM (for 128GB total system RAM).
No other VMs are running or otherwise pinned to the host.
The VM is pinned to node 0. And the required hugepages are allocated with
echo 32 > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
I then attempt to start the VM and get a pop-up error saying "cannot accommodate memory of VM's pinned virtual NUMA nodes within host's physical NUMA nodes".
However, if I remove the hugepages from node 0
echo 0 > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
Start the VM and then immediately re-create the hugepqages then everything works as expected.
It seems to me that oVirt is considering the hugepages as allocated memory even if they are not in use.
Is this correct?
Thanks,
Alan
4 years
upgrade VM cluster compatibility version
by Bill James
I'm trying to upgrade my cluster to 4.3. Currently its running
4.2.8.2-1.el7.
But the cluster compatibility version is 3.6.
I try to change that and it says I need to edit the VM.
https://www.ovirt.org/documentation/upgrade-guide/chap-Post-Upgrade_Tasks...
says:
*Important:* An error message may warn that some virtual machines and
templates are incorrectly configured. To fix this error, edit each virtual
machine manually. The *Edit Virtual Machine* window provides additional
validations and warnings that show what to correct. Sometimes the issue is
automatically corrected and the virtual machine’s configuration just needs
to be saved again. After editing each virtual machine, you will be able to
change the cluster compatibility version.
I edit the VM and there are no errors or warnings. I see no way to change
the cluster compatibility level on a VM.
How do I change the Custom Compatibility Version??
--
This email, its contents and attachments contain information from J2
Global, Inc. and/or its affiliates which may be privileged, confidential or
otherwise protected from disclosure. The information is intended to be for
the addressee(s) only. If you are not an addressee, any disclosure, copy,
distribution or use of the contents of this message is prohibited. If you
have received this email in error, please notify the sender by reply email
and delete the original message and any copies.
4 years
Get OAuth token locally without password?
by Chris Adams
I'm working on a cron job to run on the engine to do some tasks via the
API. Is there a way I can get an API OAuth token created without
actually storing a password?
For example, some script that can be run to directly create a session in
the database and return the token. Obviously this _can_ be done, it's a
matter of knowing the right bits to do (so basically wondering if that
has already been written).
I only have internal users, so no external authentication store
available.
--
Chris Adams <cma(a)cmadams.net>
4 years
Migrate engine to another storage domain?
by Simon Nylund
Is there a successful way of migrating an up and running engine to another storage domain?
I've been reading through the archive and tested a few various ways without any success. The only way i found working without errors was deploying a new engine attached to the new storage domain, and import the old data domain for the vms, but then you lose all the settings for the previous engine (ofc). And have to redo all of that.
4 years
