Lots of storage.MailBox.SpmMailMonitor
by Fabrice Bacchella
My vdsm log files are huge:
-rw-r--r-- 1 vdsm kvm 1.8G Nov 22 11:32 vdsm.log
And this is juste half an hour of logs:
$ head -1 vdsm.log
2018-11-22 11:01:12,132+0100 ERROR (mailbox-spm) [storage.MailBox.SpmMailMonitor] mailbox 2 checksum failed, not clearing mailbox, clearing new mail (data='...lots of data', expected='\xa4\x06\x08\x00') (mailbox:612)
I just upgraded vdsm:
$ rpm -qi vdsm
Name : vdsm
Version : 4.20.43
2 years, 11 months
Using IBM Spectrum Scale as POSIX compliant filesystem under oVirt 4.4 results in oVirt host reboot.
by Andi Nør Christiansen
Hi,
Does anyone know how to mount a spectrum scale filesystem as a storage domain using the POSIX compliant storage option.
I've tried to follow an earlier post about this - https://lists.ovirt.org/archives/list/users@ovirt.org/thread/LLXLSI4ZQFU3...
specifically the options provided by jtorres:
Domain Function: Data
Name: name_of_the_domain
Storage Type: Posix Compliant FS
Path: gpfs_kvm (the name of the "device"/"filesystem" created, not an
absoluthe path)
VFS Type: gpfs
Mount Options: rw,relatime,dev=gpfs_kvm (the dev parameter is the key to mount the
filesystem).
Using those options, the webinterface just tells me it cannot communicate with the storage.
If I change it to:
Domain Function: Data
Name: name_of_the_domain
Storage Type: Posix Compliant FS
Path: /mnt/b4tsz001
VFS Type: gpfs
Mount Options: rw,relatime
The oVirt host reboots immediately.
I have mounted the filesystem locally on the oVirt Host under /mnt/<filesystem name>
Change the owner to: vdsm:kvm
Am I missing something or is this a general bug between spectrum scale and oVirt.
Any ideas will be appreciated! Thanks
Best Regards
Andi Christiansen
2 years, 11 months
Help installing oVirt on single machine, without cockpit
by Cameron Showalter
The reply button is taking me to a "mailto" tab again, but hopefully
making the subject the same, will put this under the right thread? If not,
I'm not sure how to reply on the new system. The original thread is here:
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/3L6EVRT4YBEJ...,
and it continues here for a bit:
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/C4Z65RXLFPOC...
.
@Strahil Nikolov
> Have you thought about deploying HostedEngine and when you need to update
the engine -> put the host in maintenance -> backup & restore
The way I understand the update process, you can't update with a hosted
engine unless you have two hosts. One needs to run the engine VM, so the
other can go into maintenance. Is this wrong? For my use case, I'm looking
for something that'll work/update on a single host.
I'm trying to get the engine running next to the hypervisor, so it can run
when the VM's come down. I can get the engine running on a CentOS Stream
install, but it says it has 0/0 hosts running, so I don't think it can
deploy VM's. I can't find a way to install the node's hypervisor
package-set either. Unless that package exists, the only other idea I have
for this route is doing a "hosted engine" install, but destroy the engine
VM and point it to the local engine. This seems extreme to me lol.
I tried setting up the engine on a node itself, but the error I can't get
past yet is in the second url, top of this message. It can't start the
ovirt-imageio service, because of a missing file.
Happy Holidays all!
Cameron
2 years, 11 months
With libgfapi gone - ? - qcow2 backups
by lejeczek
hi guys
Now with libgfapi support removed from qemu/libvirt -
certainly in binaries from oVirt repos for CentOS 9 - how
does one take qcow2 backups?
Would that be some specific options to mount gluster vol in
order to overcome:
-> $ qemu-img convert -O qcow2 ./ubuntu.qcow2.bkp ubuntu.qcow2
qemu-img: Could not open './ubuntu-tor.qcow2.bkp': Could not
open backing file: Failed to get shared "write" lock
Is another process using the image [/VMs3/ubuntu.qcow2]?
or perhaps 'qemu-img' can do the trick somehow?
all thoughts shared are much appreciated.
many thanks, L>
2 years, 11 months
Lack of attribute "decode" in v2v module
by Diego Ercolani
Hello,
As asked by Stefano Stagnaro I'm currently writing about an issue I'm experiencing during
vm import from external VMWARE farm:
I have the system log full of error:
Dec 30 13:23:42 ovirt-node2.ovirt vdsm[3420]: *ERROR Internal server error*
*Traceback (most recent call last):*
* File "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 349, in
_handle_request*
* res = method(**params)*
* File "/usr/lib/python3.6/site-packages/vdsm/rpc/Bridge.py", line 194, in
_dynamicMethod*
* result = fn(*methodArgs)*
* File "<decorator-gen-471>", line 2, in getStats*
* File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method*
* ret = func(*args, **kwargs)*
* File "/usr/lib/python3.6/site-packages/vdsm/API.py", line 1456, in getStats*
* multipath=True)*
* File "/usr/lib/python3.6/site-packages/vdsm/host/api.py", line 49, in get_stats*
* decStats = stats.produce(first_sample, last_sample)*
* File "/usr/lib/python3.6/site-packages/vdsm/host/stats.py", line 108, in produce*
* stats['v2vJobs'] = v2v.get_jobs_status()*
* File "/usr/lib/python3.6/site-packages/vdsm/v2v.py", line 290, in get_jobs_status*
* 'description': job.description,*
*AttributeError: 'str' object has no attribute 'decode'*
As stated in the error: line 290 of */usr/lib/python3.6/site-packages/vdsm/v2v.py*
Uses the method "decode" that it doesn't seem to be enabled:
'description': job.description.decode('utf-8'),
[root@ovirt-node2 ~]# rpm -qf /usr/lib/python3.6/site-packages/vdsm/v2v.py
--
Ing. Diego Ercolani
S.S.I.S. s.p.a.
T. 0549-875910
2 years, 11 months
VMs do not run with Q35 BIOS
by Martin Marusinec
Hello,
yesterday I installed new ovirt node into the existing cluster, and I found out I cannot start any of my VMs on it. After some tries, I found out the VM simply does not start up with Q35 BIOS. It starts with legacy BIOS, or Q35 UEFI, but not with Q35 BIOS. Blank screen, nothing. The node is slightly newer then others. What could I do with it? I would rather avoid changing chipset on all my VMs to legacy, just to be able to run them on new node....
Martin
2 years, 11 months
Suggested upgrading path from CentOS based 4.4.8 to 4.4.9
by Gianluca Cecchi
I have a lab with an environment based on 4.4.8.6-1, with 3 CentOS Linux
8.4 hosts and a CentOS 8.4 external engine system (that is a VM on vSphere,
so that I can leverage a snapshot methodology for the process...).
I would like to pass to 4.4.9 and retain a full plain OS on hosts for the
moment, without going through oVirt nodes, but standing the repo problems
and CentOS 8.x going through EOL this is what I'm planning to do:
1. stop engine service on engine system
2. convert engine to CentOS Stream
This step needs some confirmation.
Could you provide an official link about the process?
I'm not able to find it again. Is it a problem of mine or all (CentOS
website, RHEL website) seem to point only to conversion from CentOS Linux
to RHEL??
Apart external websites provided workflows, I was only able to find a mid
January youtube video, when CentOS was based on 8.3, with these steps:
yum install centos-release-stream
yum swap centos-{linux,stream}-repos
yum repolist
yum distro-sync
reboot
The video link is here:
https://www.youtube.com/watch?v=Ba2ytp_8x7s
No mention at
https://www.redhat.com/en/blog/faq-centos-stream-updates
And on CentOS page I only found this:
https://centos.org/distro-faq/
with Q7 containing only the two instructions:
dnf swap centos-linux-repos centos-stream-repos
dnf distro-sync
What to use safely?
Is it possible to include some sort of documentation or links on oVirt
page, to migrate from CentOS Linux to CentOS Stream for oVirt upgrade
purposes?
3. After reboot implied, I think, in step 2., use the usual steps to update
engine to 4.4.9
4. update the first out of three hosts from CentOS Linux to CentOS Stream
and to 4.4.9.
4.a follow the same approach of engine (when defined) and pass it to Stream
retaining the 4.4.8.
4.b upgrade from the web admin gui to 4.4.9
5. Do the same for second host and third hosts
Any hints, comments, limitations in having mixed 4.4.8 and 4.4.9 hosts for
a while and such?
Thanks,
Gianluca
2 years, 11 months
Unable to start ovirt-ha-agent on all hosts
by martin＠fulmo.org
Hi list!
on a hyperconverged cluster with three hosts I am unable to start the ovirt-ha-agent.
The history:
As all three hosts were running Centos 8, I tried to upgrade host3 to Centos 8 Stream first and left all VMs and host1 and host2 untouched, basically as a test. After all migrations of VMs to host3 failed with:
```
qemu-kvm: error while loading state for instance 0x0 of device '0000:00:01.0/pcie-root-port'#0122021-12-24T00:56:49.428234Z
qemu-kvm: load of migration failed: Invalid argument
```
and since I haven't had the time to dig into that, I decided to roll back the upgrade and rebooted host3 into Centos 8 again and re-installed host3 through the engine appliance. During that process (and the restart of host3) the engine appliance became unresponsive and crashed.
The problem:
Currently all ovirt-ha-agent services on all hosts fail with the following message in /var/log/ovirt-hosted-engine-ha/agent.log
```
MainThread::INFO::2021-12-24 03:56:03,500::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.4.9 started
MainThread::INFO::2021-12-24 03:56:03,516::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Certificate common name not found, using hostname to identify host
MainThread::INFO::2021-12-24 03:56:03,575::hosted_engine::548::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2021-12-24 03:56:03,576::brokerlink::82::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'addr': 'GATEWAY_IP', 'network_test': 'dns', 'tcp_t_address': '', 'tcp_t_port': ''}
MainThread::ERROR::2021-12-24 03:56:03,577::hosted_engine::564::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors
```
Now I've stumbled upon this one [1984262](https://bugzilla.redhat.com/show_bug.cgi?id=1984262) but it doesn't seem to apply. All hosts resolve properly, all hosts also have proper hostnames set, unique /etc/hosts entries and proper A records set (in the form of hostname.subdomain.domain.tld).
The versions involved are:
```
[root@host2 ~]# rpm -qa ovirt*
ovirt-hosted-engine-setup-2.5.4-2.el8.noarch
ovirt-imageio-daemon-2.3.0-1.el8.x86_64
ovirt-host-dependencies-4.4.9-2.el8.x86_64
ovirt-vmconsole-1.0.9-1.el8.noarch
ovirt-imageio-client-2.3.0-1.el8.x86_64
ovirt-host-4.4.9-2.el8.x86_64
ovirt-python-openvswitch-2.11-1.el8.noarch
ovirt-openvswitch-ovn-host-2.11-1.el8.noarch
ovirt-provider-ovn-driver-1.2.34-1.el8.noarch
ovirt-openvswitch-ovn-2.11-1.el8.noarch
ovirt-release44-4.4.9.2-1.el8.noarch
ovirt-openvswitch-2.11-1.el8.noarch
ovirt-ansible-collection-1.6.5-1.el8.noarch
ovirt-openvswitch-ovn-common-2.11-1.el8.noarch
ovirt-hosted-engine-ha-2.4.9-1.el8.noarch
ovirt-vmconsole-host-1.0.9-1.el8.noarch
ovirt-imageio-common-2.3.0-1.el8.x86_64
```
Any hint how to fix this is really appreciated. I'd like to get the engine appliance back, remove host 3 and re-initialize it since this is a production cluster (with hosts 1 and 2 replicating the gluster storage and host 3 acting as an arbiter).
Thanks in advance, Martin
2 years, 11 months
Re: Help installing oVirt on single machine, without cockpit
by Cameron Showalter
The reply button is making me write an email, so hopefully this reaches you
all.
> I think this thread (even if the title is not so clear about the
discussions born inside) could be a good read regarding single host
limitations in terms of updating the environment, after the initial
deployment
Cool, thanks for the advice! Based on the link, it seems like the hosted
engine can't update on a "single machine" install. But if you install the
engine directly to the host, the VM doesn't need to be running, so it might
be able to. Thankfully I wasn't too far down playing with the hosted
engine, or container setup yet.
So with installing an engine directly to the node, I got it to install
ovirt-engine, by enabling the appstream, baseos, extras, and powertools
repo. Then I commented out all "includepkgs = ..." in both
`/etc/yum.repos.d/ovirt-*.repo` files. I can also get through all of the
`engine-setup` questions, but then it fails with starting the
`ovirt-imageio` service at the very end.
The logs in `/var/log/ovirt-imageio/daemon.log`:
```txt
File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/ssl.py",
line 16, in server_context
purpose=ssl.Purpose.CLIENT_AUTH, cafile=cafile)
File "/usr/lib64/python3.6/ssl.py", line 468, in create_default_context
context.load_verify_locations(cafile, capath, cadata)
FileNotFoundError: [Errno 2] No such file or directory
```
Which is the same error as this forum here:
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/DNB73ZMUB5DI...
But all the info about solving it is " Found it; Default route was set to
the NFS / SAN Network Gateway, not the Actual gateway.", and I'm not
exactly sure what that means.
I also tried the engine install/setup in a CentOS workstation VM, and it
worked great, so I thought the node is missing a package? I installed
`openssl-devel` without any luck. In both the VM and the node, I tried
`route -n`, and both CentOS vm / node had similar/expected output. I tried
to do as many default options as possible on both, so I'm surprised the
gateway is different between them.
Any idea how to get past this? I'm also open to switching to plain CentOS
if that might be more stable, and installing to that. I just love how a lot
of the node comes pre-packaged for you, and I'm not sure if you can install
a "node hypervisor" straight to CentOS. That's also outside my comfort
zone, so I'm open to advice either way.
Thanks all for letting me get just this far!
2 years, 11 months
