- Users - oVirt List Archives

broker.log not rotating
by Anton Louw 01 Jul '20

01 Jul '20

Hi All, I had a space alert on one of my nodes this morning, and when I looked around, I saw that var/log/ovirt-hosted-engine-ha/broker.log was sitting at around 30GB. Does anybody know if it is safe to delete the log file? Or is there another process that I should follow? I had a look at my other nodes, and the broker.log file does not exceed 4GB. Thank you Anton Louw Cloud Engineer: Storage and Virtualization ______________________________________ D: 087 805 1572 | M: N/A A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg anton.louw(a)voxtelecom.co.za www.vox.co.za

1 0

VMs shutdown mysteriously
by Bobby 01 Jul '20

01 Jul '20

Hello, All 4 VMs on one of my oVirt cluster node shutdown for an unknown reason almost simultaneously. Please help me to find the root cause. Thanks. Please note the host seems doing fine and never crash or hangs and I can migrate VMs back to it later. Here is the exact timeline of all the related events combined from the host and the VM(s): On oVirt host: /var/log/vdsm/vdsm.log: 2020-06-25 15:25:16,944-0500 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f4ed2f9f5f0> on ['e0257b06-28fd-4d41-83a9-adf1904d3622'] (periodic:289) 2020-06-25 15:25:19,203-0500 WARN (libvirt/events) [root] File: /var/lib/libvirt/qemu/channels/e0257b06-28fd-4d41-83a9-adf1904d3622.ovirt-guest-agent.0 already removed (fileutils:54) 2020-06-25 15:25:19,203-0500 WARN (libvirt/events) [root] File: /var/lib/libvirt/qemu/channels/e0257b06-28fd-4d41-83a9-adf1904d3622.org.qemu.guest_agent.0 already removed (fileutils:54) [root@athos log]# journalctl -u NetworkManager --since=today -- Logs begin at Wed 2020-05-20 22:07:33 CDT, end at Thu 2020-06-25 16:36:05 CDT. -- Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1136] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1146] device (vnet0): released from master device SRV-VL /var/log/messages: Jun 25 15:25:18 athos kernel: SRV-VL: port 2(vnet0) entered disabled state Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1136] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1146] device (vnet0): released from master device SRV-VL Jun 25 15:25:18 athos libvirtd: 2020-06-25 20:25:18.122+0000: 2713: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor /var/log/libvirt/qemu/aries.log: 2020-06-25T20:25:28.353975Z qemu-kvm: terminating on signal 15 from pid 2713 (/usr/sbin/libvirtd) 2020-06-25 20:25:28.584+0000: shutting down, reason=shutdown ============================================================================================= On the first VM effected (same thing on others): /var/log/ovirt-guest-agent/ovirt-guest-agent.log: MainThread::INFO::2020-06-25 15:25:20,270::ovirt-guest-agent::104::root::Stopping oVirt guest agent CredServer::INFO::2020-06-25 15:25:20,626::CredServer::262::root::CredServer has stopped. MainThread::INFO::2020-06-25 15:25:21,150::ovirt-guest-agent::78::root::oVirt guest agent is down. ============================================================================================= Packages version installated: Host OS version: CentOS 7.7.1908: ovirt-hosted-engine-ha-2.3.5-1.el7.noarch ovirt-provider-ovn-driver-1.2.22-1.el7.noarch ovirt-release43-4.3.6-1.el7.noarch ovirt-imageio-daemon-1.5.2-0.el7.noarch ovirt-vmconsole-1.0.7-2.el7.noarch ovirt-imageio-common-1.5.2-0.el7.x86_64 ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch ovirt-vmconsole-host-1.0.7-2.el7.noarch ovirt-host-4.3.4-1.el7.x86_64 libvirt-4.5.0-23.el7_7.1.x86_64 libvirt-daemon-4.5.0-23.el7_7.1.x86_6 qemu-kvm-ev-2.12.0-33.1.el7.x86_64 qemu-kvm-common-ev-2.12.0-33.1.el7.x86_64 On guest VM: ovirt-guest-agent-1.0.13-1.el6.noarch qemu-guest-agent-0.12.1.2-2.491.el6_8.3.x86_64

4 4

Re: Some nodes periodically display as Non Responsive
by Martin Perina 30 Jun '20

30 Jun '20

Hi Anton, to diagnose the issue we would need to have logs from both engine and affected host. Regards, Martin On Wed, Jul 1, 2020 at 6:51 AM Anton Louw via Users <users(a)ovirt.org> wrote: > > > Hi Everybody, > > > > I am got some strange things happening. I have got two data centers, DC1 > and DC2, in DC1, some of my nodes (Not all the time and not all the nodes) > go into a “not responding” state. I can still ping the hosts, and I can > still access the VMs on the hosts. My Engine sits in DC2, and this does not > happen to any of the hosts in DC2. > > > > It seems like the Engine loses connectivity to the hosts in DC1, and then > cannot re-establish the connection. > > > > Is there anywhere I can check to get more insight into what is actually > happening? > > > > Thanks > > > > *Anton Louw* > *Cloud Engineer: Storage and Virtualization* at *Vox* > ------------------------------ > *T:* 087 805 0000 | *D:* 087 805 1572 > *M:* N/A > *E:* anton.louw(a)voxtelecom.co.za > *A:* Rutherford Estate, 1 Scott Street, Waverley, Johannesburg > www.vox.co.za > > [image: F] <https://www.facebook.com/voxtelecomZA> > [image: T] <https://www.twitter.com/voxtelecom> > [image: I] <https://www.instagram.com/voxtelecomza/> > [image: L] <https://www.linkedin.com/company/voxtelecom> > [image: Y] <https://www.youtube.com/user/VoxTelecom> > > [image: #VoxBrand] > <https://www.vox.co.za/fibre/fibre-to-the-home/?prod=HOME> > *Disclaimer* > > The contents of this email are confidential to the sender and the intended > recipient. Unless the contents are clearly and entirely of a personal > nature, they are subject to copyright in favour of the holding company of > the Vox group of companies. Any recipient who receives this email in error > should immediately report the error to the sender and permanently delete > this email from all storage devices. > > This email has been scanned for viruses and malware, and may have been > automatically archived by *Mimecast Ltd*, an innovator in Software as a > Service (SaaS) for business. Providing a *safer* and *more useful* place > for your human generated data. Specializing in; Security, archiving and > compliance. To find out more Click Here > <https://www.voxtelecom.co.za/security/mimecast/?prod=Enterprise>. > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/TRMWV4Q6AFHG5… > -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.

1 0

Some nodes periodically display as Non Responsive
by Anton Louw 30 Jun '20

30 Jun '20

Hi Everybody, I am got some strange things happening. I have got two data centers, DC1 and DC2, in DC1, some of my nodes (Not all the time and not all the nodes) go into a "not responding" state. I can still ping the hosts, and I can still access the VMs on the hosts. My Engine sits in DC2, and this does not happen to any of the hosts in DC2. It seems like the Engine loses connectivity to the hosts in DC1, and then cannot re-establish the connection. Is there anywhere I can check to get more insight into what is actually happening? Thanks Anton Louw Cloud Engineer: Storage and Virtualization ______________________________________ D: 087 805 1572 | M: N/A A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg anton.louw(a)voxtelecom.co.za www.vox.co.za

1 0

status of oVirt 4.4.x and CentOS 8.2
by Gianluca Cecchi 30 Jun '20

30 Jun '20

Hello, what is the current status both if using plain CentOS based nodes and ovirt-node-ng? Do the release of CentOS 8.2 impact new installation for 4.4.0 and/or 4.4.1rc? Thanks, Gianluca

5 8

Localdisk hook not working
by tim-nospam＠bordemann.com 30 Jun '20

30 Jun '20

Hi, I'd like to use the localdisk hook of vdsm and have configured everything according to the readme: https://github.com/oVirt/vdsm/tree/master/vdsm_hooks/localdisk After installing the hook, configuring the ovirt-engine, creating the volume group, adding the custom property 'localdisk' to the virtual machine and fixing a small bug in the localdisk-helper, vdsm creates the logical volume on the 'ovirt-local' volume group after starting the virtual machine. Unfortunately nothing more seems to happen then. There is no activity on the NAS which would indicate that the disk image is pulled to the host. I can also see no errors on the host. Is anyone currently running ovirt 4.4 with the localdisk hook? What else can I do to find out why the image is not being copied to my host? Thanks, Tim

2 2

Re: VDSM not binding on IPv4 after ovirt-engine restart
by Dominik Holler 30 Jun '20

30 Jun '20

On Tue, Jun 30, 2020 at 6:13 PM Erez Zarum <erezz(a)nanosek.com> wrote: > While troubleshooting a fresh installation of (after a failed one) that > caused all the hosts but the one running the hosted-engine to become in > “Unassigned” state I noticed that the ovirt-engine complains about not > being able to contact the VDSM. > I noticed that VDSM has stopped listening on IPv4. > > Thanks for sharing the details. > I didn’t disable any IPv6 as it states not to disable it on hosts that are > capable running the hosted-engine and it seems that the reason behind it > is that the hosted-engine talks to the host it runs on through “localhost”, > this also explains why the host which the hosted-engine runs on is “OK”. > > Below is from a host that does not run the hosted-engine: > # ss -atn | grep 543 > LISTEN 0 5 *:54322 *:* > ESTAB 0 0 127.0.0.1:54792 127.0.0.1:54321 > ESTAB 0 0 127.0.0.1:54798 127.0.0.1:54321 > LISTEN 0 5 [::]:54321 [::]:* > ESTAB 0 0 [::ffff:127.0.0.1]:54321 > [::ffff:127.0.0.1]:54798 > ESTAB 0 0 [::ffff:127.0.0.1]:54321 > [::ffff:127.0.0.1]:54792 > ESTAB 0 0 [::1]:54321 [::1]:50238 > ESTAB 0 0 [::1]:50238 [::1]:54321 > > Below is from a host that runs the hosted-engine at the moment: > # ss -atn | grep 543 > LISTEN 0 5 *:54322 *:* > LISTEN 0 5 [::]:54321 [::]:* > ESTAB 0 0 [::1]:51230 [::1]:54321 > ESTAB 0 0 [::1]:54321 [::1]:51242 > ESTAB 0 0 [::ffff:10.46.20.23]:54321 > [::ffff:10.46.20.20]:45706 > ESTAB 0 0 [::ffff:10.46.20.23]:54321 > [::ffff:10.46.20.20]:45746 > ESTAB 0 0 [::1]:51240 [::1]:54321 > ESTAB 0 0 [::1]:54321 [::1]:51230 > ESTAB 0 0 [::1]:51242 [::1]:54321 > ESTAB 0 0 [::1]:54321 [::1]:51240 > > The hosted-engine IP is 10.46.20.20 and the host is 10.46.20.23. > > Why do you think the host does not listen to IPv4 anymore? Can you please share the output of "nc -vz 10.46.20.23 54321" executed on engine VM or another host? > /etc/hosts on all hosts: > 127.0.0.1 localhost localhost.localdomain localhost4 > localhost4.localdomain4 > ::1 localhost localhost.localdomain localhost6 > localhost6.localdomain6 > > Perhaps this is relevant but all hosts are enrolled into IDM (FreeIPA) and > as an outcome they all have a DNS record and a PTR record as well as the > ovirt-engine VM. > > # cat /etc/vdsm/vdsm.conf > [vars] > ssl = true > ssl_ciphers = HIGH:!aNULL > ssl_excludes = OP_NO_TLSv1,OP_NO_TLSv1_1 > > [addresses] > management_port = 54321 > > I have tried adding “management_ip = 0.0.0.0” but then it only binds to > IPv4 and yet, the host still shows as Unassigned, sometimes it switches to > “NonResponsive” and trying to “Reinstall” the host fails, the ovirt-engine > complains it can't contact/reach the VDSM, while using netcat from the > ovirt-engine it works. > > I have KSM and Memory Ballooning enabled on the Cluster as well. > > oVirt 4.3.10 installed on CentOS 7.8.2003 > The self-hosted Engine runs on an external GlusterFS, before reinstalling > everything (fresh start of OS, etc..) I tried iSCSI as well. > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/NCTWZLS2VPIAB… >

2 1

VDSM not binding on IPv4 after ovirt-engine restart
by Erez Zarum 30 Jun '20

30 Jun '20

While troubleshooting a fresh installation of (after a failed one) that caused all the hosts but the one running the hosted-engine to become in “Unassigned” state I noticed that the ovirt-engine complains about not being able to contact the VDSM. I noticed that VDSM has stopped listening on IPv4. I didn’t disable any IPv6 as it states not to disable it on hosts that are capable running the hosted-engine and it seems that the reason behind it is that the hosted-engine talks to the host it runs on through “localhost”, this also explains why the host which the hosted-engine runs on is “OK”. Below is from a host that does not run the hosted-engine: # ss -atn | grep 543 LISTEN 0 5 *:54322 *:* ESTAB 0 0 127.0.0.1:54792 127.0.0.1:54321 ESTAB 0 0 127.0.0.1:54798 127.0.0.1:54321 LISTEN 0 5 [::]:54321 [::]:* ESTAB 0 0 [::ffff:127.0.0.1]:54321 [::ffff:127.0.0.1]:54798 ESTAB 0 0 [::ffff:127.0.0.1]:54321 [::ffff:127.0.0.1]:54792 ESTAB 0 0 [::1]:54321 [::1]:50238 ESTAB 0 0 [::1]:50238 [::1]:54321 Below is from a host that runs the hosted-engine at the moment: # ss -atn | grep 543 LISTEN 0 5 *:54322 *:* LISTEN 0 5 [::]:54321 [::]:* ESTAB 0 0 [::1]:51230 [::1]:54321 ESTAB 0 0 [::1]:54321 [::1]:51242 ESTAB 0 0 [::ffff:10.46.20.23]:54321 [::ffff:10.46.20.20]:45706 ESTAB 0 0 [::ffff:10.46.20.23]:54321 [::ffff:10.46.20.20]:45746 ESTAB 0 0 [::1]:51240 [::1]:54321 ESTAB 0 0 [::1]:54321 [::1]:51230 ESTAB 0 0 [::1]:51242 [::1]:54321 ESTAB 0 0 [::1]:54321 [::1]:51240 The hosted-engine IP is 10.46.20.20 and the host is 10.46.20.23. /etc/hosts on all hosts: 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 Perhaps this is relevant but all hosts are enrolled into IDM (FreeIPA) and as an outcome they all have a DNS record and a PTR record as well as the ovirt-engine VM. # cat /etc/vdsm/vdsm.conf [vars] ssl = true ssl_ciphers = HIGH:!aNULL ssl_excludes = OP_NO_TLSv1,OP_NO_TLSv1_1 [addresses] management_port = 54321 I have tried adding “management_ip = 0.0.0.0” but then it only binds to IPv4 and yet, the host still shows as Unassigned, sometimes it switches to “NonResponsive” and trying to “Reinstall” the host fails, the ovirt-engine complains it can't contact/reach the VDSM, while using netcat from the ovirt-engine it works. I have KSM and Memory Ballooning enabled on the Cluster as well. oVirt 4.3.10 installed on CentOS 7.8.2003 The self-hosted Engine runs on an external GlusterFS, before reinstalling everything (fresh start of OS, etc..) I tried iSCSI as well.

1 0

4.4.1-rc5: Looking for correct way to configure machine=q35 instead of machine=pc for arch=x86_64
by Glenn Marcy 29 Jun '20

29 Jun '20

Hello, I am hoping for some insight from folks with more hosted engine install experience. When I try to install the hosted engine using the RC5 dist I get the following error during the startup of the HostedEngine VM: XML error: The PCI controller with index='0' must be model='pci-root' for this machine type, but model='pcie-root' was found instead This is due to the HE Domain XML description using machine="pc-i440fx-rhel7.6.0". I've tried to override the default of 'pc' from ovirt-ansible-hosted-engine-setup/defaults/main.yml: he_emulated_machine: pc by passing to the ovirt-hosted-engine-setup script a --config-append=file parameter where file contains: [environment:default] OVEHOSTED_VM/emulatedMachine=str:q35 When the "Create ovirt-hosted-engine-ha run directory" step finishes the vm.conf file contains: cpuType=IvyBridge,+pcid,+spec-ctrl,+ssbd,+md-clear emulatedMachine=q35 At the "Start ovirt-ha-broker service on the host" step that file is removed. When that file appears again during the "Check engine VM health" step it now contains: cpuType=IvyBridge,+pcid,+spec-ctrl,+ssbd,+md-clear emulatedMachine=pc-i440fx-rhel7.6.0 After that the install fails with the metadata from "virsh dumpxml HostedEngine" containing: <ovirt-vm:exitCode type="int">1</ovirt-vm:exitCode> <ovirt-vm:exitMessage>XML error: The PCI controller with index='0' must be model='pci-root' for this machine type, but model='pcie-root' was found instead</ovirt-vm:exitMessage> Interestingly enough, the HostedEngineLocal VM that is running the appliance image has the value I need: <type arch='x86_64' machine='pc-q35-rhel8.2.0'>hvm</type> Does anyone on the list have any experience with where this needs to be overridden? Somewhere in the hosted engine setup or do I need to do something at a deeper level like vdsm or libvirt? Help much appreciated ! Thanks, Glenn

2 3

vdsm error every hour
by eevans＠digitaldatatechs.com 29 Jun '20

29 Jun '20

In the web ui I get an error about every hour: VDSM command SetVolumeDescriptionVDS failed: Volume does not exist: (u'e3f79840-8355-45b0-ad2b-440c877be637',) I looked in storage and disks and this disk does not exist. Its more of an annoyance than a problem but if there is way to get rid of this error I would like to know. My research says I can install vdsm-tools and vdsm-cli but vdsm-cli is not available and I really don;t want to install anything until I know it's what I need. Is there a vdsm command to purge a missing disk so this error won't show up? Thanks in advance. Eric

2 1