August 2018 - Users - oVirt List Archives

Guide for setting up GlusterFS geo-replication?
by Jayme 29 Aug '18

29 Aug '18

Is there an updated guide for setting up GlusterFS geo-replication? What I am interested in is having another oVirt setup on a separate server with glusterFS volume replicated to it. If my primary cluster went down I would be able to start important VMs on the secondary oVirt build until I'm able to repair the main cluster. In oVirt 4.2.5 clicking geo-replication > new in the volume GUI produces no results. Nothing happens when I click it (using multiple browsers). And under storage domains clicking remote data-sync -> setup produces a popup but just has a loading icon in the middle and never completes loading. Why are neither of these two features working for me?

4 8

Turn Off Email Alerts
by Douglas Duckworth 28 Aug '18

28 Aug '18

Hi How do I turn off hosted engine alerts? We are in a testing phase so these are not needed. I have disabled postfix on all hosts as well as stopped the ovirt notification daemon on the hosted engine. I kept it running while putting /dev/null in /usr/share/ovirt-engine/services/ovirt-engine-notifier/ovirt-engine-notifier.conf for mail server. Yet I still get alerts for every thing done such as putting hosts in maintenance mode. Very confusing.

2 2

Re: SSSD on Hosted Engine
by Douglas Duckworth 28 Aug '18

28 Aug '18

It says I am not authorized. Can you give me permission to comment? My bugzilla account would be doug(a)med.cornell.edu. Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Weill Cornell Medicine 1300 York - LC-502 E: doug(a)med.cornell.edu O: 212-746-6305 F: 212-746-8690 On Wed, Aug 22, 2018 at 10:44 AM, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote: > > > 2018-08-22 14:59 GMT+02:00 Douglas Duckworth <dod2014(a)med.cornell.edu>: > >> Yay, I was able to restore nsswitch so things now work. >> >> Anyway that oVirt can use SSSD for web auth? >> >> Per https://ovirt.org/develop/release-management/features/infra/aaa_faq/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__ovirt.org_develop_rele…> >> there's a bug https://bugzilla.redhat.com/show_bug.cgi?id=829292 >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.redhat.com_sh…> >> that prevents it from working? >> > > I would suggest to comment on that bug with your use case, it will help > understanding the needs. > > > > > > >> >> >> >> Thanks, >> >> Douglas Duckworth, MSc, LFCS >> HPC System Administrator >> Scientific Computing Unit >> Weill Cornell Medicine >> 1300 York - LC-502 >> E: doug(a)med.cornell.edu >> O: 212-746-6305 >> F: 212-746-8690 >> >> >> On Wed, Aug 22, 2018 at 8:51 AM, Douglas Duckworth < >> dod2014(a)med.cornell.edu> wrote: >> >>> Hi >>> >>> I am trying to configure sssd on my hosted engine. Essentially we >>> control host access in LDAP so I want sssd to read that thus allow my >>> coworkers to login to hosted engine vm. >>> >>> For some reason sssd reports backend offline even though it's >>> resolvable, pingable, with ports open. I see that it's a SELinux issue >>> which I can resolve. After changing to permissive SSSD works. >>> >>> To have system read sssd database I set hosts line in /etc/nsswitch.conf >>> to: >>> >>> hosts files sss >>> >>> Though it seems that I did something bad to /etc/nsswitch.conf as now >>> yum, ping, etc does not work. >>> >>> Could someone suggest how to restore this file or could anyone share >>> theirs? >>> >>> Thanks, >>> >>> Douglas Duckworth, MSc, LFCS >>> HPC System Administrator >>> Scientific Computing Unit >>> Weill Cornell Medicine >>> 1300 York - LC-502 >>> E: doug(a)med.cornell.edu >>> O: 212-746-6305 >>> F: 212-746-8690 >>> >>> >> >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_site_pri…> >> oVirt Code of Conduct: https://www.ovirt.org/communit >> y/about/community-guidelines/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_communit…> >> List Archives: https://lists.ovirt.org/archiv >> es/list/users(a)ovirt.org/message/DSFBEFKBNAWZMLMASRJ7YKZO3PZWKJLV/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ovirt.org_archiv…> >> >> > > > -- > > SANDRO BONAZZOLA > > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > > Red Hat EMEA > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.redhat.com_&d=DwMF…> > > sbonazzo(a)redhat.com > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__red.ht_sig&d=DwMFaQ&c=…> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.redhat.com_en_even…> >

2 1

mask or disabled lldpad service
by Klaas Demter 28 Aug '18

28 Aug '18

Hi, I have a QLogic Corp. QLogic 2x1GE+2x10GE QL41162HMRJ CNA network card. This network card comes with it's own lldp implementation inside their management firmware. Running two lldp agents seems to create several issues (http://lists.us.dell.com/pipermail/linux-poweredge/2018-July/051860.html) The QLogic engineers said I need to disable lldpad in the operating system. This leads me here, what is lldp actually used for within ovirt? If it is actually needed how could I deal with a card that does not support lldpad but rather has it's own implementation? Greetings Klaas

2 1

ovirt selfhost error
by mustafa.taha.mu95＠gmail.com 28 Aug '18

28 Aug '18

hi , i am a new user of ovirt , i want to setup self host , i follow the step in https://www.ovirt.org/documentation/self-hosted/chap-Deploying_Self-Hoste... , after i configure storage this error appear fatal: [localhost]: FAILED! => {"changed": false, "msg": "Engine VM is not running, please check vdsm logs"} how can i fixed it ?

3 6

[ANN] oVirt 4.2.6 Third Release Candidate is now available
by Sandro Bonazzola 28 Aug '18

28 Aug '18

The oVirt Project is pleased to announce the availability of the oVirt 4.2.6 third Release Candidate, as of August 28th, 2018. This update is a release candidate of the sixth in a series of stabilization updates to the 4.2 series. This is pre-release software. This pre-release should not to be used in production. This release is available now for: * Red Hat Enterprise Linux 7.5 or later * CentOS Linux (or similar) 7.5 or later This release supports Hypervisor Hosts running: * Red Hat Enterprise Linux 7.5 or later * CentOS Linux (or similar) 7.5 or later See the release notes [1] for installation / upgrade instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is available - oVirt Node is available [2] - oVirt Windows Guest Tools is available [2] Additional Resources: * Read more about the oVirt 4.2.6 release highlights: http://www.ovirt.org/release/4.2.6/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.2.6/ [2] http://resources.ovirt.org/pub/ovirt-4.2-pre/iso/ -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=701f…>

1 0

VDO + HCI = failed reboots
by Donny Davis 28 Aug '18

28 Aug '18

I just spun up the latest and greatest ovirt has to offer, and I am building out an HCI cluster. The deployment went wonderfully. I had dns setup for everything, and it just worked. Great job team! I just wanted to add in something i noticed after rebooting the nodes. The gluster service, and volumes would not come back up after rebooting. I found that there needs to be an entry in the fstab for the gluster logical volumes to tell them to start the vdo service. So without this, the node will fail to boot if you are using vdo. inode64,noatime,nodiratime,discard,x-systemd.requires=vdo.service the defaults are this inode64,noatime,nodiratime Hopefully it helps someone else :) Cheers

2 4

hosted-engine --deploy erorr
by mustafa.taha.mu95＠gmail.com 28 Aug '18

28 Aug '18

hi , i am a new user of ovirt , i want to setup self host , i follow the step in https://www.ovirt.org/documentation/self-hosted/chap-Deploying_Self-Hosted_… , after i configure storage this error appear fatal: [localhost]: FAILED! => {"changed": false, "msg": "Engine VM is not running, please check vdsm logs"} how can i fixed it ?

1 0

Migrating to HCI
by Vincent Royer 28 Aug '18

28 Aug '18

I have 2 hosts using NFS storage I want to migrate to HCI and add a third host. Have added the 10Gbe cards and drives to the 3 target hosts and setup a separate physical network for storage. Can I: - deploy a single-node HCI to host 3 - export the VMs to an NFS export domain - Import the VMs to the HCI node - wipe the first 2 hosts and re-deploy them to the HCI DC Is there a guide for this? or should I just export the VMs, blow away all the nodes, and deploy as a 3-node HCI using the wizards? I'm hoping to minimize downtime of the VMs to under one day. *Vince*

2 1

One Windows VM is not responding
by Jonathan Baecker 27 Aug '18

27 Aug '18

Hello Everybody! We have here 13 VMs running under ovirt 4.2.4. Two of them are Windows Server 2016. On one runs a AD, DNS and one application. On the second one runs an SQL Server and also an application. This second one have the problem, that periodical it goes in a state where I get in ovirt the message: *not responding * Beside this the VM is running normal. I can connect it over Remote Desktop. But I can not connect it with ovirt/noVNC, the button is gray out. In a weekly cycle it runs a backup script, maybe this brings some problems, but I really don't know how to debug this. It can be that the VM runs for 3 weeks normal, and then it goes in this state, so I can not really say when this is happen. Also the fact that the other Windows Server VM runs normal, wonders me. Do you have experienced this problem, or do you know how I found out the issue? Best Regards! Jonathan

2 2

Re: External ceph storage
by Luca 'remix_tj' Lorenzetto 27 Aug '18

27 Aug '18

No, The only way you have is to configure cinder to manage ceph pool or in alternative you have to deploy an iscsi gateway, no other ways are available at the moment. So you can't use rbd directly. Luca Il dom 27 mag 2018, 16:54 Leo David <leoalex(a)gmail.com> ha scritto: > Thank you Luca, > At the moment i would try the cinder storage provider, since we already > have a proxmox cluster directly connecting to ceph. The problem is that I > just could not find a straight way to do this. > ie: Specify the ceph monitors and ceph pool to connect to. Can oVirt > directly connect to ceph monitors ? How the configuration should be done if > so ? > Thank you very much ! > > > On Sun, May 27, 2018, 17:20 Luca 'remix_tj' Lorenzetto < > lorenzetto.luca(a)gmail.com> wrote: > >> Hello, >> >> Yes, using cinder or through iscsi gateway. >> >> For a simpler setup i suggest the second option. >> >> Luca >> >> Il dom 27 mag 2018, 16:08 Leo David <leoalex(a)gmail.com> ha scritto: >> >>> Hello everyone, >>> I am new to ovirt and very impressed of its features. I would like to >>> levereage on our existing ceph cluster to provide rbd images for vm hdds, >>> is this possible to achieve ? >>> Thank you very much ! >>> Regards, >>> Leo >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> >>

3 3

Unknown CPU model Broadwell-IBRS-SSBD
by me＠brendanh.com 24 Aug '18

24 Aug '18

Hi, I've just installed hosted-engine. It proceeds to near the end of the Ansible script, then pauses at: "TASK [Check engine VM health]" then eventually errors: "Engine VM is not running, please check vdsm logs. The VDSM logs have error: "libvirtError: internal error: Unknown CPU model Broadwell-IBRS-SSBD" Also, in the oVirt Node webui, the HostedEngine VM is created, but not started. If I press Run, the following error is shown: VM START action failed error: Failed to start domain HostedEngine error: internal error: Unknown CPU model Broadwell-IBRS-SSBD I notice a Trello card was created to add similar Skylake & EPYC CPUs, but not Broadwell: https://trello.com/c/5nGTgQ9P/89-new-ibrs-and-ssbd-cpu-types# Does the Broadwell-IBRS-SSBD CPU need to be added to this supported types list? Many thanks.

3 4

Re: losing ib0 connection after activating host
by Douglas Duckworth 24 Aug '18

24 Aug '18

Sorry, I mean "migration network" for moving live migration traffic. FDR infiniband much faster than 1Gb network which currently acts as migration network, vm network, display network, mgmt network, etc. Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Weill Cornell Medicine 1300 York - LC-502 E: doug(a)med.cornell.edu O: 212-746-6305 F: 212-746-8690 On Fri, Aug 24, 2018 at 9:36 AM, Dominik Holler <dholler(a)redhat.com> wrote: > On Thu, 23 Aug 2018 13:51:39 -0400 > Douglas Duckworth <dod2014(a)med.cornell.edu> wrote: > > > THANKS! > > > > ib0 now up with NFS storage back on this hypervisor > > > > Thanks for letting us know. > > > Though how do I make it a transfer network? I don't see an option. > > > > I do not understand the meaning of "transfer network". > The network interface to use for NFS results from the routing tables of > the host. > In "Compute > Clusters > Clustername > Logical Networks > Manage > Networks" network roles for some kind of loads can be assigned, but not > for NFS access. > > > > Thanks, > > > > Douglas Duckworth, MSc, LFCS > > HPC System Administrator > > Scientific Computing Unit > > Weill Cornell Medicine > > 1300 York - LC-502 > > E: doug(a)med.cornell.edu > > O: 212-746-6305 > > F: 212-746-8690 > > > > > > On Thu, Aug 23, 2018 at 11:12 AM, Douglas Duckworth > > <dod2014(a)med.cornell.edu > > > wrote: > > > > > Hi Dominik > > > > > > Yes, the network-script was created by our Ansible role that deploys > > > CentOS hosts. It pulls the IP from DNS then templates the script > > > and copies to host. > > > > > > I will try this oVirt step then see if it works! > > > > > > Thanks, > > > > > > Douglas Duckworth, MSc, LFCS > > > HPC System Administrator > > > Scientific Computing Unit > > > Weill Cornell Medicine > > > 1300 York - LC-502 > > > E: doug(a)med.cornell.edu > > > O: 212-746-6305 > > > F: 212-746-8690 > > > > > > > > > On Thu, Aug 23, 2018 at 11:09 AM, Dominik Holler > > > <dholler(a)redhat.com> wrote: > > > > > >> Is ifcfg-ib0 created before adding the host? > > >> Can ib0 be reconfigured using engine, e.g. by > > >> "Compute > Hosts > hostx > Network Interfaces > Setup Host > > >> Networks"? If this some kind of self-hosted engine? > > >> > > >> On Thu, 23 Aug 2018 09:30:59 -0400 > > >> Douglas Duckworth <dod2014(a)med.cornell.edu> wrote: > > >> > > >> > Here's a link to the files: > > >> > > > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__bit.ly_ > > >> 2wjZ6Vo&d=DwICAg&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu > > >> 2s&r=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m=Y25- > > >> OOvgu58jlC82-fzBeNIpQ7ZscoHznffUhqE6EBM&s=QQXlC9Tisa60TvimyS > > >> 3BnFDCaDF7VPD8eCzT-Fke-p0&e= > > >> > > > >> > Thank you! > > >> > > > >> > Thanks, > > >> > > > >> > Douglas Duckworth, MSc, LFCS > > >> > HPC System Administrator > > >> > Scientific Computing Unit > > >> > Weill Cornell Medicine > > >> > 1300 York - LC-502 > > >> > E: doug(a)med.cornell.edu > > >> > O: 212-746-6305 > > >> > F: 212-746-8690 > > >> > > > >> > > > >> > On Thu, Aug 23, 2018 at 6:51 AM, Dominik Holler > > >> > <dholler(a)redhat.com> wrote: > > >> > > > >> > > Would you please share the vdsm.log and the supervdsm.log from > > >> > > this host? > > >> > > > > >> > > On Wed, 22 Aug 2018 11:36:09 -0400 > > >> > > Douglas Duckworth <dod2014(a)med.cornell.edu> wrote: > > >> > > > > >> > > > Hi > > >> > > > > > >> > > > I keep losing ib0 connection on hypervisor after adding host > > >> > > > to engine. This makes the host not really work since NFS > > >> > > > will be mounted over ib0. > > >> > > > > > >> > > > I don't really understand why this occurs. > > >> > > > > > >> > > > OS: > > >> > > > > > >> > > > [root@ovirt-hv2 ~]# cat /etc/redhat-release > > >> > > > CentOS Linux release 7.5.1804 (Core) > > >> > > > > > >> > > > Here's the network script: > > >> > > > > > >> > > > [root@ovirt-hv2 ~]# > > >> > > > cat /etc/sysconfig/network-scripts/ifcfg-ib0 DEVICE=ib0 > > >> > > > BOOTPROTO=static > > >> > > > IPADDR=172.16.0.207 > > >> > > > NETMASK=255.255.255.0 > > >> > > > ONBOOT=yes > > >> > > > ZONE=public > > >> > > > > > >> > > > When I try "ifup" > > >> > > > > > >> > > > [root@ovirt-hv2 ~]# ifup ib0 > > >> > > > Error: Connection activation failed: No suitable device > > >> > > > found for this connection. > > >> > > > > > >> > > > The error in syslog: > > >> > > > > > >> > > > Aug 22 11:31:50 ovirt-hv2 kernel: IPv4: martian source > > >> > > > 172.16.0.87 from 172.16.0.49, on dev ib0 > > >> > > > Aug 22 11:31:53 ovirt-hv2 NetworkManager[1070]: <info> > > >> > > > [1534951913.7486] audit: op="connection-activate" > > >> > > > uuid="2ab4abde-b8a5-6cbc-19b1-2bfb193e4e89" name="System ib0" > > >> > > > result="fail" reason="No suitable device found for this > > >> > > > connection. > > >> > > > > > >> > > > As you can see media state up: > > >> > > > > > >> > > > [root@ovirt-hv2 ~]# ip a > > >> > > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state > > >> > > > UNKNOWN group default qlen 1000 > > >> > > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > >> > > > inet 127.0.0.1/8 scope host lo > > >> > > > valid_lft forever preferred_lft forever > > >> > > > 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq > > >> > > > master ovirtmgmt state UP group default qlen 1000 > > >> > > > link/ether 50:9a:4c:89:d3:81 brd ff:ff:ff:ff:ff:ff > > >> > > > 3: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq > > >> > > > state DOWN group default qlen 1000 > > >> > > > link/ether 50:9a:4c:89:d3:82 brd ff:ff:ff:ff:ff:ff > > >> > > > 4: p1p1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc > > >> > > > mq state DOWN group default qlen 1000 > > >> > > > link/ether b4:96:91:13:ea:68 brd ff:ff:ff:ff:ff:ff > > >> > > > 5: p1p2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc > > >> > > > mq state DOWN group default qlen 1000 > > >> > > > link/ether b4:96:91:13:ea:6a brd ff:ff:ff:ff:ff:ff > > >> > > > 6: idrac: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > > >> > > > pfifo_fast state UNKNOWN group default qlen 1000 > > >> > > > link/ether 50:9a:4c:89:d3:84 brd ff:ff:ff:ff:ff:ff > > >> > > > inet 169.254.0.2/16 brd 169.254.255.255 scope global > > >> > > > idrac valid_lft forever preferred_lft forever > > >> > > > 7: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq > > >> > > > state UP group default qlen 256 > > >> > > > link/infiniband > > >> > > > a0:00:02:08:fe:80:00:00:00:00:00:00:ec:0d:9a:03:00:1d:13:41 > > >> > > > brd > > >> > > > 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff > > >> > > > 8: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop > > >> > > > state DOWN group default qlen 1000 link/ether > > >> > > > 12:b4:30:22:39:5b brd ff:ff:ff:ff:ff:ff 9: br-int: > > >> > > > <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group > > >> > > > default qlen 1000 link/ether 3e:32:e6:66:98:49 brd > > >> > > > ff:ff:ff:ff:ff:ff 25: ovirtmgmt: > > >> > > > <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue > > >> > > > state UP group default qlen 1000 link/ether > > >> > > > 50:9a:4c:89:d3:81 brd ff:ff:ff:ff:ff:ff inet 10.0.0.183/16 > > >> > > > brd 10.0.255.255 scope global ovirtmgmt valid_lft forever > > >> > > > preferred_lft forever 26: genev_sys_6081: > > >> > > > <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue > > >> > > > master ovs-system state UNKNOWN group default qlen 1000 > > >> > > > link/ether aa:32:82:1b:01:d9 brd ff:ff:ff:ff:ff:ff > > >> > > > 27: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop > > >> > > > state DOWN group default qlen 1000 link/ether > > >> > > > 32:ff:5d:b8:c2:b4 brd ff:ff:ff:ff:ff:ff > > >> > > > > > >> > > > The card is FDR: > > >> > > > > > >> > > > [root@ovirt-hv2 ~]# lspci -v | grep Mellanox > > >> > > > 01:00.0 Network controller: Mellanox Technologies MT27500 > > >> > > > Family [ConnectX-3] > > >> > > > Subsystem: Mellanox Technologies Device 0051 > > >> > > > > > >> > > > Latest OFED driver: > > >> > > > > > >> > > > [root@ovirt-hv2 ~]# /etc/init.d/openibd status > > >> > > > > > >> > > > HCA driver loaded > > >> > > > > > >> > > > Configured IPoIB devices: > > >> > > > ib0 > > >> > > > > > >> > > > Currently active IPoIB devices: > > >> > > > ib0 > > >> > > > Configured Mellanox EN devices: > > >> > > > > > >> > > > Currently active Mellanox devices: > > >> > > > ib0 > > >> > > > > > >> > > > The following OFED modules are loaded: > > >> > > > > > >> > > > rdma_ucm > > >> > > > rdma_cm > > >> > > > ib_ipoib > > >> > > > mlx4_core > > >> > > > mlx4_ib > > >> > > > mlx4_en > > >> > > > mlx5_core > > >> > > > mlx5_ib > > >> > > > ib_uverbs > > >> > > > ib_umad > > >> > > > ib_ucm > > >> > > > ib_cm > > >> > > > ib_core > > >> > > > mlxfw > > >> > > > mlx5_fpga_tools > > >> > > > > > >> > > > I can add an IP to ib0 using "ip addr" though I need Network > > >> > > > Manager to work with ib0. > > >> > > > > > >> > > > > > >> > > > Thanks, > > >> > > > > > >> > > > Douglas Duckworth, MSc, LFCS > > >> > > > HPC System Administrator > > >> > > > Scientific Computing Unit > > >> > > > Weill Cornell Medicine > > >> > > > 1300 York - LC-502 > > >> > > > E: doug(a)med.cornell.edu > > >> > > > O: 212-746-6305 > > >> > > > F: 212-746-8690 > > >> > > > > >> > > > > >> > > >> > > > > >

2 1

Console to HostedEngine
by Daniel Menzel 24 Aug '18

24 Aug '18

Hi at all, we cannot access our hosted engine anymore. It started with and overfull /var due to a growing database. We access the engine via SSH and tried to fix that - but somehow we seem to have produced another problem on the SSH server itself. So unfortunately we can not login anymore. We then tried to access it via its host and a "hosted-engine --console" but ran into an internal error: cannot find character device <null> which I know from KVM. With other VMs I could follow the RedHat's advice to add a console (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/ht…) but although I edited the hosted engine's profile those changes weren't applied - and after an engine restart even deleted again. That kind of makes sense to me, but limits my options. So my question is: Is there any idea how can I access the the console with my current limitations to fix the SSH server's problems and then hopefully fix everything? Regards Daniel

2 4

Re: losing ib0 connection after activating host
by Douglas Duckworth 24 Aug '18

24 Aug '18

Hi Dominik Yes, the network-script was created by our Ansible role that deploys CentOS hosts. It pulls the IP from DNS then templates the script and copies to host. I will try this oVirt step then see if it works! Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Weill Cornell Medicine 1300 York - LC-502 E: doug(a)med.cornell.edu O: 212-746-6305 F: 212-746-8690 On Thu, Aug 23, 2018 at 11:09 AM, Dominik Holler <dholler(a)redhat.com> wrote: > Is ifcfg-ib0 created before adding the host? > Can ib0 be reconfigured using engine, e.g. by > "Compute > Hosts > hostx > Network Interfaces > Setup Host Networks"? > If this some kind of self-hosted engine? > > On Thu, 23 Aug 2018 09:30:59 -0400 > Douglas Duckworth <dod2014(a)med.cornell.edu> wrote: > > > Here's a link to the files: > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__bit.ly_ > 2wjZ6Vo&d=DwICAg&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_ > 78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m=Y25-OOvgu58jlC82- > fzBeNIpQ7ZscoHznffUhqE6EBM&s=QQXlC9Tisa60TvimyS3BnFDCaDF7VP > D8eCzT-Fke-p0&e= > > > > Thank you! > > > > Thanks, > > > > Douglas Duckworth, MSc, LFCS > > HPC System Administrator > > Scientific Computing Unit > > Weill Cornell Medicine > > 1300 York - LC-502 > > E: doug(a)med.cornell.edu > > O: 212-746-6305 > > F: 212-746-8690 > > > > > > On Thu, Aug 23, 2018 at 6:51 AM, Dominik Holler <dholler(a)redhat.com> > > wrote: > > > > > Would you please share the vdsm.log and the supervdsm.log from this > > > host? > > > > > > On Wed, 22 Aug 2018 11:36:09 -0400 > > > Douglas Duckworth <dod2014(a)med.cornell.edu> wrote: > > > > > > > Hi > > > > > > > > I keep losing ib0 connection on hypervisor after adding host to > > > > engine. This makes the host not really work since NFS will be > > > > mounted over ib0. > > > > > > > > I don't really understand why this occurs. > > > > > > > > OS: > > > > > > > > [root@ovirt-hv2 ~]# cat /etc/redhat-release > > > > CentOS Linux release 7.5.1804 (Core) > > > > > > > > Here's the network script: > > > > > > > > [root@ovirt-hv2 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ib0 > > > > DEVICE=ib0 > > > > BOOTPROTO=static > > > > IPADDR=172.16.0.207 > > > > NETMASK=255.255.255.0 > > > > ONBOOT=yes > > > > ZONE=public > > > > > > > > When I try "ifup" > > > > > > > > [root@ovirt-hv2 ~]# ifup ib0 > > > > Error: Connection activation failed: No suitable device found for > > > > this connection. > > > > > > > > The error in syslog: > > > > > > > > Aug 22 11:31:50 ovirt-hv2 kernel: IPv4: martian source 172.16.0.87 > > > > from 172.16.0.49, on dev ib0 > > > > Aug 22 11:31:53 ovirt-hv2 NetworkManager[1070]: <info> > > > > [1534951913.7486] audit: op="connection-activate" > > > > uuid="2ab4abde-b8a5-6cbc-19b1-2bfb193e4e89" name="System ib0" > > > > result="fail" reason="No suitable device found for this > > > > connection. > > > > > > > > As you can see media state up: > > > > > > > > [root@ovirt-hv2 ~]# ip a > > > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state > > > > UNKNOWN group default qlen 1000 > > > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > > > inet 127.0.0.1/8 scope host lo > > > > valid_lft forever preferred_lft forever > > > > 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master > > > > ovirtmgmt state UP group default qlen 1000 > > > > link/ether 50:9a:4c:89:d3:81 brd ff:ff:ff:ff:ff:ff > > > > 3: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq > > > > state DOWN group default qlen 1000 > > > > link/ether 50:9a:4c:89:d3:82 brd ff:ff:ff:ff:ff:ff > > > > 4: p1p1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq > > > > state DOWN group default qlen 1000 > > > > link/ether b4:96:91:13:ea:68 brd ff:ff:ff:ff:ff:ff > > > > 5: p1p2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq > > > > state DOWN group default qlen 1000 > > > > link/ether b4:96:91:13:ea:6a brd ff:ff:ff:ff:ff:ff > > > > 6: idrac: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > > > > pfifo_fast state UNKNOWN group default qlen 1000 > > > > link/ether 50:9a:4c:89:d3:84 brd ff:ff:ff:ff:ff:ff > > > > inet 169.254.0.2/16 brd 169.254.255.255 scope global idrac > > > > valid_lft forever preferred_lft forever > > > > 7: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state > > > > UP group default qlen 256 > > > > link/infiniband > > > > a0:00:02:08:fe:80:00:00:00:00:00:00:ec:0d:9a:03:00:1d:13:41 brd > > > > 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff > > > > 8: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state > > > > DOWN group default qlen 1000 > > > > link/ether 12:b4:30:22:39:5b brd ff:ff:ff:ff:ff:ff > > > > 9: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN > > > > group default qlen 1000 > > > > link/ether 3e:32:e6:66:98:49 brd ff:ff:ff:ff:ff:ff > > > > 25: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > > > > noqueue state UP group default qlen 1000 > > > > link/ether 50:9a:4c:89:d3:81 brd ff:ff:ff:ff:ff:ff > > > > inet 10.0.0.183/16 brd 10.0.255.255 scope global ovirtmgmt > > > > valid_lft forever preferred_lft forever > > > > 26: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 > > > > qdisc noqueue master ovs-system state UNKNOWN group default qlen > > > > 1000 link/ether aa:32:82:1b:01:d9 brd ff:ff:ff:ff:ff:ff > > > > 27: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state > > > > DOWN group default qlen 1000 > > > > link/ether 32:ff:5d:b8:c2:b4 brd ff:ff:ff:ff:ff:ff > > > > > > > > The card is FDR: > > > > > > > > [root@ovirt-hv2 ~]# lspci -v | grep Mellanox > > > > 01:00.0 Network controller: Mellanox Technologies MT27500 Family > > > > [ConnectX-3] > > > > Subsystem: Mellanox Technologies Device 0051 > > > > > > > > Latest OFED driver: > > > > > > > > [root@ovirt-hv2 ~]# /etc/init.d/openibd status > > > > > > > > HCA driver loaded > > > > > > > > Configured IPoIB devices: > > > > ib0 > > > > > > > > Currently active IPoIB devices: > > > > ib0 > > > > Configured Mellanox EN devices: > > > > > > > > Currently active Mellanox devices: > > > > ib0 > > > > > > > > The following OFED modules are loaded: > > > > > > > > rdma_ucm > > > > rdma_cm > > > > ib_ipoib > > > > mlx4_core > > > > mlx4_ib > > > > mlx4_en > > > > mlx5_core > > > > mlx5_ib > > > > ib_uverbs > > > > ib_umad > > > > ib_ucm > > > > ib_cm > > > > ib_core > > > > mlxfw > > > > mlx5_fpga_tools > > > > > > > > I can add an IP to ib0 using "ip addr" though I need Network > > > > Manager to work with ib0. > > > > > > > > > > > > Thanks, > > > > > > > > Douglas Duckworth, MSc, LFCS > > > > HPC System Administrator > > > > Scientific Computing Unit > > > > Weill Cornell Medicine > > > > 1300 York - LC-502 > > > > E: doug(a)med.cornell.edu > > > > O: 212-746-6305 > > > > F: 212-746-8690 > > > > > > > >

2 2

Re: cannot migrate hosted engine to sandybridge hypervisor
by Douglas Duckworth 24 Aug '18

24 Aug '18

Thanks! No big deal as it's now working. On Fri, Aug 24, 2018, 9:04 AM Simone Tiraboschi <stirabos(a)redhat.com> wrote: > > > On Fri, Aug 24, 2018 at 3:06 AM Douglas Duckworth <dod2014(a)med.cornell.edu> > wrote: > >> Rebooted the hosted engine VM >> >> Migration now works >> >> Guess that's needed after changing cluster CPU type >> > > Yes, exactly: we already have an open bug about that: > https://bugzilla.redhat.com/show_bug.cgi?id=1585986 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.redhat.com_sh…> > > >> >> Thanks, >> >> Douglas Duckworth, MSc, LFCS >> HPC System Administrator >> Scientific Computing Unit >> Weill Cornell Medicine >> 1300 York - LC-502 >> E: doug(a)med.cornell.edu >> O: 212-746-6305 >> F: 212-746-8690 >> >> >> On Thu, Aug 23, 2018 at 8:46 PM, Douglas Duckworth < >> dod2014(a)med.cornell.edu> wrote: >> >>> Hi >>> >>> I am unable to migrate to hypervisor that's sandy bridge CPU. >>> >>> The cluster supports CPUs that old. >>> >>> This is the message on host when migration fails: >>> >>> Aug 23 20:40:14 lyon vdsm[9648]: WARN Worker blocked: <Worker >>> name=periodic/1 running <Task <Operation >>> action=<vdsm.virt.sampling.HostMonitor object at 0x7f2f8054b310> at >>> 0x7f2f8054b350> timeout=15, duration=15 at 0x7f2f8048d290> task#=79 at >>> 0x7f2f80540490>, traceback:#012File: "/usr/lib64/python2.7/threading.py", >>> line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: >>> "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 >>> self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in >>> run#012 self.__target(*self.__args, **self.__kwargs)#012File: >>> "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in >>> run#012 ret = func(*args, **kwargs)#012File: >>> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 >>> self._execute_task()#012File: >>> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in >>> _execute_task#012 task()#012File: >>> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in >>> __call__#012 self._callable()#012File: >>> "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 220, in >>> __call__#012 self._func()#012File: >>> "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 580, in >>> __call__#012 stats = hostapi.get_stats(self._cif, >>> self._samples.stats())#012File: >>> "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in >>> get_stats#012 ret['haStats'] = _getHaInfo()#012File: >>> "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182, in >>> _getHaInfo#012 stats = instance.get_all_stats()#012File: >>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", >>> line 94, in get_all_stats#012 stats = >>> broker.get_stats_from_storage()#012File: >>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", >>> line 135, in get_stats_from_storage#012 result = >>> self._proxy.get_stats()#012File: "/usr/lib64/python2.7/xmlrpclib.py", line >>> 1233, in __call__#012 return self.__send(self.__name, args)#012File: >>> "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request#012 >>> verbose=self.__verbose#012File: "/usr/lib64/python2.7/xmlrpclib.py", line >>> 1273, in request#012 return self.single_request(host, handler, >>> request_body, verbose)#012File: "/usr/lib64/python2.7/xmlrpclib.py", line >>> 1303, in single_request#012 response = >>> h.getresponse(buffering=True)#012File: "/usr/lib64/python2.7/httplib.py", >>> line 1113, in getresponse#012 response.begin()#012File: >>> "/usr/lib64/python2.7/httplib.py", line 444, in begin#012 version, status, >>> reason = self._read_status()#012File: "/usr/lib64/python2.7/httplib.py", >>> line 400, in _read_status#012 line = self.fp.readline(_MAXLINE + >>> 1)#012File: "/usr/lib64/python2.7/socket.py", line 476, in readline#012 >>> data = self._sock.recv(self._rbufsize) >>> Aug 23 20:43:45 lyon vdsm[9648]: WARN ping was deprecated in favor of >>> ping2 and confirmConnectivity >>> Aug 23 20:43:45 lyon vdsm[9648]: WARN Attempting to add an existing net >>> user: ovirtmgmt/adf14389-1563-4b1a-9af6-4b40370a825b >>> Aug 23 20:43:46 lyon lldpad: recvfrom(Event interface): No buffer space >>> available >>> Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered blocking >>> state >>> Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered disabled >>> state >>> Aug 23 20:43:46 lyon kernel: device vnet0 entered promiscuous mode >>> Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered blocking >>> state >>> Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered forwarding >>> state >>> Aug 23 20:43:46 lyon NetworkManager[1658]: <info> [1535071426.4326] >>> manager: (vnet0): new Tun device >>> (/org/freedesktop/NetworkManager/Devices/57) >>> Aug 23 20:43:46 lyon NetworkManager[1658]: <info> [1535071426.4336] >>> device (vnet0): state change: unmanaged -> unavailable (reason >>> 'connection-assumed', sys-iface-state: 'external') >>> Aug 23 20:43:46 lyon NetworkManager[1658]: <info> [1535071426.4341] >>> device (vnet0): state change: unavailable -> disconnected (reason 'none', >>> sys-iface-state: 'external') >>> Aug 23 20:43:46 lyon dbus[1396]: [system] Activating via systemd: >>> service name='org.freedesktop.machine1' >>> unit='dbus-org.freedesktop.machine1.service' >>> Aug 23 20:43:46 lyon systemd: Cannot add dependency job for unit >>> lvm2-lvmetad.socket, ignoring: Unit is masked. >>> Aug 23 20:43:46 lyon systemd: Starting Virtual Machine and Container >>> Registration Service... >>> Aug 23 20:43:46 lyon dbus[1396]: [system] Successfully activated service >>> 'org.freedesktop.machine1' >>> Aug 23 20:43:46 lyon systemd: Started Virtual Machine and Container >>> Registration Service. >>> Aug 23 20:43:46 lyon systemd-machined: New machine qemu-2-HostedEngine. >>> Aug 23 20:43:46 lyon systemd: Started Virtual Machine >>> qemu-2-HostedEngine. >>> Aug 23 20:43:46 lyon systemd: Starting Virtual Machine >>> qemu-2-HostedEngine. >>> Aug 23 20:43:46 lyon kvm: 1 guest now active >>> Aug 23 20:43:46 lyon libvirtd: 2018-08-24 00:43:46.940+0000: 9195: error >>> : * virCPUx86UpdateLive:2742 : operation failed: guest CPU doesn't >>> match specification: missing features: >>> fma,movbe,f16c,rdrand,fsgsbase,bmi1,hle,avx2,smep,bmi2,erms,invpcid,rtm,mpx,rdseed,adx,smap,xsavec,xgetbv1,abm,3dnowprefetch* >>> Aug 23 20:43:47 lyon kernel: ovirtmgmt: port 2(vnet0) entered disabled >>> state >>> Aug 23 20:43:47 lyon kernel: device vnet0 left promiscuous mode >>> Aug 23 20:43:47 lyon kernel: ovirtmgmt: port 2(vnet0) entered disabled >>> state >>> Aug 23 20:43:47 lyon avahi-daemon[1300]: Withdrawing workstation service >>> for vnet0. >>> Aug 23 20:43:47 lyon NetworkManager[1658]: <info> [1535071427.2753] >>> device (vnet0): state change: disconnected -> unmanaged (reason >>> 'unmanaged', sys-iface-state: 'removed') >>> Aug 23 20:43:47 lyon NetworkManager[1658]: <info> [1535071427.2753] >>> device (vnet0): released from master device ovirtmgmt >>> Aug 23 20:43:47 lyon lldpad: recvfrom(Event interface): No buffer space >>> available >>> Aug 23 20:43:47 lyon kvm: 0 guests now active >>> Aug 23 20:43:47 lyon systemd-machined: Machine qemu-2-HostedEngine >>> terminated. >>> Aug 23 20:43:47 lyon vdsm[9648]: WARN File: >>> /var/lib/libvirt/qemu/channels/adf14389-1563-4b1a-9af6-4b40370a825b.ovirt-guest-agent.0 >>> already removed >>> Aug 23 20:43:47 lyon vdsm[9648]: WARN Attempting to remove a non >>> existing network: ovirtmgmt/adf14389-1563-4b1a-9af6-4b40370a825b >>> Aug 23 20:43:47 lyon vdsm[9648]: WARN Attempting to remove a non >>> existing net user: ovirtmgmt/adf14389-1563-4b1a-9af6-4b40370a825b >>> Aug 23 20:43:47 lyon vdsm[9648]: WARN File: >>> /var/lib/libvirt/qemu/channels/adf14389-1563-4b1a-9af6-4b40370a825b.org.qemu.guest_agent.0 >>> already removed >>> Aug 23 20:43:51 lyon lldpad: setsockopt nearest_bridge: Invalid argument >>> Aug 23 20:43:52 lyon lldpad: setsockopt nearest_bridge: Invalid argument >>> >>> Can I fix this somehow? >>> >>> Thanks, >>> >>> Douglas Duckworth, MSc, LFCS >>> HPC System Administrator >>> Scientific Computing Unit >>> Weill Cornell Medicine >>> 1300 York - LC-502 >>> E: doug(a)med.cornell.edu >>> O: 212-746-6305 >>> F: 212-746-8690 >>> >>> >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_site_pri…> >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_communit…> >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/FBENIU2TRZSA2… >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ovirt.org_archiv…> >> >

1 0

cannot migrate hosted engine to sandybridge hypervisor
by Douglas Duckworth 24 Aug '18

24 Aug '18

Hi I am unable to migrate to hypervisor that's sandy bridge CPU. The cluster supports CPUs that old. This is the message on host when migration fails: Aug 23 20:40:14 lyon vdsm[9648]: WARN Worker blocked: <Worker name=periodic/1 running <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x7f2f8054b310> at 0x7f2f8054b350> timeout=15, duration=15 at 0x7f2f8048d290> task#=79 at 0x7f2f80540490>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 220, in __call__#012 self._func()#012File: "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 580, in __call__#012 stats = hostapi.get_stats(self._cif, self._samples.stats())#012File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in get_stats#012 ret['haStats'] = _getHaInfo()#012File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182, in _getHaInfo#012 stats = instance.get_all_stats()#012File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 94, in get_all_stats#012 stats = broker.get_stats_from_storage()#012File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 135, in get_stats_from_storage#012 result = self._proxy.get_stats()#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__#012 return self.__send(self.__name, args)#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request#012 verbose=self.__verbose#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request#012 return self.single_request(host, handler, request_body, verbose)#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request#012 response = h.getresponse(buffering=True)#012File: "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse#012 response.begin()#012File: "/usr/lib64/python2.7/httplib.py", line 444, in begin#012 version, status, reason = self._read_status()#012File: "/usr/lib64/python2.7/httplib.py", line 400, in _read_status#012 line = self.fp.readline(_MAXLINE + 1)#012File: "/usr/lib64/python2.7/socket.py", line 476, in readline#012 data = self._sock.recv(self._rbufsize) Aug 23 20:43:45 lyon vdsm[9648]: WARN ping was deprecated in favor of ping2 and confirmConnectivity Aug 23 20:43:45 lyon vdsm[9648]: WARN Attempting to add an existing net user: ovirtmgmt/adf14389-1563-4b1a-9af6-4b40370a825b Aug 23 20:43:46 lyon lldpad: recvfrom(Event interface): No buffer space available Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered blocking state Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered disabled state Aug 23 20:43:46 lyon kernel: device vnet0 entered promiscuous mode Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered blocking state Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered forwarding state Aug 23 20:43:46 lyon NetworkManager[1658]: <info> [1535071426.4326] manager: (vnet0): new Tun device (/org/freedesktop/NetworkManager/Devices/57) Aug 23 20:43:46 lyon NetworkManager[1658]: <info> [1535071426.4336] device (vnet0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external') Aug 23 20:43:46 lyon NetworkManager[1658]: <info> [1535071426.4341] device (vnet0): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'external') Aug 23 20:43:46 lyon dbus[1396]: [system] Activating via systemd: service name='org.freedesktop.machine1' unit='dbus-org.freedesktop.machine1.service' Aug 23 20:43:46 lyon systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 23 20:43:46 lyon systemd: Starting Virtual Machine and Container Registration Service... Aug 23 20:43:46 lyon dbus[1396]: [system] Successfully activated service 'org.freedesktop.machine1' Aug 23 20:43:46 lyon systemd: Started Virtual Machine and Container Registration Service. Aug 23 20:43:46 lyon systemd-machined: New machine qemu-2-HostedEngine. Aug 23 20:43:46 lyon systemd: Started Virtual Machine qemu-2-HostedEngine. Aug 23 20:43:46 lyon systemd: Starting Virtual Machine qemu-2-HostedEngine. Aug 23 20:43:46 lyon kvm: 1 guest now active Aug 23 20:43:46 lyon libvirtd: 2018-08-24 00:43:46.940+0000: 9195: error : *virCPUx86UpdateLive:2742 : operation failed: guest CPU doesn't match specification: missing features: fma,movbe,f16c,rdrand,fsgsbase,bmi1,hle,avx2,smep,bmi2,erms,invpcid,rtm,mpx,rdseed,adx,smap,xsavec,xgetbv1,abm,3dnowprefetch* Aug 23 20:43:47 lyon kernel: ovirtmgmt: port 2(vnet0) entered disabled state Aug 23 20:43:47 lyon kernel: device vnet0 left promiscuous mode Aug 23 20:43:47 lyon kernel: ovirtmgmt: port 2(vnet0) entered disabled state Aug 23 20:43:47 lyon avahi-daemon[1300]: Withdrawing workstation service for vnet0. Aug 23 20:43:47 lyon NetworkManager[1658]: <info> [1535071427.2753] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Aug 23 20:43:47 lyon NetworkManager[1658]: <info> [1535071427.2753] device (vnet0): released from master device ovirtmgmt Aug 23 20:43:47 lyon lldpad: recvfrom(Event interface): No buffer space available Aug 23 20:43:47 lyon kvm: 0 guests now active Aug 23 20:43:47 lyon systemd-machined: Machine qemu-2-HostedEngine terminated. Aug 23 20:43:47 lyon vdsm[9648]: WARN File: /var/lib/libvirt/qemu/channels/adf14389-1563-4b1a-9af6-4b40370a825b.ovirt-guest-agent.0 already removed Aug 23 20:43:47 lyon vdsm[9648]: WARN Attempting to remove a non existing network: ovirtmgmt/adf14389-1563-4b1a-9af6-4b40370a825b Aug 23 20:43:47 lyon vdsm[9648]: WARN Attempting to remove a non existing net user: ovirtmgmt/adf14389-1563-4b1a-9af6-4b40370a825b Aug 23 20:43:47 lyon vdsm[9648]: WARN File: /var/lib/libvirt/qemu/channels/adf14389-1563-4b1a-9af6-4b40370a825b.org.qemu.guest_agent.0 already removed Aug 23 20:43:51 lyon lldpad: setsockopt nearest_bridge: Invalid argument Aug 23 20:43:52 lyon lldpad: setsockopt nearest_bridge: Invalid argument Can I fix this somehow? Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Weill Cornell Medicine 1300 York - LC-502 E: doug(a)med.cornell.edu O: 212-746-6305 F: 212-746-8690

2 2

Upgrading librbd1 and librados2 for oVirt 4.2.x
by andreas.elvers+ovirtforum＠solutions.work 24 Aug '18

24 Aug '18

Hallo Group ! I am in need of upgrading librbd1 and librados2 for a oVirt 4.2.x cluster. The cluster was installed via node ng. Taking the repository for Ceph Mimic or Luminous will end in a dependency problem because liburcu-cds.so.1 is already installed as a more recent version provided by the oVirt repository. A hint on how to get a more recent versions of the above libraries would be very much appreciated. Thanks, - Andreas

1 1

HA VMs not starting automatically on another host after host crash.
by Eduardo Mayoral 24 Aug '18

24 Aug '18

Hi, Recently I have had 2 oVirt hosts (oVirt 4.2, CentOS 7.5) crash unexpectedly (not at the same time). Both seem hardware related. In both cases oVirt did detect the host as non responsive, did a fence on the hosts and set the VMs which were running on the host at the time as "Down". So far, so good. The hosts were on 2 different clusters. Both had Migration policy / resilience policy set to "migrate Virtual Machines". The VMs are not configured as "Highly available", but my understanding is that with that resiliency policy, this makes no difference. My (probably wrong) expectation, is that oVirt should have attempted to start the VMs in another host. It did not. A colleague started the VMs manually and they started with no further issues. So. How should I configure oVirt to try to start the VMs on another host when the physical host they are running in crashes? Thanks in advance! -- Eduardo Mayoral.

2 2

VM has been paused due to unknown storage error. - Only on NFS / EMC
by jeanbaptiste＠nfrance.com 24 Aug '18

24 Aug '18

Hello, I'm facing a strange issue on my OVirt Dev pool. Indeed, when I create high disk load a VM (kickstart installation or iozone test for example) , VM is paused due to storage I/O error. Problem is 100% reproducible, and is located only on NFS (v3 and v4) on my EMC VNXe3200 NAS's (I have a 10TB and a 20TB NAS) I have done test (simple iozone -a) with VM 1 vCPU / 2GB RAM and 2 disk (1*20GB + 1*10GB). Both VMs disk are placed into the same SAN / NAS for each test. Results are: - EMC VNXe3200 (10TB) NFSv3 => VM stopped after 10- 30s iozone lauch - EMC VNXe3200 (20TB) NFSv3 => VM stopped after 10- 30s iozone lauch - EMC VNXe3200 (10TB) ISCSI => No problem, iozone test finish, and performance are "standard" regarding load of the VNXe (60MB/s sequential Write for info) - EMC VNXe3200 (20TB) ISCSI=> No problem, iozone test finish, and performance are "standard"regarding load of the VNXe (40-60MB/S sequential Write for info) - NETAPP FAS2240 NFSv3 => No problem, iozone test finish, and performance are good (100MB/s sequential Write for info) - Freebsd10 NAS NFSv3 => No problem, iozone test finish, and performance are good regarding NAS conf (80MB/s sequential Write for info) I can't explain why I have an issue on NFS and I have not issue on ISCSI (on the same EMC VNxe3200...). NFS default params are keeped when storage added to datacenter:(rw,relatime,vers=4.0,rsize=131072,wsize=131072,namlen=255,soft,nosharecache,proto=tcp,port=0,timeo=600,retrans=6,sec=sys,clientaddr=XXXXXXX,local_lock=none,addr=XXXXXX) (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,soft,nolock,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,mountaddr=XXXXXX,mountvers=3,mountport=1234,mountproto=udp,local_lock=all,addr=XXXXXX) Debug logs on host does not help me a lot: 2018-08-22 15:36:13,883+0200 INFO (periodic/22) [vdsm.api] START multipath_health() from=internal, task_id=53da2eca-eb66-400c-8367-ab62cedc5dc1 (api:46) 2018-08-22 15:36:13,883+0200 INFO (periodic/22) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=53da2eca-eb66-400c-8367-ab62cedc5dc1 (api:52) 2018-08-22 15:36:15,161+0200 INFO (libvirt/events) [virt.vm] (vmId='b139a9b9-16bc-40ee-ba84-d1d59e5ce17a') abnormal vm stop device ua-179375b0-0a18-4fcb-a884-4aeb1c 8fed97 error eother (vm:5116) 2018-08-22 15:36:15,161+0200 INFO (libvirt/events) [virt.vm] (vmId='b139a9b9-16bc-40ee-ba84-d1d59e5ce17a') CPU stopped: onIOError (vm:6157) 2018-08-22 15:36:15,162+0200 DEBUG (libvirt/events) [virt.metadata.Descriptor] values: {'minGuaranteedMemoryMb': 1024, 'clusterVersion': '4.2', 'resumeBehavior': 'au to_resume', 'memGuaranteedSize': 1024, 'launchPaused': 'false', 'startTime': 1534944832.058459, 'destroy_on_reboot': False, 'pauseTime': 4999289.49} (metadata:596) 2018-08-22 15:36:15,162+0200 DEBUG (libvirt/events) [virt.metadata.Descriptor] values updated: {'minGuaranteedMemoryMb': 1024, 'clusterVersion': '4.2', 'resumeBehavi or': 'auto_resume', 'memGuaranteedSize': 1024, 'launchPaused': 'false', 'startTime': 1534944832.058459, 'destroy_on_reboot': False, 'pauseTime': 4999289.49} (metadat a:601) 2018-08-22 15:36:15,168+0200 DEBUG (libvirt/events) [virt.metadata.Descriptor] dumped metadata for b139a9b9-16bc-40ee-ba84-d1d59e5ce17a: <?xml version='1.0' encoding metadata blablablabla............................> 2018-08-22 15:36:15,169+0200 DEBUG (libvirt/events) [virt.vm] (vmId='b139a9b9-16bc-40ee-ba84-d1d59e5ce17a') event Suspended detail 2 opaque None (vm:5520) 2018-08-22 15:36:15,169+0200 INFO (libvirt/events) [virt.vm] (vmId='b139a9b9-16bc-40ee-ba84-d1d59e5ce17a') CPU stopped: onSuspend (vm:6157) 2018-08-22 15:36:15,174+0200 WARN (libvirt/events) [virt.vm] (vmId='b139a9b9-16bc-40ee-ba84-d1d59e5ce17a') device sda reported I/O error (vm:4065) 2018-08-22 15:36:15,340+0200 DEBUG (vmchannels) [virt.vm] (vmId='46d496af-e2d0-4caa-9a13-10c624f265d8') Guest's message heartbeat: {u'memory-stat': {u'swap_out': 0, u'majflt': 0, u'swap_usage': 0, u'mem_cached': 119020, u'mem_free': 3693900, u'mem_buffers': 2108, u'swap_in': 0, u'swap_total': 8257532, u'pageflt': 141, u'mem_tota l': 3878980, u'mem_unused': 3572772}, u'free-ram': u'3607', u'apiVersion': 3} (guestagent:337) Do you have some idea ?

2 2

oVirt: powerful opensource virtualization
by Sandro Bonazzola 24 Aug '18

24 Aug '18

Hi, just to let you know that Simone Tiraboschi will present "oVirt: powerful opensource virtualization" in Bergamo (Italy) on September 8th 2018 within Download Innovation IT & Conference event (https://download-event.io/en/) -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=701f…>

3 2

Update disk alias by using ovirtsdk4
by branimirp＠gmail.com 24 Aug '18

24 Aug '18

Hello list! I have been trying to rename a disk alias that defaults to "some_template_name-Disk1" to something more logical by using ovirtsdk4 (in my example to "vm2_disk_0"). On the list itself, I saw a couple of examples for older version of SDK. Basically, I am trying to create a VM and then update its properties, notably "alias" attribute that can be found in types.Disk. The first portion of the code works well (vm provisioning), but second one (alias update) does not, causing the TypeError exception. Obviously, I am doing something wrong here. I have spent most of the time trying to figure out how to actually do this. I would kindly ask for some hint. Thank you in advance! Regards, Branimir ******************************* # create a vm vms_service = connection.system_service().vms_service() vms_service.add( types.Vm( # --- vm name name='vm2', # --- vCpus cpu=types.Cpu( topology=types.CpuTopology(cores=2, sockets=1) ), # --- memory memory = 512*1024*1024, # -- oVirt cluster cluster = types.Cluster( name='cluster_1', ), # -- template to use template = types.Template( name='centos7-tmplt-compute-node-v1'), # -- boot device os = types.OperatingSystem(boot=types.Boot(devices=[types.BootDevice.HD])), ), ) # update the vm in terms of disk name vm = vms_service.list(search='name=vm2')[0] vm_service = vms_service.vm_service(vm.id) updated_vm = vm_service.update( types.Vm( disk_attachment = types.Disk(alias='vm2_disk_0') ) )

2 2

Upgrading librbd1 and librados2 for oVirt 4.2.x
by Andreas Elvers 23 Aug '18

23 Aug '18

Hello, I am in need of upgrading librbd1 and librados2 for a oVirt 4.2.x cluster. The cluster was installed via node ng. Taking the repository for Ceph Mimic or Luminous will end with a dependency problem because liburcu-cds.so.1 is already installed as a more recent version. A hint on how to get a more recent versions of the above libraries would be very much appreciated. Thanks, - Andreas

1 0

Re: losing ib0 connection after activating host
by Douglas Duckworth 23 Aug '18

23 Aug '18

Here's a link to the files: https://bit.ly/2wjZ6Vo Thank you! Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Weill Cornell Medicine 1300 York - LC-502 E: doug(a)med.cornell.edu O: 212-746-6305 F: 212-746-8690 On Thu, Aug 23, 2018 at 6:51 AM, Dominik Holler <dholler(a)redhat.com> wrote: > Would you please share the vdsm.log and the supervdsm.log from this > host? > > On Wed, 22 Aug 2018 11:36:09 -0400 > Douglas Duckworth <dod2014(a)med.cornell.edu> wrote: > > > Hi > > > > I keep losing ib0 connection on hypervisor after adding host to > > engine. This makes the host not really work since NFS will be mounted > > over ib0. > > > > I don't really understand why this occurs. > > > > OS: > > > > [root@ovirt-hv2 ~]# cat /etc/redhat-release > > CentOS Linux release 7.5.1804 (Core) > > > > Here's the network script: > > > > [root@ovirt-hv2 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ib0 > > DEVICE=ib0 > > BOOTPROTO=static > > IPADDR=172.16.0.207 > > NETMASK=255.255.255.0 > > ONBOOT=yes > > ZONE=public > > > > When I try "ifup" > > > > [root@ovirt-hv2 ~]# ifup ib0 > > Error: Connection activation failed: No suitable device found for this > > connection. > > > > The error in syslog: > > > > Aug 22 11:31:50 ovirt-hv2 kernel: IPv4: martian source 172.16.0.87 > > from 172.16.0.49, on dev ib0 > > Aug 22 11:31:53 ovirt-hv2 NetworkManager[1070]: <info> > > [1534951913.7486] audit: op="connection-activate" > > uuid="2ab4abde-b8a5-6cbc-19b1-2bfb193e4e89" name="System ib0" > > result="fail" reason="No suitable device found for this connection. > > > > As you can see media state up: > > > > [root@ovirt-hv2 ~]# ip a > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN > > group default qlen 1000 > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > inet 127.0.0.1/8 scope host lo > > valid_lft forever preferred_lft forever > > 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master > > ovirtmgmt state UP group default qlen 1000 > > link/ether 50:9a:4c:89:d3:81 brd ff:ff:ff:ff:ff:ff > > 3: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state > > DOWN group default qlen 1000 > > link/ether 50:9a:4c:89:d3:82 brd ff:ff:ff:ff:ff:ff > > 4: p1p1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state > > DOWN group default qlen 1000 > > link/ether b4:96:91:13:ea:68 brd ff:ff:ff:ff:ff:ff > > 5: p1p2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state > > DOWN group default qlen 1000 > > link/ether b4:96:91:13:ea:6a brd ff:ff:ff:ff:ff:ff > > 6: idrac: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast > > state UNKNOWN group default qlen 1000 > > link/ether 50:9a:4c:89:d3:84 brd ff:ff:ff:ff:ff:ff > > inet 169.254.0.2/16 brd 169.254.255.255 scope global idrac > > valid_lft forever preferred_lft forever > > 7: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP > > group default qlen 256 > > link/infiniband > > a0:00:02:08:fe:80:00:00:00:00:00:00:ec:0d:9a:03:00:1d:13:41 brd > > 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff > > 8: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN > > group default qlen 1000 > > link/ether 12:b4:30:22:39:5b brd ff:ff:ff:ff:ff:ff > > 9: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group > > default qlen 1000 > > link/ether 3e:32:e6:66:98:49 brd ff:ff:ff:ff:ff:ff > > 25: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > > noqueue state UP group default qlen 1000 > > link/ether 50:9a:4c:89:d3:81 brd ff:ff:ff:ff:ff:ff > > inet 10.0.0.183/16 brd 10.0.255.255 scope global ovirtmgmt > > valid_lft forever preferred_lft forever > > 26: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc > > noqueue master ovs-system state UNKNOWN group default qlen 1000 > > link/ether aa:32:82:1b:01:d9 brd ff:ff:ff:ff:ff:ff > > 27: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN > > group default qlen 1000 > > link/ether 32:ff:5d:b8:c2:b4 brd ff:ff:ff:ff:ff:ff > > > > The card is FDR: > > > > [root@ovirt-hv2 ~]# lspci -v | grep Mellanox > > 01:00.0 Network controller: Mellanox Technologies MT27500 Family > > [ConnectX-3] > > Subsystem: Mellanox Technologies Device 0051 > > > > Latest OFED driver: > > > > [root@ovirt-hv2 ~]# /etc/init.d/openibd status > > > > HCA driver loaded > > > > Configured IPoIB devices: > > ib0 > > > > Currently active IPoIB devices: > > ib0 > > Configured Mellanox EN devices: > > > > Currently active Mellanox devices: > > ib0 > > > > The following OFED modules are loaded: > > > > rdma_ucm > > rdma_cm > > ib_ipoib > > mlx4_core > > mlx4_ib > > mlx4_en > > mlx5_core > > mlx5_ib > > ib_uverbs > > ib_umad > > ib_ucm > > ib_cm > > ib_core > > mlxfw > > mlx5_fpga_tools > > > > I can add an IP to ib0 using "ip addr" though I need Network Manager > > to work with ib0. > > > > > > Thanks, > > > > Douglas Duckworth, MSc, LFCS > > HPC System Administrator > > Scientific Computing Unit > > Weill Cornell Medicine > > 1300 York - LC-502 > > E: doug(a)med.cornell.edu > > O: 212-746-6305 > > F: 212-746-8690 > >

2 1

Data Center non-responsive, storage not loading, and psql errors after importing an ova
by josh＠nightscapetech.com 23 Aug '18

23 Aug '18

I posted about this a few months ago on Reddit when it happened the first time, and I have now repeated it on two more separate installations. I have a simple oVirt 4.2 hosted-engine setup, two hosts, with a qnap NAS as the shared NFS storage. I copied an ova file (exported from vmware) to the NFS storage. I then imported it by logging into the engine gui, going to "Virtual machines", selecting the host/file path/datacenter/etc, and starting the import. Once I started the import, I get an alert that it failed almost immediately, followed by a notice that the data center is in a non-responsive state. Clicking on the "storage" tab under Data Center, or going to the "Storage Domain" page directly yields the three ". . ." loading animation, which never ends. I can still see the storage mounts on the hosts, and I can move files to and from them. the engine.log file on the hosted-engine VM contains a lot of the following lines: Caused by: org.springframework.dao.DataIntegrityViolationException: PreparedStatementCallback; SQL [select * from getstorage_domains_list_by_imageid (?)]; ERROR: integer out of range Where: PL/pgSQL function getstorage_domains_list_by_imageid(uuid) line 3 at RETURN QUERY; nested exception is org.postgresql.util.PSQLException: ER ROR: integer out of range Where: PL/pgSQL function getstorage_domains_list_by_imageid(uuid) line 3 at RETURN QUERY at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:102) [spring-jdbc.jar:4.3. 9.RELEASE] What I have tried: Restarting the hosted-engine service Restarting nfs and vdsm services Rebooting the hosted-engine Rebooting both hosts Entering an exiting maintenance mode (one host never came out of maintenance, and refuses to with the error " General command validation failure ") Any idea what might have happened? And what should I have done to try to rectify it? Entering maintenance mode and restarting services seems to have made things much worse.

3 3