August 2019 - Users - oVirt List Archives

ovirt api snapshot date format?
by Bill James 28 Aug '19

28 Aug '19

I'm trying to use the ovirt REST api to read snapshot date but the number I'm getting back doesn't appear to be in epoch time. What format is it? Example: "date":1565299725928,"snapshot_status":"ok" Should be "Aug 8, 2019", epoch conversion put it at Tue May 16 16:25:28 51572. (accessed via https://$HOSTNAME/ovirt-engine/api/vms/$value/snapshots)

2 1

Removing Unknown VM's on Dead Cluster
by Ross, Travis 28 Aug '19

28 Aug '19

I've clearly seen this is a pretty common issue with many different methods for cleaning up dead clusters. I'm having issues with unknown VM's not allowing my host into Maintenance Mode. Version: ovirt-release36-3.6.7-1.noarch Situation: Another site repurposed a host entirely without properly cleaning it's status in my Engine. No option to delete host. I've attempted the "Confirm Host is Rebooted" -> Maintenance Mode. However the error is: Cannot Switch the following Hosts to Maintenance mode: ... One or more running VMs are indicated as non-migratable. The status of all the listed VM's is "UNKNOWN" w/ "?" next to them. If I look at the settings for these VM's they already have "Do not allow migration" checked. Also I'm unable to modify the VM since the Host doesn't exist. Can't just delete the Domain because the Host exists. Can't delete Storage because host is not in Maintenance Mode. Can't set Maintenance Mode because VM's can't migrate. At this point all I want to do is upgrade to latest ovirt, however I need to get this orphaned Domain out of my single node cluster, which still runs a ~10 VM's hosted on Gluster Storage. Thoughts on whether I could just backup these VM's, re-install ovirt fresh then import the VM's? Thanks, Travis Ross W 512.231.4235 M 512.547.9437 [cid:image003.png@01D55D96.2A607620] KLA-Tencor is now KLA! Learn more at kla.com <https://www.kla.com/>

1 0

Update single node environment from 4.3.3 to 4.3.5 problem
by Gianluca Cecchi 28 Aug '19

28 Aug '19

Hello, after updating hosted engine from 4.3.3 to 4.3.5 and then the only host composing the environment (plain CentOS 7.6) it seems it is not able to start vdsm daemons kernel installed with update is kernel-3.10.0-957.27.2.el7.x86_64 Same problem also if using previous running kernel 3.10.0-957.12.2.el7.x86_64 [root@ovirt01 vdsm]# uptime 00:50:08 up 25 min, 3 users, load average: 0.60, 0.67, 0.60 [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# systemctl status vdsmd -l ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/etc/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: failed (Result: start-limit) since Fri 2019-08-23 00:37:27 CEST; 7s ago Process: 25810 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=1/FAILURE) Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Failed to start Virtual Desktop Server Manager. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Unit vdsmd.service entered failed state. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: vdsmd.service failed. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: vdsmd.service holdoff time over, scheduling restart. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Stopped Virtual Desktop Server Manager. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: start request repeated too quickly for vdsmd.service Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Failed to start Virtual Desktop Server Manager. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Unit vdsmd.service entered failed state. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: vdsmd.service failed. [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# pwd /var/log/vdsm [root@ovirt01 vdsm]# ll -t | head total 118972 -rw-r--r--. 1 root root 3406465 Aug 23 00:25 supervdsm.log -rw-r--r--. 1 root root 73621 Aug 23 00:25 upgrade.log -rw-r--r--. 1 vdsm kvm 0 Aug 23 00:01 vdsm.log -rw-r--r--. 1 vdsm kvm 538480 Aug 22 23:46 vdsm.log.1.xz -rw-r--r--. 1 vdsm kvm 187486 Aug 22 23:46 mom.log -rw-r--r--. 1 vdsm kvm 621320 Aug 22 22:01 vdsm.log.2.xz -rw-r--r--. 1 root root 374464 Aug 22 22:00 supervdsm.log.1.xz -rw-r--r--. 1 vdsm kvm 2097122 Aug 22 21:53 mom.log.1 -rw-r--r--. 1 vdsm kvm 636212 Aug 22 20:01 vdsm.log.3.xz [root@ovirt01 vdsm]# link to upgrade.log contents here: https://drive.google.com/file/d/17jtX36oH1hlbNUAiVhdBkVDbd28QegXG/view?usp=… link to supervdsm.log (in gzip format) here: https://drive.google.com/file/d/1l61ePU-eFS_xVHEAHnJthzTTnTyzu0MP/view?usp=… It seems that since update I get these kind of lines inside it... restore-net::DEBUG::2019-08-22 23:56:38,591::cmdutils::133::root::(exec_cmd) /sbin/tc filter del dev eth0 pref 5000 (cwd None) restore-net::DEBUG::2019-08-22 23:56:38,595::cmdutils::141::root::(exec_cmd) FAILED: <err> = 'RTNETLINK answers: Invalid argument\nWe have an error talking to the kernel\n'; <rc> = 2 [root@ovirt01 vdsm]# systemctl status supervdsmd -l ● supervdsmd.service - Auxiliary vdsm service for running helper functions as root Loaded: loaded (/usr/lib/systemd/system/supervdsmd.service; static; vendor preset: enabled) Active: active (running) since Fri 2019-08-23 00:25:17 CEST; 23min ago Main PID: 4540 (supervdsmd) Tasks: 3 CGroup: /system.slice/supervdsmd.service └─4540 /usr/bin/python2 /usr/share/vdsm/supervdsmd --sockfile /var/run/vdsm/svdsm.sock Aug 23 00:25:17 ovirt01.mydomain systemd[1]: Started Auxiliary vdsm service for running helper functions as root. [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovirtmgmt state UP group default qlen 1000 link/ether b8:ae:ed:7f:17:11 brd ff:ff:ff:ff:ff:ff 3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:c2:c6:a4:18:c5 brd ff:ff:ff:ff:ff:ff 4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 36:21:c1:5e:70:aa brd ff:ff:ff:ff:ff:ff 5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 46:d8:db:81:41:4e brd ff:ff:ff:ff:ff:ff 22: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:ae:ed:7f:17:11 brd ff:ff:ff:ff:ff:ff inet 192.168.1.211/24 brd 192.168.1.255 scope global ovirtmgmt valid_lft forever preferred_lft forever [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# ip route show default via 192.168.1.1 dev ovirtmgmt 192.168.1.0/24 dev ovirtmgmt proto kernel scope link src 192.168.1.211 [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# brctl show bridge name bridge id STP enabled interfaces ovirtmgmt 8000.b8aeed7f1711 no eth0 [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# systemctl status openvswitch ● openvswitch.service - Open vSwitch Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; enabled; vendor preset: disabled) Active: active (exited) since Fri 2019-08-23 00:25:09 CEST; 26min ago Process: 3894 ExecStart=/bin/true (code=exited, status=0/SUCCESS) Main PID: 3894 (code=exited, status=0/SUCCESS) Tasks: 0 CGroup: /system.slice/openvswitch.service Aug 23 00:25:09 ovirt01.mydomain systemd[1]: Starting Open vSwitch... Aug 23 00:25:09 ovirt01.mydomain systemd[1]: Started Open vSwitch. [root@ovirt01 vdsm]# ovs-vsctl show 02539902-1788-4796-9cdf-cf11ce8436bb Bridge br-int fail_mode: secure Port br-int Interface br-int type: internal ovs_version: "2.11.0" [root@ovirt01 vdsm]# any hints? Thanks Gianluca

4 16

Evaluate oVirt to remove VMware
by dsalade＠gmail.com 27 Aug '19

27 Aug '19

Howdy All!!! Somewhat new user to Linux and having some issues with oVirt node networking. We are mainly a Windows/VMware medium shop but have a pretty good network. Our VMware setup has redundant switches using multiple VLANs. This was fairly easy to setup with VMware. We never had an issue with VMware, add 2 NICs to the ESXi host and then adding VLANs into the Virtual Switch. Biggest issue I am having, should I be bonding, teaming and VLANing on top of that? Tried setting up node with Engine inside which fails to deploy because of networking. Then I tried a standalone Engine and could not add node to it - I believe networking as well as Power Management agent was the problem. Would like the nodes/engine to be on one subnet (lets say 10.10.11.x) and VMs to be allowed to run there are well as 10 other VLANs. I guess welcome to the wonderful world of Linux networking and learning some new technologies is in store for me. I hope I am not being vague, like I said fairly new to Linux. Thanks for any responses! Allen

4 6

VDSM Hooks during migration
by Vrgotic, Marko 27 Aug '19

27 Aug '19

Dear oVIrt, Would you be so kind to help me/tell me or point me how to find which Hooks, and in which order, are triggered when VM is being migrated? Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic

2 6

Re: VM --- is not responding.
by Strahil 26 Aug '19

26 Aug '19

Hm... It was supposes to show controller status. Maybe the hpssacli you have is not supporting your raid cards.Check for newer version on HPE's support page. Best Regards, Strahil Nikolov Best Regards, Strahil NikolovOn Aug 14, 2019 11:40, Edoardo Mazza <edo7411(a)gmail.com> wrote: > > I installed hpssacli-2.40-13.0.x86_64.rpm and the result of "hpssacli ctrl all show status" is: > Error: No controllers detected. Possible causes:..... > The s.o. run on sd cards and the vm runs on array on traditional disk > thanks > Edoardo > > Il giorno lun 12 ago 2019 alle ore 05:59 Strahil <hunter86_bg(a)yahoo.com> ha scritto: >> >> Would you check the health status of the controllers : >> hpssacli ctrl all show status >> >> Best Regards, >> Strahil Nikolov >> >> On Aug 11, 2019 09:55, Edoardo Mazza <edo7411(a)gmail.com> wrote: >>> >>> The hosts are 3 ProLiant DL380 Gen10, 2 hosts with HPE Smart Array P816i-a SR Gen10 like controller and the other host with >>> HPE Smart Array P408i-a SR Gen10. The storage for ovirt enviroment is gluster and the last host is the arbiter in the gluster enviroment. >>> The S.M.A.R.T. healt status is ok for all host >>> Edoardo >>> >>> >>> >>> >>> >>> Il giorno gio 8 ago 2019 alle ore 16:19 Sandro Bonazzola <sbonazzo(a)redhat.com> ha scritto: >>>> >>>> >>>> >>>> Il giorno gio 8 ago 2019 alle ore 11:19 Edoardo Mazza <edo7411(a)gmail.com> ha scritto: >>>>> >>>>> Hi all, >>>>> It is more days that for same vm I received this error, but I don't underdand why. >>>>> The traffic of the virtual machine is not excessive, cpu and ram to, but for few minutes the vm is not responding. and in the messages log file of the vm I received the error under, yo can help me? >>>>> thanks >>>> >>>> >>>> can you check the S.M.A.R.T. health status of the disks? >>>> >>>> >>>>> >>>>> Edoardo >>>>> kernel: NMI watchdog: BUG: soft lockup - CPU#2 stuck for 25s! [kworker/2:0:26227] >>>>> Aug 8 02:51:11 vmmysql kernel: Modules linked in: binfmt_misc ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ >>>>> ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_ >>>>> nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_con >>>>> ntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter snd_hda_c >>>>> odec_generic iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel snd_hda_intel snd_hda_codec aesni_intel snd_hda_core lrw gf128mul >>>>> glue_helper ablk_helper snd_hwdep cryptd snd_seq snd_seq_device snd_pcm snd_timer snd soundcore virtio_rng sg virtio_balloon >>>>> i2c_piix4 parport_pc parport joydev pcspkr ip_tables xfs libcrc32c sd_mod >>>>> Aug 8 02:51:14 vmmysql kernel: crc_t10dif crct10dif_generic sr_mod cdrom virtio_net virtio_console virtio_scsi ata_generic p >>>>> ata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw qxl floppy drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci drm_panel_orientation_quirks virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod >>>>> Aug 8 02:51:14 vmmysql kernel: CPU: 2 PID: 26227 Comm: kworker/2:0 Kdump: loaded Tainted: G L ------------ 3.10.0-957.12.1.el7.x86_64 #1 >>>>> Aug 8 02:51:14 vmmysql kernel: Hardware name: oVirt oVirt Node, BIOS 1.11.0-2.el7 04/01/2014 >>>>> Aug 8 02:51:14 vmmysql kernel: Workqueue: events_freezable disk_events_workfn >>>>> Aug 8 02:51:14 vmmysql kernel: task: ffff9e25b6609040 ti: ffff9e27b1610000 task.ti: ffff9e27b1610000 >>>>> Aug 8 02:51:14 vmmysql kernel: RIP: 00

2 1

Re: Need to enable STP on ovirt bridges
by Strahil 25 Aug '19

25 Aug '19

It seems that according to https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.0/ht… the ports of interest are: 16514/TCP 49152 - 49216/TCP Best Regards, Strahil NikolovOn Aug 25, 2019 08:48, Strahil <hunter86_bg(a)yahoo.com> wrote: > > Curtis, > > Do you have enough space to run tcpdump (port not 22) on both hosts and on the small VM you have done previously - and then start the migration? > > Best Regards, > Strahil NikolovOn Aug 24, 2019 22:15, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > > > I applied a 90Mbs QOS Rate Limit with 10 set for the shares to both > > interfaces of 2 of the hosts. My hosts names are swm-01 and swm-02. > > > > Creating a small VM from a Cinder template and running it gave me a test VM. > > > > When I migrated from swm-01 to swm-02, swm-01 immediately became > > unresponsive to pings, SSH'es, and to the ovirt interface which marked > > it as "NonResponsive" soon after the VM finished. The VM did finish > > migrating, however I'm unsure if that's a good migration or not. > > > > Thank you, Strahil. > > > > On Sat, Aug 24, 2019 at 12:39 PM Strahil <hunter86_bg(a)yahoo.com> wrote: > > > > > > What is your bandwidth threshold for the network used for VM migration ? > > > Can you set a 90 mbit/s threshold (yes, less than 100mbit/s) and try to migrate a small (1 GB RAM) VM ? > > > > > > Do you see disconnects ? > > > > > > If no, try a little bit up (the threshold) and check again. > > > > > > Best Regards, > > > Strahil NikolovOn Aug 23, 2019 23:19, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > > > > > > > It took a while for my servers to come back on the network this time. > > > > I think it's due to ovirt continuing to try to migrate the VMs around > > > > like I requested. The 3 servers' names are "swm-01, swm-02 and > > > > swm-03". Eventually (about 2-3 minutes ago) they all came back online. > > > > > > > > So I disabled and stopped the lldpad service. > > > > > > > > Nope. Started some more migrations and swm-02 and swm-03 disappeared > > > > again. No ping, SSH hung, same as before - almost as soon as the > > > > migration started. > > > > > > > > If you wall have any ideas what switch-level setting might be enabled, > > > > let me know, cause I'm stumped. I can add it to the ticket that's > > > > requesting the port configurations. I've already added the port > > > > numbers and switch name that I got from CDP. > > > > > > > > Thanks again, I really appreciate the help! > > > > cecjr > > > > > > > > > > > > > > > > On Fri, Aug 23, 2019 at 3:28 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > > > > > > > > > > > > > > > > > On Fri, Aug 23, 2019 at 9:19 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > >> > > > > >> > > > > >> > > > > >> On Fri, Aug 23, 2019 at 8:03 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > > >>> > > > > >>> This little cluster isn't in production or anything like that yet. > > > > >>> > > > > >>> So, I went ahead and used your ethtool commands to disable pause > > > > >>> frames on both interfaces of each server. I then, chose a few VMs to > > > > >>> migrate around at random. > > > > >>> > > > > >>> swm-02 and swm-03 both went out again. Unreachable. Can't ping, can't > > > > >>> ssh, and the SSH session that I had open was unresponsive. > > > > >>> > > > > >>> Any other ideas? > > > > >>> > > > > >> > > > > >> Sorry, no. Looks like two different NICs with different drivers and frimware goes down together. > > > > >> This is a strong indication that the root cause is related to the switch. > > > > >> Maybe you can get some information about the switch config by > > > > >> 'lldptool get-tlv -n -i em1' > > > > >> > > > > > > > > > > Another guess: > > > > > After the optional 'lldptool get-tlv -n -i em1' > > > > > 'systemctl stop lldpad' > > > > > another try to migrate. > > > > > > > > > > > > > > >> > > > > >> > > > > >>> > > > > >>> On Fri, Aug 23, 2019 at 1:50 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > On Fri, Aug 23, 2019 at 6:45 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > > >>> >> > > > > >>> >> Unfortunately, I can't check on the switch. Trust me, I've tried. > > > > >>> >> These servers are in a Co-Lo and I've put 5 tickets in asking about > > > > >>> >> the port configuration. They just get ignored - but that's par for the > > > > >>> >> coarse for IT here. Only about 2 out of 10 of our tickets get any > > > > >>> >> response and usually the response doesn't help. Then

2 1

attach untagged vlan internally on vm
by ernestclydeachua＠gmail.com 25 Aug '19

25 Aug '19

good day currently i am testing oVirt on a single box and setup some tagged vms and non tagged vm. the non tagged vm is a firewall but it has limitations on the number of nic so i cannot attach tagged vnic and wish to handdle vlan tagging on it is it possible to pass untaged franes internally?

6 7

Re: Need to enable STP on ovirt bridges
by Strahil 25 Aug '19

25 Aug '19

Curtis, Do you have enough space to run tcpdump (port not 22) on both hosts and on the small VM you have done previously - and then start the migration? Best Regards, Strahil NikolovOn Aug 24, 2019 22:15, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > I applied a 90Mbs QOS Rate Limit with 10 set for the shares to both > interfaces of 2 of the hosts. My hosts names are swm-01 and swm-02. > > Creating a small VM from a Cinder template and running it gave me a test VM. > > When I migrated from swm-01 to swm-02, swm-01 immediately became > unresponsive to pings, SSH'es, and to the ovirt interface which marked > it as "NonResponsive" soon after the VM finished. The VM did finish > migrating, however I'm unsure if that's a good migration or not. > > Thank you, Strahil. > > On Sat, Aug 24, 2019 at 12:39 PM Strahil <hunter86_bg(a)yahoo.com> wrote: > > > > What is your bandwidth threshold for the network used for VM migration ? > > Can you set a 90 mbit/s threshold (yes, less than 100mbit/s) and try to migrate a small (1 GB RAM) VM ? > > > > Do you see disconnects ? > > > > If no, try a little bit up (the threshold) and check again. > > > > Best Regards, > > Strahil NikolovOn Aug 23, 2019 23:19, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > > > > > It took a while for my servers to come back on the network this time. > > > I think it's due to ovirt continuing to try to migrate the VMs around > > > like I requested. The 3 servers' names are "swm-01, swm-02 and > > > swm-03". Eventually (about 2-3 minutes ago) they all came back online. > > > > > > So I disabled and stopped the lldpad service. > > > > > > Nope. Started some more migrations and swm-02 and swm-03 disappeared > > > again. No ping, SSH hung, same as before - almost as soon as the > > > migration started. > > > > > > If you wall have any ideas what switch-level setting might be enabled, > > > let me know, cause I'm stumped. I can add it to the ticket that's > > > requesting the port configurations. I've already added the port > > > numbers and switch name that I got from CDP. > > > > > > Thanks again, I really appreciate the help! > > > cecjr > > > > > > > > > > > > On Fri, Aug 23, 2019 at 3:28 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > > > > > > > > > > > > > On Fri, Aug 23, 2019 at 9:19 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > >> > > > >> > > > >> > > > >> On Fri, Aug 23, 2019 at 8:03 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > >>> > > > >>> This little cluster isn't in production or anything like that yet. > > > >>> > > > >>> So, I went ahead and used your ethtool commands to disable pause > > > >>> frames on both interfaces of each server. I then, chose a few VMs to > > > >>> migrate around at random. > > > >>> > > > >>> swm-02 and swm-03 both went out again. Unreachable. Can't ping, can't > > > >>> ssh, and the SSH session that I had open was unresponsive. > > > >>> > > > >>> Any other ideas? > > > >>> > > > >> > > > >> Sorry, no. Looks like two different NICs with different drivers and frimware goes down together. > > > >> This is a strong indication that the root cause is related to the switch. > > > >> Maybe you can get some information about the switch config by > > > >> 'lldptool get-tlv -n -i em1' > > > >> > > > > > > > > Another guess: > > > > After the optional 'lldptool get-tlv -n -i em1' > > > > 'systemctl stop lldpad' > > > > another try to migrate. > > > > > > > > > > > >> > > > >> > > > >>> > > > >>> On Fri, Aug 23, 2019 at 1:50 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > >>> > > > > >>> > > > > >>> > > > > >>> > On Fri, Aug 23, 2019 at 6:45 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > >>> >> > > > >>> >> Unfortunately, I can't check on the switch. Trust me, I've tried. > > > >>> >> These servers are in a Co-Lo and I've put 5 tickets in asking about > > > >>> >> the port configuration. They just get ignored - but that's par for the > > > >>> >> coarse for IT here. Only about 2 out of 10 of our tickets get any > > > >>> >> response and usually the response doesn't help. Then

1 0

Re: Need to enable STP on ovirt bridges
by Strahil 25 Aug '19

25 Aug '19

Curtis, Do you have enough space to run tcpdump (port not 22) on both hosts and on the small VM you have done previously - and then start the migration? Best Regards, Strahil NikolovOn Aug 24, 2019 22:15, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > I applied a 90Mbs QOS Rate Limit with 10 set for the shares to both > interfaces of 2 of the hosts. My hosts names are swm-01 and swm-02. > > Creating a small VM from a Cinder template and running it gave me a test VM. > > When I migrated from swm-01 to swm-02, swm-01 immediately became > unresponsive to pings, SSH'es, and to the ovirt interface which marked > it as "NonResponsive" soon after the VM finished. The VM did finish > migrating, however I'm unsure if that's a good migration or not. > > Thank you, Strahil. > > On Sat, Aug 24, 2019 at 12:39 PM Strahil <hunter86_bg(a)yahoo.com> wrote: > > > > What is your bandwidth threshold for the network used for VM migration ? > > Can you set a 90 mbit/s threshold (yes, less than 100mbit/s) and try to migrate a small (1 GB RAM) VM ? > > > > Do you see disconnects ? > > > > If no, try a little bit up (the threshold) and check again. > > > > Best Regards, > > Strahil NikolovOn Aug 23, 2019 23:19, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > > > > > It took a while for my servers to come back on the network this time. > > > I think it's due to ovirt continuing to try to migrate the VMs around > > > like I requested. The 3 servers' names are "swm-01, swm-02 and > > > swm-03". Eventually (about 2-3 minutes ago) they all came back online. > > > > > > So I disabled and stopped the lldpad service. > > > > > > Nope. Started some more migrations and swm-02 and swm-03 disappeared > > > again. No ping, SSH hung, same as before - almost as soon as the > > > migration started. > > > > > > If you wall have any ideas what switch-level setting might be enabled, > > > let me know, cause I'm stumped. I can add it to the ticket that's > > > requesting the port configurations. I've already added the port > > > numbers and switch name that I got from CDP. > > > > > > Thanks again, I really appreciate the help! > > > cecjr > > > > > > > > > > > > On Fri, Aug 23, 2019 at 3:28 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > > > > > > > > > > > > > On Fri, Aug 23, 2019 at 9:19 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > >> > > > >> > > > >> > > > >> On Fri, Aug 23, 2019 at 8:03 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > >>> > > > >>> This little cluster isn't in production or anything like that yet. > > > >>> > > > >>> So, I went ahead and used your ethtool commands to disable pause > > > >>> frames on both interfaces of each server. I then, chose a few VMs to > > > >>> migrate around at random. > > > >>> > > > >>> swm-02 and swm-03 both went out again. Unreachable. Can't ping, can't > > > >>> ssh, and the SSH session that I had open was unresponsive. > > > >>> > > > >>> Any other ideas? > > > >>> > > > >> > > > >> Sorry, no. Looks like two different NICs with different drivers and frimware goes down together. > > > >> This is a strong indication that the root cause is related to the switch. > > > >> Maybe you can get some information about the switch config by > > > >> 'lldptool get-tlv -n -i em1' > > > >> > > > > > > > > Another guess: > > > > After the optional 'lldptool get-tlv -n -i em1' > > > > 'systemctl stop lldpad' > > > > another try to migrate. > > > > > > > > > > > >> > > > >> > > > >>> > > > >>> On Fri, Aug 23, 2019 at 1:50 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > >>> > > > > >>> > > > > >>> > > > > >>> > On Fri, Aug 23, 2019 at 6:45 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > >>> >> > > > >>> >> Unfortunately, I can't check on the switch. Trust me, I've tried. > > > >>> >> These servers are in a Co-Lo and I've put 5 tickets in asking about > > > >>> >> the port configuration. They just get ignored - but that's par for the > > > >>> >> coarse for IT here. Only about 2 out of 10 of our tickets get any > > > >>> >> response and usually the response doesn't help. Then

1 0

Re: Need to enable STP on ovirt bridges
by Strahil 25 Aug '19

25 Aug '19

What is your bandwidth threshold for the network used for VM migration ? Can you set a 90 mbit/s threshold (yes, less than 100mbit/s) and try to migrate a small (1 GB RAM) VM ? Do you see disconnects ? If no, try a little bit up (the threshold) and check again. Best Regards, Strahil NikolovOn Aug 23, 2019 23:19, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > It took a while for my servers to come back on the network this time. > I think it's due to ovirt continuing to try to migrate the VMs around > like I requested. The 3 servers' names are "swm-01, swm-02 and > swm-03". Eventually (about 2-3 minutes ago) they all came back online. > > So I disabled and stopped the lldpad service. > > Nope. Started some more migrations and swm-02 and swm-03 disappeared > again. No ping, SSH hung, same as before - almost as soon as the > migration started. > > If you wall have any ideas what switch-level setting might be enabled, > let me know, cause I'm stumped. I can add it to the ticket that's > requesting the port configurations. I've already added the port > numbers and switch name that I got from CDP. > > Thanks again, I really appreciate the help! > cecjr > > > > On Fri, Aug 23, 2019 at 3:28 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > > > > > On Fri, Aug 23, 2019 at 9:19 PM Dominik Holler <dholler(a)redhat.com> wrote: > >> > >> > >> > >> On Fri, Aug 23, 2019 at 8:03 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > >>> > >>> This little cluster isn't in production or anything like that yet. > >>> > >>> So, I went ahead and used your ethtool commands to disable pause > >>> frames on both interfaces of each server. I then, chose a few VMs to > >>> migrate around at random. > >>> > >>> swm-02 and swm-03 both went out again. Unreachable. Can't ping, can't > >>> ssh, and the SSH session that I had open was unresponsive. > >>> > >>> Any other ideas? > >>> > >> > >> Sorry, no. Looks like two different NICs with different drivers and frimware goes down together. > >> This is a strong indication that the root cause is related to the switch. > >> Maybe you can get some information about the switch config by > >> 'lldptool get-tlv -n -i em1' > >> > > > > Another guess: > > After the optional 'lldptool get-tlv -n -i em1' > > 'systemctl stop lldpad' > > another try to migrate. > > > > > >> > >> > >>> > >>> On Fri, Aug 23, 2019 at 1:50 PM Dominik Holler <dholler(a)redhat.com> wrote: > >>> > > >>> > > >>> > > >>> > On Fri, Aug 23, 2019 at 6:45 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > >>> >> > >>> >> Unfortunately, I can't check on the switch. Trust me, I've tried. > >>> >> These servers are in a Co-Lo and I've put 5 tickets in asking about > >>> >> the port configuration. They just get ignored - but that's par for the > >>> >> coarse for IT here. Only about 2 out of 10 of our tickets get any > >>> >> response and usually the response doesn't help. Then the system they > >>> >> use auto-closes the ticket. That was why I was suspecting STP before. > >>> >> > >>> >> I can do ethtool. I do have root on these servers, though. Are you > >>> >> trying to get me to turn off link-speed auto-negotiation? Would you > >>> >> like me to try that? > >>> >> > >>> > > >>> > It is just a suspicion, that the reason is pause frames. > >>> > Let's start on a NIC which is not used for ovirtmgmt, I guess em1. > >>> > Does 'ethtool -S em1 | grep pause' show something? > >>> > Does 'ethtool em1 | grep pause' indicates support for pause? > >>> > The current config is shown by 'ethtool -a em1'. > >>> > '-A autoneg' "Specifies whether pause autonegotiation should be enabled." according to ethtool doc. > >>> > Assuming flow control is enabled by default, I would try to disable it via > >>> > 'ethtool -A em1 autoneg off rx off tx off' > >>> > and check if it is applied via > >>> > 'ethtool -a em1' > >>> > and check if the behavior under load changes. > >>> > > >>> > > >>> > > >>> >> > >>> >> On Fri, Aug 23, 2019 at 12:24 PM Dominik Holler <dholler(a)redhat.com> wrote: > >>> >> > > >>> >> > > >>> >> > > >>> >> > On Fri, Aug 23, 2019 at 5:49 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > >>> >> >> > >>> >> >> Sure! Right now, I only have a 5

3 2

Moving ovirt engine disk to another storage volume
by Erick Perez - Quadrian Enterprises 25 Aug '19

25 Aug '19

Good morning, I am running Ovirt 4.3.5 in Centos 7.6 with one virt node an a NFS storage node. I did the self-hosted engine setup and i plan to add a second virt host in a few days. I need to do heavy maintenance on the storage node (VDO and mdadm things) and would like to know how (or a link to an article) can I move the ovirt engine disk to another storage. Currentl the NFS storage has two volumens (volA,volB) and the physical host have spare space too. Virtual machines are in VolB and the engine is in VolA. I would like to move the engine disk from VolA to VolB or to local storage. BTW I am not sure if I should say "move the engine" or should I say move "hosted_storage domain" domain thanks in advance. --------------------- Erick Perez

3 6

TASK [ovirt.hosted_engine_setup : Get local VM IP] Problem
by Emre Özkan 24 Aug '19

24 Aug '19

hi guys, ı waiting this part when I try to install with hosted-engine --deploy command.have you got any knowledge about that.My environment(hyper-v running nested vm on azure and rhev running on hyper-v) [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Create local VM] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Get local VM IP]

1 0

numa pinning and reserved hugepages (1G) Bug in scheduler calculation or decision ?
by Ralf Schenk 24 Aug '19

24 Aug '19

Hello List, i ran into problems using numa-pinning and reserved hugepages. - My EPYC 7281 based Servers (Dual Socket) have 8 Numa-Nodes each having 32 GB of memory for a total of 256 GB System Memory - I'm using 192 x 1 GB hugepages reserved on the kernel cmdline default_hugepagesz=1G hugepagesz=1G hugepages=192 This reserves 24 hugepages on each numa-node. I wanted to pin a MariaDB VM using 32 GB (Custom Property hugepages=1048576) to numa-nodes 0-3 of CPU-Socket 1. Pinning in GUI etc. no problem. When trying to start the vm this can't be done since ovirt claims that the host can't fullfill the memory requirements - which is simply not correct since there were > 164 hugepages free. It should have taken 8 hugepages from each numa node 0-3 to fullfill the 32 GB Memory requirement. I also freed the system completely from other VM's but that didn't work either. Is it possible that the scheduler only takes into account the "free memory" (as seen in numactl -H below) *not reserved* by hugepages for its decisions ? Since the host has only < 8 GB of free mem per numa-node I can understand that VM was not able to start under that condition. VM is runnig and using 32 hugepages without pinning but a warning states "VM dbserver01b does not fit to a single NUMA node on host myhost.mydomain.de. This may negatively impact its performance. Consider using vNUMA and NUMA pinning for this VM." This is the numa Hardware Layout and hugepages usage now with other VM's running: from cat /proc/meminfo HugePages_Total: 192 HugePages_Free: 160 HugePages_Rsvd: 0 HugePages_Surp: 0 I can confirm that also under the condition of running other VM's there are at least 8 hugepages free for each numa-node 0-3: grep "" /sys/devices/system/node/*/hugepages/hugepages-1048576kB/free_hugepages /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/free_hugepages:8 /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/free_hugepages:23 /sys/devices/system/node/node2/hugepages/hugepages-1048576kB/free_hugepages:20 /sys/devices/system/node/node3/hugepages/hugepages-1048576kB/free_hugepages:22 /sys/devices/system/node/node4/hugepages/hugepages-1048576kB/free_hugepages:16 /sys/devices/system/node/node5/hugepages/hugepages-1048576kB/free_hugepages:5 /sys/devices/system/node/node6/hugepages/hugepages-1048576kB/free_hugepages:19 /sys/devices/system/node/node7/hugepages/hugepages-1048576kB/free_hugepages:24 numactl -h: available: 8 nodes (0-7) node 0 cpus: 0 1 2 3 32 33 34 35 node 0 size: 32673 MB node 0 free: 3779 MB node 1 cpus: 4 5 6 7 36 37 38 39 node 1 size: 32767 MB node 1 free: 6162 MB node 2 cpus: 8 9 10 11 40 41 42 43 node 2 size: 32767 MB node 2 free: 6698 MB node 3 cpus: 12 13 14 15 44 45 46 47 node 3 size: 32767 MB node 3 free: 1589 MB node 4 cpus: 16 17 18 19 48 49 50 51 node 4 size: 32767 MB node 4 free: 2630 MB node 5 cpus: 20 21 22 23 52 53 54 55 node 5 size: 32767 MB node 5 free: 2487 MB node 6 cpus: 24 25 26 27 56 57 58 59 node 6 size: 32767 MB node 6 free: 3279 MB node 7 cpus: 28 29 30 31 60 61 62 63 node 7 size: 32767 MB node 7 free: 5513 MB node distances: node 0 1 2 3 4 5 6 7 0: 10 16 16 16 32 32 32 32 1: 16 10 16 16 32 32 32 32 2: 16 16 10 16 32 32 32 32 3: 16 16 16 10 32 32 32 32 4: 32 32 32 32 10 16 16 16 5: 32 32 32 32 16 10 16 16 6: 32 32 32 32 16 16 10 16 7: 32 32 32 32 16 16 16 10 -- *Ralf Schenk* fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail *rs(a)databay.de* <mailto:rs@databay.de> *Databay AG* Jens-Otto-Krag-Straße 11 D-52146 Würselen *www.databay.de* <http://www.databay.de> Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen ------------------------------------------------------------------------

2 2

Need to enable STP on ovirt bridges
by ej.albany＠gmail.com 23 Aug '19

23 Aug '19

Hello. I have been trying to figure out an issue for a very long time. That issue relates to the ethernet and 10gb fc links that I have on my cluster being disabled any time a migration occurs. I believe this is because I need to have STP turned on in order to participate with the switch. However, there does not seem to be any way to tell oVirt to stop turning it off! Very frustrating. After entering a cronjob that enables stp on all bridges every 1 minute, the migration issue disappears.... Is there any way at all to do without this cronjob and set STP to be ON without having to resort to such a silly solution? Here are some details about my systems, if you need it. selinux is disabled. [root@swm-02 ~]# rpm -qa | grep ovirt ovirt-imageio-common-1.5.1-0.el7.x86_64 ovirt-release43-4.3.5.2-1.el7.noarch ovirt-imageio-daemon-1.5.1-0.el7.noarch ovirt-vmconsole-host-1.0.7-2.el7.noarch ovirt-hosted-engine-setup-2.3.11-1.el7.noarch ovirt-ansible-hosted-engine-setup-1.0.26-1.el7.noarch python2-ovirt-host-deploy-1.8.0-1.el7.noarch ovirt-ansible-engine-setup-1.1.9-1.el7.noarch python2-ovirt-setup-lib-1.2.0-1.el7.noarch cockpit-machines-ovirt-195.1-1.el7.noarch ovirt-hosted-engine-ha-2.3.3-1.el7.noarch ovirt-vmconsole-1.0.7-2.el7.noarch cockpit-ovirt-dashboard-0.13.5-1.el7.noarch ovirt-provider-ovn-driver-1.2.22-1.el7.noarch ovirt-host-deploy-common-1.8.0-1.el7.noarch ovirt-host-4.3.4-1.el7.x86_64 python-ovirt-engine-sdk4-4.3.2-2.el7.x86_64 ovirt-host-dependencies-4.3.4-1.el7.x86_64 ovirt-ansible-repositories-1.1.5-1.el7.noarch [root@swm-02 ~]# cat /etc/redhat-release CentOS Linux release 7.6.1810 (Core) [root@swm-02 ~]# uname -r 3.10.0-957.27.2.el7.x86_64 You have new mail in /var/spool/mail/root [root@swm-02 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master test state UP group default qlen 1000 link/ether d4:ae:52:8d:50:48 brd ff:ff:ff:ff:ff:ff 3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether d4:ae:52:8d:50:49 brd ff:ff:ff:ff:ff:ff 4: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovirtmgmt state UP group default qlen 1000 link/ether 90:e2:ba:1e:14:80 brd ff:ff:ff:ff:ff:ff 5: p1p2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 90:e2:ba:1e:14:81 brd ff:ff:ff:ff:ff:ff 6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether a2:b8:d6:e8:b3:d8 brd ff:ff:ff:ff:ff:ff 7: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 96:a0:c1:4a:45:4b brd ff:ff:ff:ff:ff:ff 25: test: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether d4:ae:52:8d:50:48 brd ff:ff:ff:ff:ff:ff inet 10.15.11.21/24 brd 10.15.11.255 scope global test valid_lft forever preferred_lft forever 26: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 90:e2:ba:1e:14:80 brd ff:ff:ff:ff:ff:ff inet 10.15.28.31/24 brd 10.15.28.255 scope global ovirtmgmt valid_lft forever preferred_lft forever 27: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 62:e5:e5:07:99:eb brd ff:ff:ff:ff:ff:ff 29: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovirtmgmt state UNKNOWN group default qlen 1000 link/ether fe:6f:9c:95:00:02 brd ff:ff:ff:ff:ff:ff [root@swm-02 ~]# free -m total used free shared buff/cache available Mem: 64413 1873 61804 9 735 62062 Swap: 16383 0 16383 [root@swm-02 ~]# free -h total used free shared buff/cache available Mem: 62G 1.8G 60G 9.5M 735M 60G Swap: 15G 0B 15G [root@swm-02 ~]# ls ls lsb_release lshw lslocks lsmod lspci lssubsys lsusb.py lsattr lscgroup lsinitrd lslogins lsns lss16toppm lstopo-no-graphics lsblk lscpu lsipc lsmem lsof lsscsi lsusb [root@swm-02 ~]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 44 Model name: Intel(R) Xeon(R) CPU X5672 @ 3.20GHz Stepping: 2 CPU MHz: 3192.064 BogoMIPS: 6384.12 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 12288K NUMA node0 CPU(s): 0,2,4,6,8,10,12,14 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid dtherm ida arat spec_ctrl intel_stibp flush_l1d [root@swm-02 ~]#

6 34

Migration from bare metal engine to hosted doesn't seem to work at all in 4.3.2
by Clinton Goudie-Nice 23 Aug '19

23 Aug '19

Hi all, I'm trying to migrate from an ovirt bare metal instance to the hosted appliance. The instructions here: https://www.ovirt.org/documentation/self-hosted/chap-Migrating_from_Bare_Me… are at best murky, and at worst quite wrong, and I'm looking for some help. The instructions say "You must answer No to the following question so that you can restore the BareMetal-Engine backup file on HostedEngine-VM before running engine-setup. Automatically execute engine-setup on the engine appliance on first boot (Yes, No)[Yes]? No" Makes complete sense, except this question is not asked at all. Some research on the internet says we should be able to do this via append-answers: [environment:default] OVEHOSTED_VM/cloudinitExecuteEngineSetup=bool:False Except this doesn't work either. When the appliance deploys the engine is setup and already running. How do I take a backup file, and a backup log, and deploy them into a hosted engine? Clint

1 0

Creating hosts via the REST API using SSH Public Key authentication
by schill-julian＠gmx.de 23 Aug '19

23 Aug '19

In the UI one can create hosts using two authentication methods: 'Password' and 'SSH Public Key'. I have only found the Password authentication in the API Docs (/ovirt-engine/apidoc/#/services/hosts/methods/add). My question is: How can i create hosts using SSH Public Key authentication via the REST API? I would appreciate an example POST request!

3 2

nodectl on plain CentOS hypervisors
by Gianluca Cecchi 23 Aug '19

23 Aug '19

Does it make sense to install nodectl utility on plain CentOS 7.x nodes? Or any other alternative for plain OS nodes vs ovirt-node-ng ones? On my updated CentOS 7.6 oVirt node I have not the command; I think it is provided by the package ovirt-node-ng-nodectl, that is one of the available ones if I run "yum search" on the system. Thanks Gianluca

2 2

oVirt 3.6: Node went into Error state while migrations happening
by Christopher Cox 23 Aug '19

23 Aug '19

On the node in question, the metadata isn't coming across (state) wise. It shows VMs being in an unknown state (some are up and some are down), some show as migrating and there are 9 forever hung migrating tasks. We tried to bring up some of the down VMs that had a state of Down, but that ended up getting them the state of "Wait for Lauch", though those VMs are actually started. Right now, my plan is attempt a restart of vdsmd on the node in question. Just trying to get the node to a working state again. There a total of 9 nodes in our cluster, but we can't manage any VMs on the affected node right now. Is there a way in 3.6 to cancel the hung tasks? I'm worried that if vdsmd is restarted on the node, the tasks might be "attempted"... I really need them to be forgotten if possible. Ideally want all "Unknown" to return to either an "up" or "down" state (depending if the VM is up or down) and for "Wait for Launch" for those, to go to "up" and for all the "Migrating" to go to "up" or "down" (I think only one is actually down). I'm concerned that any attempt manually maniplate the state in the ovirt mgmt head db will be moot because the node will be queried for state and that state will be taken and override anything I attempt to do. Thoughts??

2 1

When I create a new domain in storage , it report : VDSM command ActivateStorageDomainVDS failed: Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842', )
by wangyu13476969128＠126.com 23 Aug '19

23 Aug '19

The version of ovirt-engine is 4.2.8 The version of ovirt-node is 4.2.8 When I create a new domain in storage , the storage type is NFS , it report : VDSM command ActivateStorageDomainVDS failed: Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842',) The error of vdsm.log is : 2019-08-23 11:02:14,740+0800 INFO (jsonrpc/4) [vdsm.api] START connectStorageServer(domType=1, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'id': u'c6893a09-ab28-4328-b186-d2f88f2320d4', u'connection': u'172.16.10.74:/ovirt-data', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', u'port': u''}], options=None) from=::ffff:172.16.90.10,52962, flow_id=5d720a32, task_id=2af924ff-37a6-46a1-b79f-4251d21d5ff9 (api:46) 2019-08-23 11:02:14,743+0800 INFO (jsonrpc/4) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'c6893a09-ab28-4328-b186-d2f88f2320d4'}]} from=::ffff:172.16.90.10,52962, flow_id=5d720a32, task_id=2af924ff-37a6-46a1-b79f-4251d21d5ff9 (api:52) 2019-08-23 11:02:14,743+0800 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.00 seconds (__init__:573) 2019-08-23 11:02:14,751+0800 INFO (jsonrpc/6) [vdsm.api] START activateStorageDomain(sdUUID=u'3bcdc32c-040e-4a4c-90fb-de950f54f1b4', spUUID=u'b87012a1-8f7a-4af5-8884-e0fb8002e842', options=None) from=::ffff:172.16.90.10,53080, flow_id=709ee722, task_id=9b8f32af-5fdd-4ffa-9520-c474af03db70 (api:46) 2019-08-23 11:02:14,752+0800 INFO (jsonrpc/6) [vdsm.api] FINISH activateStorageDomain error=Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842',) from=::ffff:172.16.90.10,53080, flow_id=709ee722, task_id=9b8f32af-5fdd-4ffa-9520-c474af03db70 (api:50) 2019-08-23 11:02:14,752+0800 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='9b8f32af-5fdd-4ffa-9520-c474af03db70') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1262, in activateStorageDomain pool = self.getPool(spUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 350, in getPool raise se.StoragePoolUnknown(spUUID) StoragePoolUnknown: Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842',) 2019-08-23 11:02:14,752+0800 INFO (jsonrpc/6) [storage.TaskManager.Task] (Task='9b8f32af-5fdd-4ffa-9520-c474af03db70') aborting: Task is aborted: "Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842',)" - code 309 (task:1181) 2019-08-23 11:02:14,752+0800 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH activateStorageDomain error=Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842',) (dispatcher:82) How can I solve this problem ?

2 1

large size OVA import fails
by jason.l.cox＠l3harris.com 22 Aug '19

22 Aug '19

I have a fairly large OVA (~200GB) that was exported from oVirt4.3.5. I'm trying to import it into a new cluster, also oVirt4.3.5. The import starts fine but fails again and again. Everything I can find online appears to be outdated, mentioning incorrect log file locations and saying virt-v2v does the import. On the engine in /var/log/ovirt-engine/engine.log I can see where it is doing the CreateImageVDSCommand, then a few outputs concerning adding the disk, which end with USER_ADD_DISK_TO_VM_FINISHED_SUCCESS, then the ansible command: 2019-08-20 15:40:38,653-04 Executing Ansible command: /usr/bin/ansible-playbook --ssh-common-args=-F /var/lib/ovirt-engine/.ssh/config -v --private-key=/etc/pki/ovirt-engine/keys/engine_id_rsa --inventory=/tmp/ansible-inventory8416464991088315694 --extra-vars=ovirt_import_ova_path="/mnt/vm_backups/myvm.ova" --extra-vars=ovirt_import_ova_disks="['/rhev/data-center/mnt/glusterSD/myhost.mydomain.com:_vmstore/59502c8b-fd1e-482b-bff7-39c699c196b3/images/886a3313-19a9-435d-aeac-64c2d507bb54/465ce2ba-8883-4378-bae7-e231047ea09d']" --extra-vars=ovirt_import_ova_image_mappings="{}" /usr/share/ovirt-engine/playbooks/ovirt-ova-import.yml [Logfile: /var/log/ovirt-engine/ova/ovirt-import-ova-ansible-20190820154038-myhost.mydomain.com-25f6ac6f-9bdc-4301-b896-d357712dbf01.log] Then nothing about the import until: 2019-08-20 16:11:08,859-04 INFO [org.ovirt.engine.core.bll.exportimport.ImportVmFromOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-88) [3321d4f6] Lock freed to object 'EngineLock:{exclusiveLocks='[myvm=VM_NAME, 464a25ba-8f0a-421d-a6ab-13eff67b4c96=VM]', sharedLocks=''}' 2019-08-20 16:11:08,894-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-88) [3321d4f6] EVENT_ID: IMPORTEXPORT_IMPORT_VM_FAILED(1,153), Failed to import Vm myvm to Data Center Default, Cluster Default I've found the import logs on the engine, in /var/log/ovirt-engine/ova, but the ovirt-import-ova-ansible*.logs for the imports of concern only contain: 2019-08-20 19:59:48,799 p=44701 u=ovirt | Using /usr/share/ovirt-engine/playbooks/ansible.cfg as config file 2019-08-20 19:59:49,271 p=44701 u=ovirt | PLAY [all] ********************************************************************* 2019-08-20 19:59:49,280 p=44701 u=ovirt | TASK [ovirt-ova-extract : Run extraction script] ******************************* Watching the host selected for the import, I can see the qemu-img convert process running, but then the engine frees the lock on the VM and reports the import as having failed. However, the qemu-img process continues to run on the host. I don't know where else to look to try and find out what's going on and I cannot see anything that says why the import failed. Since the qemu-img process on the host is still running after the engine log shows the lock has been freed and import failed, I'm guessing what's happening is on the engine side. Looking at the time between the start of the ansible command and when the lock is freed it is consistently around 30 minutes. # first try 2019-08-20 15:40:38,653-04 ansible command start 2019-08-20 16:11:08,859-04 lock freed 31 minutes # second try 2019-08-20 19:59:48,463-04 ansible command start 2019-08-20 20:30:21,697-04 lock freed 30 minutes, 33 seconds # third try 2019-08-21 09:16:42,706-04 ansible command start 2019-08-21 09:46:47,103-04 lock freed 30 minutes, 45 seconds With that in mind, I took a look at the available configuration keys from engine-config --list. After getting each the only one set to ~30 minutes and looks like it could be the problem was SSHInactivityHardTimeoutSeconds (set to 1800 seconds). I set it to 3600 and tried the import again, but it still failed at ~30 minutes, so that's apparently not the correct key. Also, just FYI, I also tried to import the ova using virt-v2v, but that fails immediately: virt-v2v: error: expecting XML expression to return an integer (expression: rasd:Parent/text(), matching string: 00000000-0000-0000-0000-000000000000) If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] Does virt-v2v not support OVAs created by the oVirt 'export to ova' option? So my main question is, Is there a timeout for VM imports through the engine web UI? And if so, is if configurable? Thanks in advance.

2 1

ovn networking
by Staniforth, Paul 22 Aug '19

22 Aug '19

Hello I the latest release of the engine 4.3.5.5-1.el7 ovn-nbctl show ovn-sbctl show both seem to work but produce the errors net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5) PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4) should libibverbs be a dependency? Paul S. To view the terms under which this email is distributed, please go to:- http://leedsbeckett.ac.uk/disclaimer/email/

3 5

VM won't migrate back to original node
by Alexander Schichow 22 Aug '19

22 Aug '19

I set up a high available VM which is using a shareable LUN. When migrating said VM from the original Node everything is fine. When I try to migrate it back, I get an error. If I suspend or shutdown the VM, it returns to the original host. Then it repeats. I'm really out of ideas so maybe any of you happen to have any? I would really appreciate that. Thanks

2 1

Custum driver in oVirt node image?
by thomas＠hoberg.net 22 Aug '19

22 Aug '19

Using 3 Mini-ITX Atoms J5005 boards to build a three node HC test environment (actually there is a fourth to simulate a remote DC) These boards have a RealTek 8169 Gbit controller, which for hardware/firmware or driver reasons comes up just fine on cold boots, but struggles on warm-boots or whenever the interface is re-configured, as is the case when the OVS overlay is configured, ... * unless link-speed autonegotiation is enabled after which it works just fine. So for a normal system, I can just put a proper ETHTOOL_OPTS="autoneg on" into /etc/sysconfig/network-scrkipts/ifcfg-<device-name> to avoid trouble on warm boots, but once the overlay network is put on top, the network manager no longer controls the device and the standard "ignore" setting of driver/hardware evidently causes the link to fail: dmesg reports the interface toggling and ovs isn't happy either. The Ethernet switch is unmanaged, so there is nothing I can do there to speed up or eliminate negotiations, and I currently see three options: 1. Find/be told where to renable Ethernet link speed autonegotiation with the OVS/VDSM/?? so the RTL8169 is happy again 2. Use a known USB3 GBit NIC (no space in the chassis to put e.g. an Intel NIC) 3. Make an USB3 2.5Gbit NIC based on RealTek's 8156 work, but currently that requires compiling from source and loading it at boot Option 3 works just fine with a CentOS base, but I got tons of problem making oVirt hosted engine work with that. Option 3 for the oVirt node image seems to class with the read-only nature of the oVirt node imge: While I can load a pre-compile driver interactively, I can't make it stick with dracut -f. I haven't immediately found a way to patch/fix/build derivative oVirt node images with that driver included and am somewhat to impatient to wait until it's mainstream, so any help would be appreciated. I've verified option 2 with a single USB3 GBit adapter natively supported by the kernel that I already had available, but I then opted to jump the gun and buy the slightly more expensive 2.5GBit variant as a better match to a Gluster environment: A mistake in this context as it turned out.

2 2

[ANN] oVirt 4.3.6 Third Release Candidate is now available for testing
by Sandro Bonazzola 22 Aug '19

22 Aug '19

The oVirt Project is pleased to announce the availability of the oVirt 4.3.6 Third Release Candidate for testing, as of August 22nd, 2019. This update is a release candidate of the sixth in a series of stabilization updates to the 4.3 series. This is pre-release software. This pre-release should not to be used in production. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) * oVirt Node 4.3 (available for x86_64 only) will be made available when CentOS 7.7 will be released. See the release notes [1] for known issues, new features and bugs fixed. Notes: - oVirt Appliance is already available - oVirt Node is not yet available, pending CentOS 7.7 release to be available Additional Resources: * Read more about the oVirt 4.3.6 release highlights: http://www.ovirt.org/release/4.3.6/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.6/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0