August 2019 - Users - oVirt List Archives

ovirt api snapshot date format?
by Bill James 29 Aug '19

29 Aug '19

I'm trying to use the ovirt REST api to read snapshot date but the number I'm getting back doesn't appear to be in epoch time. What format is it? Example: "date":1565299725928,"snapshot_status":"ok" Should be "Aug 8, 2019", epoch conversion put it at Tue May 16 16:25:28 51572. (accessed via https://$HOSTNAME/ovirt-engine/api/vms/$value/snapshots)

2 1

Removing Unknown VM's on Dead Cluster
by Ross, Travis 28 Aug '19

28 Aug '19

I've clearly seen this is a pretty common issue with many different methods for cleaning up dead clusters. I'm having issues with unknown VM's not allowing my host into Maintenance Mode. Version: ovirt-release36-3.6.7-1.noarch Situation: Another site repurposed a host entirely without properly cleaning it's status in my Engine. No option to delete host. I've attempted the "Confirm Host is Rebooted" -> Maintenance Mode. However the error is: Cannot Switch the following Hosts to Maintenance mode: ... One or more running VMs are indicated as non-migratable. The status of all the listed VM's is "UNKNOWN" w/ "?" next to them. If I look at the settings for these VM's they already have "Do not allow migration" checked. Also I'm unable to modify the VM since the Host doesn't exist. Can't just delete the Domain because the Host exists. Can't delete Storage because host is not in Maintenance Mode. Can't set Maintenance Mode because VM's can't migrate. At this point all I want to do is upgrade to latest ovirt, however I need to get this orphaned Domain out of my single node cluster, which still runs a ~10 VM's hosted on Gluster Storage. Thoughts on whether I could just backup these VM's, re-install ovirt fresh then import the VM's? Thanks, Travis Ross W 512.231.4235 M 512.547.9437 [cid:image003.png@01D55D96.2A607620] KLA-Tencor is now KLA! Learn more at kla.com <https://www.kla.com/>

1 0

Update single node environment from 4.3.3 to 4.3.5 problem
by Gianluca Cecchi 28 Aug '19

28 Aug '19

Hello, after updating hosted engine from 4.3.3 to 4.3.5 and then the only host composing the environment (plain CentOS 7.6) it seems it is not able to start vdsm daemons kernel installed with update is kernel-3.10.0-957.27.2.el7.x86_64 Same problem also if using previous running kernel 3.10.0-957.12.2.el7.x86_64 [root@ovirt01 vdsm]# uptime 00:50:08 up 25 min, 3 users, load average: 0.60, 0.67, 0.60 [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# systemctl status vdsmd -l ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/etc/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: failed (Result: start-limit) since Fri 2019-08-23 00:37:27 CEST; 7s ago Process: 25810 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=1/FAILURE) Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Failed to start Virtual Desktop Server Manager. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Unit vdsmd.service entered failed state. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: vdsmd.service failed. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: vdsmd.service holdoff time over, scheduling restart. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Stopped Virtual Desktop Server Manager. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: start request repeated too quickly for vdsmd.service Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Failed to start Virtual Desktop Server Manager. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Unit vdsmd.service entered failed state. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: vdsmd.service failed. [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# pwd /var/log/vdsm [root@ovirt01 vdsm]# ll -t | head total 118972 -rw-r--r--. 1 root root 3406465 Aug 23 00:25 supervdsm.log -rw-r--r--. 1 root root 73621 Aug 23 00:25 upgrade.log -rw-r--r--. 1 vdsm kvm 0 Aug 23 00:01 vdsm.log -rw-r--r--. 1 vdsm kvm 538480 Aug 22 23:46 vdsm.log.1.xz -rw-r--r--. 1 vdsm kvm 187486 Aug 22 23:46 mom.log -rw-r--r--. 1 vdsm kvm 621320 Aug 22 22:01 vdsm.log.2.xz -rw-r--r--. 1 root root 374464 Aug 22 22:00 supervdsm.log.1.xz -rw-r--r--. 1 vdsm kvm 2097122 Aug 22 21:53 mom.log.1 -rw-r--r--. 1 vdsm kvm 636212 Aug 22 20:01 vdsm.log.3.xz [root@ovirt01 vdsm]# link to upgrade.log contents here: https://drive.google.com/file/d/17jtX36oH1hlbNUAiVhdBkVDbd28QegXG/view?usp=… link to supervdsm.log (in gzip format) here: https://drive.google.com/file/d/1l61ePU-eFS_xVHEAHnJthzTTnTyzu0MP/view?usp=… It seems that since update I get these kind of lines inside it... restore-net::DEBUG::2019-08-22 23:56:38,591::cmdutils::133::root::(exec_cmd) /sbin/tc filter del dev eth0 pref 5000 (cwd None) restore-net::DEBUG::2019-08-22 23:56:38,595::cmdutils::141::root::(exec_cmd) FAILED: <err> = 'RTNETLINK answers: Invalid argument\nWe have an error talking to the kernel\n'; <rc> = 2 [root@ovirt01 vdsm]# systemctl status supervdsmd -l ● supervdsmd.service - Auxiliary vdsm service for running helper functions as root Loaded: loaded (/usr/lib/systemd/system/supervdsmd.service; static; vendor preset: enabled) Active: active (running) since Fri 2019-08-23 00:25:17 CEST; 23min ago Main PID: 4540 (supervdsmd) Tasks: 3 CGroup: /system.slice/supervdsmd.service └─4540 /usr/bin/python2 /usr/share/vdsm/supervdsmd --sockfile /var/run/vdsm/svdsm.sock Aug 23 00:25:17 ovirt01.mydomain systemd[1]: Started Auxiliary vdsm service for running helper functions as root. [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovirtmgmt state UP group default qlen 1000 link/ether b8:ae:ed:7f:17:11 brd ff:ff:ff:ff:ff:ff 3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:c2:c6:a4:18:c5 brd ff:ff:ff:ff:ff:ff 4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 36:21:c1:5e:70:aa brd ff:ff:ff:ff:ff:ff 5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 46:d8:db:81:41:4e brd ff:ff:ff:ff:ff:ff 22: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:ae:ed:7f:17:11 brd ff:ff:ff:ff:ff:ff inet 192.168.1.211/24 brd 192.168.1.255 scope global ovirtmgmt valid_lft forever preferred_lft forever [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# ip route show default via 192.168.1.1 dev ovirtmgmt 192.168.1.0/24 dev ovirtmgmt proto kernel scope link src 192.168.1.211 [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# brctl show bridge name bridge id STP enabled interfaces ovirtmgmt 8000.b8aeed7f1711 no eth0 [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# systemctl status openvswitch ● openvswitch.service - Open vSwitch Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; enabled; vendor preset: disabled) Active: active (exited) since Fri 2019-08-23 00:25:09 CEST; 26min ago Process: 3894 ExecStart=/bin/true (code=exited, status=0/SUCCESS) Main PID: 3894 (code=exited, status=0/SUCCESS) Tasks: 0 CGroup: /system.slice/openvswitch.service Aug 23 00:25:09 ovirt01.mydomain systemd[1]: Starting Open vSwitch... Aug 23 00:25:09 ovirt01.mydomain systemd[1]: Started Open vSwitch. [root@ovirt01 vdsm]# ovs-vsctl show 02539902-1788-4796-9cdf-cf11ce8436bb Bridge br-int fail_mode: secure Port br-int Interface br-int type: internal ovs_version: "2.11.0" [root@ovirt01 vdsm]# any hints? Thanks Gianluca

4 16

Evaluate oVirt to remove VMware
by dsalade＠gmail.com 28 Aug '19

28 Aug '19

Howdy All!!! Somewhat new user to Linux and having some issues with oVirt node networking. We are mainly a Windows/VMware medium shop but have a pretty good network. Our VMware setup has redundant switches using multiple VLANs. This was fairly easy to setup with VMware. We never had an issue with VMware, add 2 NICs to the ESXi host and then adding VLANs into the Virtual Switch. Biggest issue I am having, should I be bonding, teaming and VLANing on top of that? Tried setting up node with Engine inside which fails to deploy because of networking. Then I tried a standalone Engine and could not add node to it - I believe networking as well as Power Management agent was the problem. Would like the nodes/engine to be on one subnet (lets say 10.10.11.x) and VMs to be allowed to run there are well as 10 other VLANs. I guess welcome to the wonderful world of Linux networking and learning some new technologies is in store for me. I hope I am not being vague, like I said fairly new to Linux. Thanks for any responses! Allen

4 6

VDSM Hooks during migration
by Vrgotic, Marko 27 Aug '19

27 Aug '19

Dear oVIrt, Would you be so kind to help me/tell me or point me how to find which Hooks, and in which order, are triggered when VM is being migrated? Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic

2 6

Re: VM --- is not responding.
by Strahil 26 Aug '19

26 Aug '19

Hm... It was supposes to show controller status. Maybe the hpssacli you have is not supporting your raid cards.Check for newer version on HPE's support page. Best Regards, Strahil Nikolov Best Regards, Strahil NikolovOn Aug 14, 2019 11:40, Edoardo Mazza <edo7411(a)gmail.com> wrote: > > I installed hpssacli-2.40-13.0.x86_64.rpm and the result of "hpssacli ctrl all show status" is: > Error: No controllers detected. Possible causes:..... > The s.o. run on sd cards and the vm runs on array on traditional disk > thanks > Edoardo > > Il giorno lun 12 ago 2019 alle ore 05:59 Strahil <hunter86_bg(a)yahoo.com> ha scritto: >> >> Would you check the health status of the controllers : >> hpssacli ctrl all show status >> >> Best Regards, >> Strahil Nikolov >> >> On Aug 11, 2019 09:55, Edoardo Mazza <edo7411(a)gmail.com> wrote: >>> >>> The hosts are 3 ProLiant DL380 Gen10, 2 hosts with HPE Smart Array P816i-a SR Gen10 like controller and the other host with >>> HPE Smart Array P408i-a SR Gen10. The storage for ovirt enviroment is gluster and the last host is the arbiter in the gluster enviroment. >>> The S.M.A.R.T. healt status is ok for all host >>> Edoardo >>> >>> >>> >>> >>> >>> Il giorno gio 8 ago 2019 alle ore 16:19 Sandro Bonazzola <sbonazzo(a)redhat.com> ha scritto: >>>> >>>> >>>> >>>> Il giorno gio 8 ago 2019 alle ore 11:19 Edoardo Mazza <edo7411(a)gmail.com> ha scritto: >>>>> >>>>> Hi all, >>>>> It is more days that for same vm I received this error, but I don't underdand why. >>>>> The traffic of the virtual machine is not excessive, cpu and ram to, but for few minutes the vm is not responding. and in the messages log file of the vm I received the error under, yo can help me? >>>>> thanks >>>> >>>> >>>> can you check the S.M.A.R.T. health status of the disks? >>>> >>>> >>>>> >>>>> Edoardo >>>>> kernel: NMI watchdog: BUG: soft lockup - CPU#2 stuck for 25s! [kworker/2:0:26227] >>>>> Aug 8 02:51:11 vmmysql kernel: Modules linked in: binfmt_misc ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ >>>>> ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_ >>>>> nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_con >>>>> ntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter snd_hda_c >>>>> odec_generic iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel snd_hda_intel snd_hda_codec aesni_intel snd_hda_core lrw gf128mul >>>>> glue_helper ablk_helper snd_hwdep cryptd snd_seq snd_seq_device snd_pcm snd_timer snd soundcore virtio_rng sg virtio_balloon >>>>> i2c_piix4 parport_pc parport joydev pcspkr ip_tables xfs libcrc32c sd_mod >>>>> Aug 8 02:51:14 vmmysql kernel: crc_t10dif crct10dif_generic sr_mod cdrom virtio_net virtio_console virtio_scsi ata_generic p >>>>> ata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw qxl floppy drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci drm_panel_orientation_quirks virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod >>>>> Aug 8 02:51:14 vmmysql kernel: CPU: 2 PID: 26227 Comm: kworker/2:0 Kdump: loaded Tainted: G L ------------ 3.10.0-957.12.1.el7.x86_64 #1 >>>>> Aug 8 02:51:14 vmmysql kernel: Hardware name: oVirt oVirt Node, BIOS 1.11.0-2.el7 04/01/2014 >>>>> Aug 8 02:51:14 vmmysql kernel: Workqueue: events_freezable disk_events_workfn >>>>> Aug 8 02:51:14 vmmysql kernel: task: ffff9e25b6609040 ti: ffff9e27b1610000 task.ti: ffff9e27b1610000 >>>>> Aug 8 02:51:14 vmmysql kernel: RIP: 00

2 1

Re: Need to enable STP on ovirt bridges
by Strahil 25 Aug '19

25 Aug '19

It seems that according to https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.0/ht… the ports of interest are: 16514/TCP 49152 - 49216/TCP Best Regards, Strahil NikolovOn Aug 25, 2019 08:48, Strahil <hunter86_bg(a)yahoo.com> wrote: > > Curtis, > > Do you have enough space to run tcpdump (port not 22) on both hosts and on the small VM you have done previously - and then start the migration? > > Best Regards, > Strahil NikolovOn Aug 24, 2019 22:15, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > > > I applied a 90Mbs QOS Rate Limit with 10 set for the shares to both > > interfaces of 2 of the hosts. My hosts names are swm-01 and swm-02. > > > > Creating a small VM from a Cinder template and running it gave me a test VM. > > > > When I migrated from swm-01 to swm-02, swm-01 immediately became > > unresponsive to pings, SSH'es, and to the ovirt interface which marked > > it as "NonResponsive" soon after the VM finished. The VM did finish > > migrating, however I'm unsure if that's a good migration or not. > > > > Thank you, Strahil. > > > > On Sat, Aug 24, 2019 at 12:39 PM Strahil <hunter86_bg(a)yahoo.com> wrote: > > > > > > What is your bandwidth threshold for the network used for VM migration ? > > > Can you set a 90 mbit/s threshold (yes, less than 100mbit/s) and try to migrate a small (1 GB RAM) VM ? > > > > > > Do you see disconnects ? > > > > > > If no, try a little bit up (the threshold) and check again. > > > > > > Best Regards, > > > Strahil NikolovOn Aug 23, 2019 23:19, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > > > > > > > It took a while for my servers to come back on the network this time. > > > > I think it's due to ovirt continuing to try to migrate the VMs around > > > > like I requested. The 3 servers' names are "swm-01, swm-02 and > > > > swm-03". Eventually (about 2-3 minutes ago) they all came back online. > > > > > > > > So I disabled and stopped the lldpad service. > > > > > > > > Nope. Started some more migrations and swm-02 and swm-03 disappeared > > > > again. No ping, SSH hung, same as before - almost as soon as the > > > > migration started. > > > > > > > > If you wall have any ideas what switch-level setting might be enabled, > > > > let me know, cause I'm stumped. I can add it to the ticket that's > > > > requesting the port configurations. I've already added the port > > > > numbers and switch name that I got from CDP. > > > > > > > > Thanks again, I really appreciate the help! > > > > cecjr > > > > > > > > > > > > > > > > On Fri, Aug 23, 2019 at 3:28 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > > > > > > > > > > > > > > > > > On Fri, Aug 23, 2019 at 9:19 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > >> > > > > >> > > > > >> > > > > >> On Fri, Aug 23, 2019 at 8:03 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > > >>> > > > > >>> This little cluster isn't in production or anything like that yet. > > > > >>> > > > > >>> So, I went ahead and used your ethtool commands to disable pause > > > > >>> frames on both interfaces of each server. I then, chose a few VMs to > > > > >>> migrate around at random. > > > > >>> > > > > >>> swm-02 and swm-03 both went out again. Unreachable. Can't ping, can't > > > > >>> ssh, and the SSH session that I had open was unresponsive. > > > > >>> > > > > >>> Any other ideas? > > > > >>> > > > > >> > > > > >> Sorry, no. Looks like two different NICs with different drivers and frimware goes down together. > > > > >> This is a strong indication that the root cause is related to the switch. > > > > >> Maybe you can get some information about the switch config by > > > > >> 'lldptool get-tlv -n -i em1' > > > > >> > > > > > > > > > > Another guess: > > > > > After the optional 'lldptool get-tlv -n -i em1' > > > > > 'systemctl stop lldpad' > > > > > another try to migrate. > > > > > > > > > > > > > > >> > > > > >> > > > > >>> > > > > >>> On Fri, Aug 23, 2019 at 1:50 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > On Fri, Aug 23, 2019 at 6:45 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > > >>> >> > > > > >>> >> Unfortunately, I can't check on the switch. Trust me, I've tried. > > > > >>> >> These servers are in a Co-Lo and I've put 5 tickets in asking about > > > > >>> >> the port configuration. They just get ignored - but that's par for the > > > > >>> >> coarse for IT here. Only about 2 out of 10 of our tickets get any > > > > >>> >> response and usually the response doesn't help. Then

2 1

attach untagged vlan internally on vm
by ernestclydeachua＠gmail.com 25 Aug '19

25 Aug '19

good day currently i am testing oVirt on a single box and setup some tagged vms and non tagged vm. the non tagged vm is a firewall but it has limitations on the number of nic so i cannot attach tagged vnic and wish to handdle vlan tagging on it is it possible to pass untaged franes internally?

6 7

Re: Need to enable STP on ovirt bridges
by Strahil 25 Aug '19

25 Aug '19

Curtis, Do you have enough space to run tcpdump (port not 22) on both hosts and on the small VM you have done previously - and then start the migration? Best Regards, Strahil NikolovOn Aug 24, 2019 22:15, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > I applied a 90Mbs QOS Rate Limit with 10 set for the shares to both > interfaces of 2 of the hosts. My hosts names are swm-01 and swm-02. > > Creating a small VM from a Cinder template and running it gave me a test VM. > > When I migrated from swm-01 to swm-02, swm-01 immediately became > unresponsive to pings, SSH'es, and to the ovirt interface which marked > it as "NonResponsive" soon after the VM finished. The VM did finish > migrating, however I'm unsure if that's a good migration or not. > > Thank you, Strahil. > > On Sat, Aug 24, 2019 at 12:39 PM Strahil <hunter86_bg(a)yahoo.com> wrote: > > > > What is your bandwidth threshold for the network used for VM migration ? > > Can you set a 90 mbit/s threshold (yes, less than 100mbit/s) and try to migrate a small (1 GB RAM) VM ? > > > > Do you see disconnects ? > > > > If no, try a little bit up (the threshold) and check again. > > > > Best Regards, > > Strahil NikolovOn Aug 23, 2019 23:19, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > > > > > It took a while for my servers to come back on the network this time. > > > I think it's due to ovirt continuing to try to migrate the VMs around > > > like I requested. The 3 servers' names are "swm-01, swm-02 and > > > swm-03". Eventually (about 2-3 minutes ago) they all came back online. > > > > > > So I disabled and stopped the lldpad service. > > > > > > Nope. Started some more migrations and swm-02 and swm-03 disappeared > > > again. No ping, SSH hung, same as before - almost as soon as the > > > migration started. > > > > > > If you wall have any ideas what switch-level setting might be enabled, > > > let me know, cause I'm stumped. I can add it to the ticket that's > > > requesting the port configurations. I've already added the port > > > numbers and switch name that I got from CDP. > > > > > > Thanks again, I really appreciate the help! > > > cecjr > > > > > > > > > > > > On Fri, Aug 23, 2019 at 3:28 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > > > > > > > > > > > > > On Fri, Aug 23, 2019 at 9:19 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > >> > > > >> > > > >> > > > >> On Fri, Aug 23, 2019 at 8:03 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > >>> > > > >>> This little cluster isn't in production or anything like that yet. > > > >>> > > > >>> So, I went ahead and used your ethtool commands to disable pause > > > >>> frames on both interfaces of each server. I then, chose a few VMs to > > > >>> migrate around at random. > > > >>> > > > >>> swm-02 and swm-03 both went out again. Unreachable. Can't ping, can't > > > >>> ssh, and the SSH session that I had open was unresponsive. > > > >>> > > > >>> Any other ideas? > > > >>> > > > >> > > > >> Sorry, no. Looks like two different NICs with different drivers and frimware goes down together. > > > >> This is a strong indication that the root cause is related to the switch. > > > >> Maybe you can get some information about the switch config by > > > >> 'lldptool get-tlv -n -i em1' > > > >> > > > > > > > > Another guess: > > > > After the optional 'lldptool get-tlv -n -i em1' > > > > 'systemctl stop lldpad' > > > > another try to migrate. > > > > > > > > > > > >> > > > >> > > > >>> > > > >>> On Fri, Aug 23, 2019 at 1:50 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > >>> > > > > >>> > > > > >>> > > > > >>> > On Fri, Aug 23, 2019 at 6:45 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > >>> >> > > > >>> >> Unfortunately, I can't check on the switch. Trust me, I've tried. > > > >>> >> These servers are in a Co-Lo and I've put 5 tickets in asking about > > > >>> >> the port configuration. They just get ignored - but that's par for the > > > >>> >> coarse for IT here. Only about 2 out of 10 of our tickets get any > > > >>> >> response and usually the response doesn't help. Then

1 0

Re: Need to enable STP on ovirt bridges
by Strahil 25 Aug '19

25 Aug '19

Curtis, Do you have enough space to run tcpdump (port not 22) on both hosts and on the small VM you have done previously - and then start the migration? Best Regards, Strahil NikolovOn Aug 24, 2019 22:15, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > I applied a 90Mbs QOS Rate Limit with 10 set for the shares to both > interfaces of 2 of the hosts. My hosts names are swm-01 and swm-02. > > Creating a small VM from a Cinder template and running it gave me a test VM. > > When I migrated from swm-01 to swm-02, swm-01 immediately became > unresponsive to pings, SSH'es, and to the ovirt interface which marked > it as "NonResponsive" soon after the VM finished. The VM did finish > migrating, however I'm unsure if that's a good migration or not. > > Thank you, Strahil. > > On Sat, Aug 24, 2019 at 12:39 PM Strahil <hunter86_bg(a)yahoo.com> wrote: > > > > What is your bandwidth threshold for the network used for VM migration ? > > Can you set a 90 mbit/s threshold (yes, less than 100mbit/s) and try to migrate a small (1 GB RAM) VM ? > > > > Do you see disconnects ? > > > > If no, try a little bit up (the threshold) and check again. > > > > Best Regards, > > Strahil NikolovOn Aug 23, 2019 23:19, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > > > > > It took a while for my servers to come back on the network this time. > > > I think it's due to ovirt continuing to try to migrate the VMs around > > > like I requested. The 3 servers' names are "swm-01, swm-02 and > > > swm-03". Eventually (about 2-3 minutes ago) they all came back online. > > > > > > So I disabled and stopped the lldpad service. > > > > > > Nope. Started some more migrations and swm-02 and swm-03 disappeared > > > again. No ping, SSH hung, same as before - almost as soon as the > > > migration started. > > > > > > If you wall have any ideas what switch-level setting might be enabled, > > > let me know, cause I'm stumped. I can add it to the ticket that's > > > requesting the port configurations. I've already added the port > > > numbers and switch name that I got from CDP. > > > > > > Thanks again, I really appreciate the help! > > > cecjr > > > > > > > > > > > > On Fri, Aug 23, 2019 at 3:28 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > > > > > > > > > > > > > On Fri, Aug 23, 2019 at 9:19 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > >> > > > >> > > > >> > > > >> On Fri, Aug 23, 2019 at 8:03 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > >>> > > > >>> This little cluster isn't in production or anything like that yet. > > > >>> > > > >>> So, I went ahead and used your ethtool commands to disable pause > > > >>> frames on both interfaces of each server. I then, chose a few VMs to > > > >>> migrate around at random. > > > >>> > > > >>> swm-02 and swm-03 both went out again. Unreachable. Can't ping, can't > > > >>> ssh, and the SSH session that I had open was unresponsive. > > > >>> > > > >>> Any other ideas? > > > >>> > > > >> > > > >> Sorry, no. Looks like two different NICs with different drivers and frimware goes down together. > > > >> This is a strong indication that the root cause is related to the switch. > > > >> Maybe you can get some information about the switch config by > > > >> 'lldptool get-tlv -n -i em1' > > > >> > > > > > > > > Another guess: > > > > After the optional 'lldptool get-tlv -n -i em1' > > > > 'systemctl stop lldpad' > > > > another try to migrate. > > > > > > > > > > > >> > > > >> > > > >>> > > > >>> On Fri, Aug 23, 2019 at 1:50 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > >>> > > > > >>> > > > > >>> > > > > >>> > On Fri, Aug 23, 2019 at 6:45 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > >>> >> > > > >>> >> Unfortunately, I can't check on the switch. Trust me, I've tried. > > > >>> >> These servers are in a Co-Lo and I've put 5 tickets in asking about > > > >>> >> the port configuration. They just get ignored - but that's par for the > > > >>> >> coarse for IT here. Only about 2 out of 10 of our tickets get any > > > >>> >> response and usually the response doesn't help. Then

1 0

Re: Need to enable STP on ovirt bridges
by Strahil 25 Aug '19

25 Aug '19

What is your bandwidth threshold for the network used for VM migration ? Can you set a 90 mbit/s threshold (yes, less than 100mbit/s) and try to migrate a small (1 GB RAM) VM ? Do you see disconnects ? If no, try a little bit up (the threshold) and check again. Best Regards, Strahil NikolovOn Aug 23, 2019 23:19, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > It took a while for my servers to come back on the network this time. > I think it's due to ovirt continuing to try to migrate the VMs around > like I requested. The 3 servers' names are "swm-01, swm-02 and > swm-03". Eventually (about 2-3 minutes ago) they all came back online. > > So I disabled and stopped the lldpad service. > > Nope. Started some more migrations and swm-02 and swm-03 disappeared > again. No ping, SSH hung, same as before - almost as soon as the > migration started. > > If you wall have any ideas what switch-level setting might be enabled, > let me know, cause I'm stumped. I can add it to the ticket that's > requesting the port configurations. I've already added the port > numbers and switch name that I got from CDP. > > Thanks again, I really appreciate the help! > cecjr > > > > On Fri, Aug 23, 2019 at 3:28 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > > > > > On Fri, Aug 23, 2019 at 9:19 PM Dominik Holler <dholler(a)redhat.com> wrote: > >> > >> > >> > >> On Fri, Aug 23, 2019 at 8:03 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > >>> > >>> This little cluster isn't in production or anything like that yet. > >>> > >>> So, I went ahead and used your ethtool commands to disable pause > >>> frames on both interfaces of each server. I then, chose a few VMs to > >>> migrate around at random. > >>> > >>> swm-02 and swm-03 both went out again. Unreachable. Can't ping, can't > >>> ssh, and the SSH session that I had open was unresponsive. > >>> > >>> Any other ideas? > >>> > >> > >> Sorry, no. Looks like two different NICs with different drivers and frimware goes down together. > >> This is a strong indication that the root cause is related to the switch. > >> Maybe you can get some information about the switch config by > >> 'lldptool get-tlv -n -i em1' > >> > > > > Another guess: > > After the optional 'lldptool get-tlv -n -i em1' > > 'systemctl stop lldpad' > > another try to migrate. > > > > > >> > >> > >>> > >>> On Fri, Aug 23, 2019 at 1:50 PM Dominik Holler <dholler(a)redhat.com> wrote: > >>> > > >>> > > >>> > > >>> > On Fri, Aug 23, 2019 at 6:45 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > >>> >> > >>> >> Unfortunately, I can't check on the switch. Trust me, I've tried. > >>> >> These servers are in a Co-Lo and I've put 5 tickets in asking about > >>> >> the port configuration. They just get ignored - but that's par for the > >>> >> coarse for IT here. Only about 2 out of 10 of our tickets get any > >>> >> response and usually the response doesn't help. Then the system they > >>> >> use auto-closes the ticket. That was why I was suspecting STP before. > >>> >> > >>> >> I can do ethtool. I do have root on these servers, though. Are you > >>> >> trying to get me to turn off link-speed auto-negotiation? Would you > >>> >> like me to try that? > >>> >> > >>> > > >>> > It is just a suspicion, that the reason is pause frames. > >>> > Let's start on a NIC which is not used for ovirtmgmt, I guess em1. > >>> > Does 'ethtool -S em1 | grep pause' show something? > >>> > Does 'ethtool em1 | grep pause' indicates support for pause? > >>> > The current config is shown by 'ethtool -a em1'. > >>> > '-A autoneg' "Specifies whether pause autonegotiation should be enabled." according to ethtool doc. > >>> > Assuming flow control is enabled by default, I would try to disable it via > >>> > 'ethtool -A em1 autoneg off rx off tx off' > >>> > and check if it is applied via > >>> > 'ethtool -a em1' > >>> > and check if the behavior under load changes. > >>> > > >>> > > >>> > > >>> >> > >>> >> On Fri, Aug 23, 2019 at 12:24 PM Dominik Holler <dholler(a)redhat.com> wrote: > >>> >> > > >>> >> > > >>> >> > > >>> >> > On Fri, Aug 23, 2019 at 5:49 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > >>> >> >> > >>> >> >> Sure! Right now, I only have a 5

3 2

Moving ovirt engine disk to another storage volume
by Erick Perez - Quadrian Enterprises 25 Aug '19

25 Aug '19

Good morning, I am running Ovirt 4.3.5 in Centos 7.6 with one virt node an a NFS storage node. I did the self-hosted engine setup and i plan to add a second virt host in a few days. I need to do heavy maintenance on the storage node (VDO and mdadm things) and would like to know how (or a link to an article) can I move the ovirt engine disk to another storage. Currentl the NFS storage has two volumens (volA,volB) and the physical host have spare space too. Virtual machines are in VolB and the engine is in VolA. I would like to move the engine disk from VolA to VolB or to local storage. BTW I am not sure if I should say "move the engine" or should I say move "hosted_storage domain" domain thanks in advance. --------------------- Erick Perez

3 6

TASK [ovirt.hosted_engine_setup : Get local VM IP] Problem
by Emre Özkan 24 Aug '19

24 Aug '19

hi guys, ı waiting this part when I try to install with hosted-engine --deploy command.have you got any knowledge about that.My environment(hyper-v running nested vm on azure and rhev running on hyper-v) [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Create local VM] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Get local VM IP]

1 0

numa pinning and reserved hugepages (1G) Bug in scheduler calculation or decision ?
by Ralf Schenk 24 Aug '19

24 Aug '19

Hello List, i ran into problems using numa-pinning and reserved hugepages. - My EPYC 7281 based Servers (Dual Socket) have 8 Numa-Nodes each having 32 GB of memory for a total of 256 GB System Memory - I'm using 192 x 1 GB hugepages reserved on the kernel cmdline default_hugepagesz=1G hugepagesz=1G hugepages=192 This reserves 24 hugepages on each numa-node. I wanted to pin a MariaDB VM using 32 GB (Custom Property hugepages=1048576) to numa-nodes 0-3 of CPU-Socket 1. Pinning in GUI etc. no problem. When trying to start the vm this can't be done since ovirt claims that the host can't fullfill the memory requirements - which is simply not correct since there were > 164 hugepages free. It should have taken 8 hugepages from each numa node 0-3 to fullfill the 32 GB Memory requirement. I also freed the system completely from other VM's but that didn't work either. Is it possible that the scheduler only takes into account the "free memory" (as seen in numactl -H below) *not reserved* by hugepages for its decisions ? Since the host has only < 8 GB of free mem per numa-node I can understand that VM was not able to start under that condition. VM is runnig and using 32 hugepages without pinning but a warning states "VM dbserver01b does not fit to a single NUMA node on host myhost.mydomain.de. This may negatively impact its performance. Consider using vNUMA and NUMA pinning for this VM." This is the numa Hardware Layout and hugepages usage now with other VM's running: from cat /proc/meminfo HugePages_Total: 192 HugePages_Free: 160 HugePages_Rsvd: 0 HugePages_Surp: 0 I can confirm that also under the condition of running other VM's there are at least 8 hugepages free for each numa-node 0-3: grep "" /sys/devices/system/node/*/hugepages/hugepages-1048576kB/free_hugepages /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/free_hugepages:8 /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/free_hugepages:23 /sys/devices/system/node/node2/hugepages/hugepages-1048576kB/free_hugepages:20 /sys/devices/system/node/node3/hugepages/hugepages-1048576kB/free_hugepages:22 /sys/devices/system/node/node4/hugepages/hugepages-1048576kB/free_hugepages:16 /sys/devices/system/node/node5/hugepages/hugepages-1048576kB/free_hugepages:5 /sys/devices/system/node/node6/hugepages/hugepages-1048576kB/free_hugepages:19 /sys/devices/system/node/node7/hugepages/hugepages-1048576kB/free_hugepages:24 numactl -h: available: 8 nodes (0-7) node 0 cpus: 0 1 2 3 32 33 34 35 node 0 size: 32673 MB node 0 free: 3779 MB node 1 cpus: 4 5 6 7 36 37 38 39 node 1 size: 32767 MB node 1 free: 6162 MB node 2 cpus: 8 9 10 11 40 41 42 43 node 2 size: 32767 MB node 2 free: 6698 MB node 3 cpus: 12 13 14 15 44 45 46 47 node 3 size: 32767 MB node 3 free: 1589 MB node 4 cpus: 16 17 18 19 48 49 50 51 node 4 size: 32767 MB node 4 free: 2630 MB node 5 cpus: 20 21 22 23 52 53 54 55 node 5 size: 32767 MB node 5 free: 2487 MB node 6 cpus: 24 25 26 27 56 57 58 59 node 6 size: 32767 MB node 6 free: 3279 MB node 7 cpus: 28 29 30 31 60 61 62 63 node 7 size: 32767 MB node 7 free: 5513 MB node distances: node 0 1 2 3 4 5 6 7 0: 10 16 16 16 32 32 32 32 1: 16 10 16 16 32 32 32 32 2: 16 16 10 16 32 32 32 32 3: 16 16 16 10 32 32 32 32 4: 32 32 32 32 10 16 16 16 5: 32 32 32 32 16 10 16 16 6: 32 32 32 32 16 16 10 16 7: 32 32 32 32 16 16 16 10 -- *Ralf Schenk* fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail *rs(a)databay.de* <mailto:rs@databay.de> *Databay AG* Jens-Otto-Krag-Straße 11 D-52146 Würselen *www.databay.de* <http://www.databay.de> Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen ------------------------------------------------------------------------

2 2

Need to enable STP on ovirt bridges
by ej.albany＠gmail.com 24 Aug '19

24 Aug '19

Hello. I have been trying to figure out an issue for a very long time. That issue relates to the ethernet and 10gb fc links that I have on my cluster being disabled any time a migration occurs. I believe this is because I need to have STP turned on in order to participate with the switch. However, there does not seem to be any way to tell oVirt to stop turning it off! Very frustrating. After entering a cronjob that enables stp on all bridges every 1 minute, the migration issue disappears.... Is there any way at all to do without this cronjob and set STP to be ON without having to resort to such a silly solution? Here are some details about my systems, if you need it. selinux is disabled. [root@swm-02 ~]# rpm -qa | grep ovirt ovirt-imageio-common-1.5.1-0.el7.x86_64 ovirt-release43-4.3.5.2-1.el7.noarch ovirt-imageio-daemon-1.5.1-0.el7.noarch ovirt-vmconsole-host-1.0.7-2.el7.noarch ovirt-hosted-engine-setup-2.3.11-1.el7.noarch ovirt-ansible-hosted-engine-setup-1.0.26-1.el7.noarch python2-ovirt-host-deploy-1.8.0-1.el7.noarch ovirt-ansible-engine-setup-1.1.9-1.el7.noarch python2-ovirt-setup-lib-1.2.0-1.el7.noarch cockpit-machines-ovirt-195.1-1.el7.noarch ovirt-hosted-engine-ha-2.3.3-1.el7.noarch ovirt-vmconsole-1.0.7-2.el7.noarch cockpit-ovirt-dashboard-0.13.5-1.el7.noarch ovirt-provider-ovn-driver-1.2.22-1.el7.noarch ovirt-host-deploy-common-1.8.0-1.el7.noarch ovirt-host-4.3.4-1.el7.x86_64 python-ovirt-engine-sdk4-4.3.2-2.el7.x86_64 ovirt-host-dependencies-4.3.4-1.el7.x86_64 ovirt-ansible-repositories-1.1.5-1.el7.noarch [root@swm-02 ~]# cat /etc/redhat-release CentOS Linux release 7.6.1810 (Core) [root@swm-02 ~]# uname -r 3.10.0-957.27.2.el7.x86_64 You have new mail in /var/spool/mail/root [root@swm-02 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master test state UP group default qlen 1000 link/ether d4:ae:52:8d:50:48 brd ff:ff:ff:ff:ff:ff 3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether d4:ae:52:8d:50:49 brd ff:ff:ff:ff:ff:ff 4: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovirtmgmt state UP group default qlen 1000 link/ether 90:e2:ba:1e:14:80 brd ff:ff:ff:ff:ff:ff 5: p1p2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 90:e2:ba:1e:14:81 brd ff:ff:ff:ff:ff:ff 6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether a2:b8:d6:e8:b3:d8 brd ff:ff:ff:ff:ff:ff 7: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 96:a0:c1:4a:45:4b brd ff:ff:ff:ff:ff:ff 25: test: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether d4:ae:52:8d:50:48 brd ff:ff:ff:ff:ff:ff inet 10.15.11.21/24 brd 10.15.11.255 scope global test valid_lft forever preferred_lft forever 26: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 90:e2:ba:1e:14:80 brd ff:ff:ff:ff:ff:ff inet 10.15.28.31/24 brd 10.15.28.255 scope global ovirtmgmt valid_lft forever preferred_lft forever 27: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 62:e5:e5:07:99:eb brd ff:ff:ff:ff:ff:ff 29: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovirtmgmt state UNKNOWN group default qlen 1000 link/ether fe:6f:9c:95:00:02 brd ff:ff:ff:ff:ff:ff [root@swm-02 ~]# free -m total used free shared buff/cache available Mem: 64413 1873 61804 9 735 62062 Swap: 16383 0 16383 [root@swm-02 ~]# free -h total used free shared buff/cache available Mem: 62G 1.8G 60G 9.5M 735M 60G Swap: 15G 0B 15G [root@swm-02 ~]# ls ls lsb_release lshw lslocks lsmod lspci lssubsys lsusb.py lsattr lscgroup lsinitrd lslogins lsns lss16toppm lstopo-no-graphics lsblk lscpu lsipc lsmem lsof lsscsi lsusb [root@swm-02 ~]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 44 Model name: Intel(R) Xeon(R) CPU X5672 @ 3.20GHz Stepping: 2 CPU MHz: 3192.064 BogoMIPS: 6384.12 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 12288K NUMA node0 CPU(s): 0,2,4,6,8,10,12,14 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid dtherm ida arat spec_ctrl intel_stibp flush_l1d [root@swm-02 ~]#

6 34

Migration from bare metal engine to hosted doesn't seem to work at all in 4.3.2
by Clinton Goudie-Nice 24 Aug '19

24 Aug '19

Hi all, I'm trying to migrate from an ovirt bare metal instance to the hosted appliance. The instructions here: https://www.ovirt.org/documentation/self-hosted/chap-Migrating_from_Bare_Me… are at best murky, and at worst quite wrong, and I'm looking for some help. The instructions say "You must answer No to the following question so that you can restore the BareMetal-Engine backup file on HostedEngine-VM before running engine-setup. Automatically execute engine-setup on the engine appliance on first boot (Yes, No)[Yes]? No" Makes complete sense, except this question is not asked at all. Some research on the internet says we should be able to do this via append-answers: [environment:default] OVEHOSTED_VM/cloudinitExecuteEngineSetup=bool:False Except this doesn't work either. When the appliance deploys the engine is setup and already running. How do I take a backup file, and a backup log, and deploy them into a hosted engine? Clint

1 0

Creating hosts via the REST API using SSH Public Key authentication
by schill-julian＠gmx.de 23 Aug '19

23 Aug '19

In the UI one can create hosts using two authentication methods: 'Password' and 'SSH Public Key'. I have only found the Password authentication in the API Docs (/ovirt-engine/apidoc/#/services/hosts/methods/add). My question is: How can i create hosts using SSH Public Key authentication via the REST API? I would appreciate an example POST request!

3 2

nodectl on plain CentOS hypervisors
by Gianluca Cecchi 23 Aug '19

23 Aug '19

Does it make sense to install nodectl utility on plain CentOS 7.x nodes? Or any other alternative for plain OS nodes vs ovirt-node-ng ones? On my updated CentOS 7.6 oVirt node I have not the command; I think it is provided by the package ovirt-node-ng-nodectl, that is one of the available ones if I run "yum search" on the system. Thanks Gianluca

2 2

oVirt 3.6: Node went into Error state while migrations happening
by Christopher Cox 23 Aug '19

23 Aug '19

On the node in question, the metadata isn't coming across (state) wise. It shows VMs being in an unknown state (some are up and some are down), some show as migrating and there are 9 forever hung migrating tasks. We tried to bring up some of the down VMs that had a state of Down, but that ended up getting them the state of "Wait for Lauch", though those VMs are actually started. Right now, my plan is attempt a restart of vdsmd on the node in question. Just trying to get the node to a working state again. There a total of 9 nodes in our cluster, but we can't manage any VMs on the affected node right now. Is there a way in 3.6 to cancel the hung tasks? I'm worried that if vdsmd is restarted on the node, the tasks might be "attempted"... I really need them to be forgotten if possible. Ideally want all "Unknown" to return to either an "up" or "down" state (depending if the VM is up or down) and for "Wait for Launch" for those, to go to "up" and for all the "Migrating" to go to "up" or "down" (I think only one is actually down). I'm concerned that any attempt manually maniplate the state in the ovirt mgmt head db will be moot because the node will be queried for state and that state will be taken and override anything I attempt to do. Thoughts??

2 1

When I create a new domain in storage , it report : VDSM command ActivateStorageDomainVDS failed: Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842', )
by wangyu13476969128＠126.com 23 Aug '19

23 Aug '19

The version of ovirt-engine is 4.2.8 The version of ovirt-node is 4.2.8 When I create a new domain in storage , the storage type is NFS , it report : VDSM command ActivateStorageDomainVDS failed: Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842',) The error of vdsm.log is : 2019-08-23 11:02:14,740+0800 INFO (jsonrpc/4) [vdsm.api] START connectStorageServer(domType=1, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'id': u'c6893a09-ab28-4328-b186-d2f88f2320d4', u'connection': u'172.16.10.74:/ovirt-data', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', u'port': u''}], options=None) from=::ffff:172.16.90.10,52962, flow_id=5d720a32, task_id=2af924ff-37a6-46a1-b79f-4251d21d5ff9 (api:46) 2019-08-23 11:02:14,743+0800 INFO (jsonrpc/4) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'c6893a09-ab28-4328-b186-d2f88f2320d4'}]} from=::ffff:172.16.90.10,52962, flow_id=5d720a32, task_id=2af924ff-37a6-46a1-b79f-4251d21d5ff9 (api:52) 2019-08-23 11:02:14,743+0800 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.00 seconds (__init__:573) 2019-08-23 11:02:14,751+0800 INFO (jsonrpc/6) [vdsm.api] START activateStorageDomain(sdUUID=u'3bcdc32c-040e-4a4c-90fb-de950f54f1b4', spUUID=u'b87012a1-8f7a-4af5-8884-e0fb8002e842', options=None) from=::ffff:172.16.90.10,53080, flow_id=709ee722, task_id=9b8f32af-5fdd-4ffa-9520-c474af03db70 (api:46) 2019-08-23 11:02:14,752+0800 INFO (jsonrpc/6) [vdsm.api] FINISH activateStorageDomain error=Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842',) from=::ffff:172.16.90.10,53080, flow_id=709ee722, task_id=9b8f32af-5fdd-4ffa-9520-c474af03db70 (api:50) 2019-08-23 11:02:14,752+0800 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='9b8f32af-5fdd-4ffa-9520-c474af03db70') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1262, in activateStorageDomain pool = self.getPool(spUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 350, in getPool raise se.StoragePoolUnknown(spUUID) StoragePoolUnknown: Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842',) 2019-08-23 11:02:14,752+0800 INFO (jsonrpc/6) [storage.TaskManager.Task] (Task='9b8f32af-5fdd-4ffa-9520-c474af03db70') aborting: Task is aborted: "Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842',)" - code 309 (task:1181) 2019-08-23 11:02:14,752+0800 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH activateStorageDomain error=Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842',) (dispatcher:82) How can I solve this problem ?

2 1

large size OVA import fails
by jason.l.cox＠l3harris.com 22 Aug '19

22 Aug '19

I have a fairly large OVA (~200GB) that was exported from oVirt4.3.5. I'm trying to import it into a new cluster, also oVirt4.3.5. The import starts fine but fails again and again. Everything I can find online appears to be outdated, mentioning incorrect log file locations and saying virt-v2v does the import. On the engine in /var/log/ovirt-engine/engine.log I can see where it is doing the CreateImageVDSCommand, then a few outputs concerning adding the disk, which end with USER_ADD_DISK_TO_VM_FINISHED_SUCCESS, then the ansible command: 2019-08-20 15:40:38,653-04 Executing Ansible command: /usr/bin/ansible-playbook --ssh-common-args=-F /var/lib/ovirt-engine/.ssh/config -v --private-key=/etc/pki/ovirt-engine/keys/engine_id_rsa --inventory=/tmp/ansible-inventory8416464991088315694 --extra-vars=ovirt_import_ova_path="/mnt/vm_backups/myvm.ova" --extra-vars=ovirt_import_ova_disks="['/rhev/data-center/mnt/glusterSD/myhost.mydomain.com:_vmstore/59502c8b-fd1e-482b-bff7-39c699c196b3/images/886a3313-19a9-435d-aeac-64c2d507bb54/465ce2ba-8883-4378-bae7-e231047ea09d']" --extra-vars=ovirt_import_ova_image_mappings="{}" /usr/share/ovirt-engine/playbooks/ovirt-ova-import.yml [Logfile: /var/log/ovirt-engine/ova/ovirt-import-ova-ansible-20190820154038-myhost.mydomain.com-25f6ac6f-9bdc-4301-b896-d357712dbf01.log] Then nothing about the import until: 2019-08-20 16:11:08,859-04 INFO [org.ovirt.engine.core.bll.exportimport.ImportVmFromOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-88) [3321d4f6] Lock freed to object 'EngineLock:{exclusiveLocks='[myvm=VM_NAME, 464a25ba-8f0a-421d-a6ab-13eff67b4c96=VM]', sharedLocks=''}' 2019-08-20 16:11:08,894-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-88) [3321d4f6] EVENT_ID: IMPORTEXPORT_IMPORT_VM_FAILED(1,153), Failed to import Vm myvm to Data Center Default, Cluster Default I've found the import logs on the engine, in /var/log/ovirt-engine/ova, but the ovirt-import-ova-ansible*.logs for the imports of concern only contain: 2019-08-20 19:59:48,799 p=44701 u=ovirt | Using /usr/share/ovirt-engine/playbooks/ansible.cfg as config file 2019-08-20 19:59:49,271 p=44701 u=ovirt | PLAY [all] ********************************************************************* 2019-08-20 19:59:49,280 p=44701 u=ovirt | TASK [ovirt-ova-extract : Run extraction script] ******************************* Watching the host selected for the import, I can see the qemu-img convert process running, but then the engine frees the lock on the VM and reports the import as having failed. However, the qemu-img process continues to run on the host. I don't know where else to look to try and find out what's going on and I cannot see anything that says why the import failed. Since the qemu-img process on the host is still running after the engine log shows the lock has been freed and import failed, I'm guessing what's happening is on the engine side. Looking at the time between the start of the ansible command and when the lock is freed it is consistently around 30 minutes. # first try 2019-08-20 15:40:38,653-04 ansible command start 2019-08-20 16:11:08,859-04 lock freed 31 minutes # second try 2019-08-20 19:59:48,463-04 ansible command start 2019-08-20 20:30:21,697-04 lock freed 30 minutes, 33 seconds # third try 2019-08-21 09:16:42,706-04 ansible command start 2019-08-21 09:46:47,103-04 lock freed 30 minutes, 45 seconds With that in mind, I took a look at the available configuration keys from engine-config --list. After getting each the only one set to ~30 minutes and looks like it could be the problem was SSHInactivityHardTimeoutSeconds (set to 1800 seconds). I set it to 3600 and tried the import again, but it still failed at ~30 minutes, so that's apparently not the correct key. Also, just FYI, I also tried to import the ova using virt-v2v, but that fails immediately: virt-v2v: error: expecting XML expression to return an integer (expression: rasd:Parent/text(), matching string: 00000000-0000-0000-0000-000000000000) If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] Does virt-v2v not support OVAs created by the oVirt 'export to ova' option? So my main question is, Is there a timeout for VM imports through the engine web UI? And if so, is if configurable? Thanks in advance.

2 1

ovn networking
by Staniforth, Paul 22 Aug '19

22 Aug '19

Hello I the latest release of the engine 4.3.5.5-1.el7 ovn-nbctl show ovn-sbctl show both seem to work but produce the errors net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5) PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4) should libibverbs be a dependency? Paul S. To view the terms under which this email is distributed, please go to:- http://leedsbeckett.ac.uk/disclaimer/email/

3 5

VM won't migrate back to original node
by Alexander Schichow 22 Aug '19

22 Aug '19

I set up a high available VM which is using a shareable LUN. When migrating said VM from the original Node everything is fine. When I try to migrate it back, I get an error. If I suspend or shutdown the VM, it returns to the original host. Then it repeats. I'm really out of ideas so maybe any of you happen to have any? I would really appreciate that. Thanks

2 1

Custum driver in oVirt node image?
by thomas＠hoberg.net 22 Aug '19

22 Aug '19

Using 3 Mini-ITX Atoms J5005 boards to build a three node HC test environment (actually there is a fourth to simulate a remote DC) These boards have a RealTek 8169 Gbit controller, which for hardware/firmware or driver reasons comes up just fine on cold boots, but struggles on warm-boots or whenever the interface is re-configured, as is the case when the OVS overlay is configured, ... * unless link-speed autonegotiation is enabled after which it works just fine. So for a normal system, I can just put a proper ETHTOOL_OPTS="autoneg on" into /etc/sysconfig/network-scrkipts/ifcfg-<device-name> to avoid trouble on warm boots, but once the overlay network is put on top, the network manager no longer controls the device and the standard "ignore" setting of driver/hardware evidently causes the link to fail: dmesg reports the interface toggling and ovs isn't happy either. The Ethernet switch is unmanaged, so there is nothing I can do there to speed up or eliminate negotiations, and I currently see three options: 1. Find/be told where to renable Ethernet link speed autonegotiation with the OVS/VDSM/?? so the RTL8169 is happy again 2. Use a known USB3 GBit NIC (no space in the chassis to put e.g. an Intel NIC) 3. Make an USB3 2.5Gbit NIC based on RealTek's 8156 work, but currently that requires compiling from source and loading it at boot Option 3 works just fine with a CentOS base, but I got tons of problem making oVirt hosted engine work with that. Option 3 for the oVirt node image seems to class with the read-only nature of the oVirt node imge: While I can load a pre-compile driver interactively, I can't make it stick with dracut -f. I haven't immediately found a way to patch/fix/build derivative oVirt node images with that driver included and am somewhat to impatient to wait until it's mainstream, so any help would be appreciated. I've verified option 2 with a single USB3 GBit adapter natively supported by the kernel that I already had available, but I then opted to jump the gun and buy the slightly more expensive 2.5GBit variant as a better match to a Gluster environment: A mistake in this context as it turned out.

2 2

[ANN] oVirt 4.3.6 Third Release Candidate is now available for testing
by Sandro Bonazzola 22 Aug '19

22 Aug '19

The oVirt Project is pleased to announce the availability of the oVirt 4.3.6 Third Release Candidate for testing, as of August 22nd, 2019. This update is a release candidate of the sixth in a series of stabilization updates to the 4.3 series. This is pre-release software. This pre-release should not to be used in production. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) * oVirt Node 4.3 (available for x86_64 only) will be made available when CentOS 7.7 will be released. See the release notes [1] for known issues, new features and bugs fixed. Notes: - oVirt Appliance is already available - oVirt Node is not yet available, pending CentOS 7.7 release to be available Additional Resources: * Read more about the oVirt 4.3.6 release highlights: http://www.ovirt.org/release/4.3.6/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.6/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

Re: Where to configure iscsi initiator name?
by Gianluca Cecchi 22 Aug '19

22 Aug '19

On Wed, Aug 21, 2019 at 7:07 PM Dan Poltawski <dan.poltawski(a)tnp.net.uk> wrote: > When I added the first node a 'random' initiator name was generated of > form: > > # cat /etc/iscsi/initiatorname.iscsi > InitiatorName=iqn.1994-05.com.redhat:[RANDOM] > > Having attempted to add another node, this node has another initiator name > generated and can't access the storage. Is there a way to configure this > initiator name to a static value which will be configured when new nodes > get added to the cluster? Or is there some reason for this i'm missing? > > thanks, > > Dan > oVirt nodes are iSCSI clients and each node needs to have a different InitiatorName value. Eventually you can modify initiatorname of a node before running discovery against oVirt and rebooting it (or restarting iscsi services). You can edit the file or use the iscsi-iname command. It must be named with an iqn name in the format of iqn.YYYY-MM.reverse.domain.name:OptionalIdentifier Typically you configure your iSCSI storage array giving access to the lun to be used as a storage domain to all oVirt hosts initiators. Or you can use chap authentication or both Se also here: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/ht… https://www.certdepot.net/rhel7-configure-iscsi-target-initiator-persistent… HIH, Gianluca

1 0

Data Center Contending/Non responsive - No SPM
by Matthew B 21 Aug '19

21 Aug '19

Hello, I'm having a problem with oVirt - we're running 4.3 on CentOS 7: [root@ovirt ~]# rpm -q ovirt-engine; uname -r ovirt-engine-4.3.4.3-1.el7.noarch 3.10.0-957.21.2.el7.x86_64 Currently the Data Center alternates between Non responsive and contending status, and SPM selection fails. The error in the events tab is: VDSM compute5.domain command HSMGetTaskStatusVDS failed: Volume does not exist: (u'2bffb8d0-dfb5-4b08-9c6b-716e11f280c2',) Full error: 2019-08-21 13:53:53,507-0700 ERROR (tasks/8) [storage.TaskManager.Task] (Task='3076bb8c-7462-4573-a832-337da478ae0e') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 333, in startSpm self._upgradePool(expectedDomVersion, __securityOverride=True) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 484, in _upgradePool str(targetDomVersion)) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1108, in _convertDomain targetFormat) File "/usr/lib/python2.7/site-packages/vdsm/storage/formatconverter.py", line 447, in convert converter(repoPath, hostId, imageRepo, isMsd) File "/usr/lib/python2.7/site-packages/vdsm/storage/formatconverter.py", line 405, in v5DomainConverter domain.convert_volumes_metadata(target_version) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 813, in convert_volumes_metadata for vol in self.iter_volumes(): File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 764, in iter_volumes yield self.produceVolume(img_id, vol_id) File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 846, in produceVolume volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/glusterVolume.py", line 45, in __init__ volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 817, in __init__ self._manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line 71, in __init__ volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 86, in __init__ self.validate() File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 112, in validate self.validateVolumePath() File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line 131, in validateVolumePath raise se.VolumeDoesNotExist(self.volUUID) VolumeDoesNotExist: Volume does not exist: (u'2bffb8d0-dfb5-4b08-9c6b-716e11f280c2',) Most of our VMs are still running, but we can't start or restart any VMs. All of the domains show as down since SPM selection fails. Any thoughts? Thanks, -Matthew

1 0

Re: Where to configure iscsi initiator name?
by Vincent Royer 21 Aug '19

21 Aug '19

You have to add all the initiators for each node to the iscsi LUN. At least thats how I got it working. Then I think it's recommended to setup multipath. However take this with a grain because iscsi did not work well for me. On Wed, Aug 21, 2019, 10:08 AM Dan Poltawski <dan.poltawski(a)tnp.net.uk> wrote: > When I added the first node a 'random' initiator name was generated of > form: > > # cat /etc/iscsi/initiatorname.iscsi > InitiatorName=iqn.1994-05.com.redhat:[RANDOM] > > Having attempted to add another node, this node has another initiator name > generated and can't access the storage. Is there a way to configure this > initiator name to a static value which will be configured when new nodes > get added to the cluster? Or is there some reason for this i'm missing? > > thanks, > > Dan > ------------------------------ > > The Networking People (TNP) Limited. Registered office: Network House, > Caton Rd, Lancaster, LA1 3PE. Registered in England & Wales with company > number: 07667393 > > This email and any files transmitted with it are confidential and intended > solely for the use of the individual or entity to whom they are addressed. > If you have received this email in error please notify the system manager. > This message contains confidential information and is intended only for the > individual named. If you are not the named addressee you should not > disseminate, distribute or copy this e-mail. Please notify the sender > immediately by e-mail if you have received this e-mail by mistake and > delete this e-mail from your system. If you are not the intended recipient > you are notified that disclosing, copying, distributing or taking any > action in reliance on the contents of this information is strictly > prohibited. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/UKNN77NZJ4ZTB… >

1 0

hosted engine setup, iSCSI no LUNs shown
by billyburly＠gmail.com 21 Aug '19

21 Aug '19

I'm trying to setup the hosted engine on top of iSCSI storage. It successfully logs in and gets the target, however the process errors out claiming there are no LUNs. But if you look on the host, the disks were added to the system. [ INFO ] TASK [ovirt.hosted_engine_setup : iSCSI discover with REST API] [ INFO ] ok: [localhost] The following targets have been found: [1] iqn.2001-04.com.billdurr.durrnet.vm-int:vmdata TPGT: 1, portals: 192.168.47.10:3260 Please select a target (1) [1]: 1 [ INFO ] Getting iSCSI LUNs list ... [ INFO ] TASK [ovirt.hosted_engine_setup : Get iSCSI LUNs] [ INFO ] ok: [localhost] [ ERROR ] Cannot find any LUN on the selected target [ ERROR ] Unable to get target list Here's what the config in targetcli looks like [root@vm1 ~]# targetcli ls o- / ..................................................................... [...] o- backstores .......................................................... [...] | o- block .............................................. [Storage Objects: 2] | | o- p_iscsi_lun1 .............. [/dev/drbd0 (62.0GiB) write-thru activated] | | | o- alua ............................................... [ALUA Groups: 1] | | | o- default_tg_pt_gp ................... [ALUA state: Active/optimized] | | o- p_iscsi_lun2 ............. [/dev/drbd1 (310.6GiB) write-thru activated] | | o- alua ............................................... [ALUA Groups: 1] | | o- default_tg_pt_gp ................... [ALUA state: Active/optimized] | o- fileio ............................................. [Storage Objects: 0] | o- pscsi .............................................. [Storage Objects: 0] | o- ramdisk ............................................ [Storage Objects: 0] o- iscsi ........................................................ [Targets: 1] | o- iqn.2001-04.com.billdurr.durrnet.vm-int:vmdata ................ [TPGs: 1] | o- tpg1 .............................................. [gen-acls, no-auth] | o- acls ...................................................... [ACLs: 0] | o- luns ...................................................... [LUNs: 2] | | o- lun0 ......... [block/p_iscsi_lun1 (/dev/drbd0) (default_tg_pt_gp)] | | o- lun1 ......... [block/p_iscsi_lun2 (/dev/drbd1) (default_tg_pt_gp)] | o- portals ................................................ [Portals: 1] | o- 192.168.47.10:3260 ........................................... [OK] o- loopback ..................................................... [Targets: 0] o- srpt ......................................................... [Targets: 0] The two LUNs show up on the host after the hosted engine setup tries to enumerate the LUNs for the target [root@vm1 ~]# lsscsi [0:0:0:0] storage HP P420i 8.32 - [0:1:0:0] disk HP LOGICAL VOLUME 8.32 /dev/sda [0:1:0:1] disk HP LOGICAL VOLUME 8.32 /dev/sdb [0:1:0:2] disk HP LOGICAL VOLUME 8.32 /dev/sdc [11:0:0:0] disk LIO-ORG p_iscsi_lun1 4.0 /dev/sdd [11:0:0:1] disk LIO-ORG p_iscsi_lun2 4.0 /dev/sde

3 2

Reaction and timeouts to APD (All Paths Down)
by Gianluca Cecchi 21 Aug '19

21 Aug '19

Hello, similar questions/arguments took already place in the past, but I think it could be a good point to dig more if possible. I focus on block based storage and my environments have iSCSI based with multipath, connected to Equallogic. Currently I have set no_path_retry to 4, so I have a 20 seconds timeout (polling_interval=5). Sometimes some network planned activity (not under my domain) that should have no impact "doesn't go so well" and I could have longer delays (also 60-70 seconds) and so oVirt reactions where I see soft fencing, VMs going into paused state or "question mark" state... Comparing to vSphere, the same events don't apparently cause anything (and I see same lost paths events in datastore and ESXi host monitoring--> Events pane). This basically depends on APD default timeout that seems to be 140 seconds. This difference of behavior has the effect of showing an apparent better SLA of vSphere during these short time outages and a sort of show stopper for extending oVirt implementation.... Interesting vSphere KBs here: Storage device has entered the All Paths Down state (2032934) https://kb.vmware.com/s/article/2032934 Containing: "Note: By default, the APD timeout is set to 140 seconds." All Paths Down timeout for a storage device has expired (2032940) https://kb.vmware.com/s/article/2032940 Path redundancy to the storage device is degraded (1009555) https://kb.vmware.com/s/article/1009555 Storage device has recovered from the APD state (2032945) https://kb.vmware.com/s/article/2032945 So the question is what kind of real risks I have if I simulate that behavior and set a no_path_retry value so that polling_interval x no_path_retry = 140 (in default config of polling_interval=5 it would mean no_path_retry = 28) BTW: I also have an environment based on RHV 4.3.5 and iSCSI and in parallel I opened a case (02452597) for asking clarifications/chances remaining in supported configuration, so for logs and so on it could help going into it for Red Hat developers Thanks, Gianluca

1 0

Where to configure iscsi initiator name?
by Dan Poltawski 21 Aug '19

21 Aug '19

When I added the first node a 'random' initiator name was generated of form: # cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1994-05.com.redhat:[RANDOM] Having attempted to add another node, this node has another initiator name generated and can't access the storage. Is there a way to configure this initiator name to a static value which will be configured when new nodes get added to the cluster? Or is there some reason for this i'm missing? thanks, Dan ________________________________ The Networking People (TNP) Limited. Registered office: Network House, Caton Rd, Lancaster, LA1 3PE. Registered in England & Wales with company number: 07667393 This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

1 0

engine-setup is failing in fedora30
by Gobinda Das 21 Aug '19

21 Aug '19

Hi, I am facing issue to do engine-setup in fedora30. [godas@godas ovirt-engine]$ $HOME/ovirt-engine/bin/engine-setup [ INFO ] Stage: Initializing Setup was run under unprivileged user this will produce development installation do you wish to proceed? (Yes, No) [No]: Yes [ INFO ] Stage: Environment setup Configuration files: ['/home/gobinda/ovirt-engine/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /home/gobinda/ovirt-engine/var/log/ovirt-engine/setup/ovirt-engine-setup-20190821121001-bkc6we.log Version: otopi-1.8.1_master (otopi-1.8.1-0.0.master.20190228094531.git1953bdc.fc30) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup (late) [ INFO ] Stage: Environment customization --== PRODUCT OPTIONS ==-- --== PACKAGES ==-- --== NETWORK CONFIGURATION ==-- [WARNING] Failed to resolve godas.lab.eng.blr.redhat.com using DNS, it can be resolved only locally --== DATABASE CONFIGURATION ==-- --== OVIRT ENGINE CONFIGURATION ==-- Perform full vacuum on the engine database engine@localhost? This operation may take a while depending on this setup health and the configuration of the db vacuum process. See https://www.postgresql.org/docs/10/sql-vacuum.html (Yes, No) [No]: --== STORAGE CONFIGURATION ==-- --== PKI CONFIGURATION ==-- --== APACHE CONFIGURATION ==-- --== SYSTEM CONFIGURATION ==-- --== MISC CONFIGURATION ==-- --== END OF CONFIGURATION ==-- [ INFO ] Stage: Setup validation --== CONFIGURATION PREVIEW ==-- Default SAN wipe after delete : False Host FQDN : godas.lab.eng.blr.redhat.com Set up Cinderlib integration : False Engine database host : localhost Engine database port : 5432 Engine database secured connection : False Engine database host name validation : False Engine database name : engine Engine database user name : engine Engine installation : True PKI organization : lab.eng.blr.redhat.com Set up ovirt-provider-ovn : False Configure WebSocket Proxy : True Configure VMConsole Proxy : True Please confirm installation settings (OK, Cancel) [OK]: OK [ INFO ] Checking the Engine database consistency [ INFO ] Stage: Transaction setup [ INFO ] Stage: Misc configuration (early) [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ INFO ] Upgrading CA [ INFO ] Configuring WebSocket Proxy [ INFO ] Backing up database localhost:engine to '/home/gobinda/ovirt-engine/var/lib/ovirt-engine/backups/engine-20190821121010.0iip56rw.dump'. [ INFO ] Creating/refreshing Engine database schema Unregistering existing client registration info. [ INFO ] Generating post install configuration file '/home/gobinda/ovirt-engine/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf' [ ERROR ] Failed to execute stage 'Misc configuration': a bytes-like object is required, not 'str' [ INFO ] Rolling back database schema [ INFO ] Clearing Engine database engine [ INFO ] Restoring Engine database engine [ INFO ] Restoring file '/home/gobinda/ovirt-engine/var/lib/ovirt-engine/backups/engine-20190821121010.0iip56rw.dump' to database localhost:engine. [ ERROR ] Engine database rollback failed: cannot use a string pattern on a bytes-like object [ INFO ] Stage: Clean up Log file is located at /home/gobinda/ovirt-engine/var/log/ovirt-engine/setup/ovirt-engine-setup-20190821121001-bkc6we.log [ INFO ] Generating answer file '/home/gobinda/ovirt-engine/var/lib/ovirt-engine/setup/answers/20190821121039-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed I*t was working before but the moment i pulled latest master code the issue started.* -- Thanks, Gobinda

2 1

When I added the ovirt-node to the ovirt-engine through the web management interface, I get the following error: Host 172.16.41.100 installation failed. Failed to configure management network on the host. Host 172.16.41.100 does not comply with the cluster cluster-kvm networks, the following networks are missing on host: 'ovirtmgmt'
by wangyu13476969128＠126.com 21 Aug '19

21 Aug '19

When I added the ovirt-node to the ovirt-engine through the web management interface, I get the following error: Host 172.16.41.100 installation failed. Failed to configure management network on the host. Host 172.16.41.100 does not comply with the cluster cluster-kvm networks, the following networks are missing on host: 'ovirtmgmt' How can I solve this problem ?

3 12

VDSM ovirt-node-2 command Get Host Capabilities failed: Internal JSON-RPC error: {'reason': "invalid argument: KVM is not supported by '/usr/libexec/qemu-kvm' on this host"}
by wangyu13476969128＠126.com 21 Aug '19

21 Aug '19

The version of ovirt-engine is 4.3.5.5-1.el7 The version of ovirt-node-2 is 4.3.5.2-1.el7 When I add ovirt-node-2 to ovirt-engine ，it report: VDSM ovirt-node-2 command Get Host Capabilities failed: Internal JSON-RPC error: {'reason': "invalid argument: KVM is not supported by '/usr/libexec/qemu-kvm' on this host"} What 's the root cause of this problem ? How can I solve this problem ?

2 2

Hyperconverged setup ovirt 4.3.x using ansible?
by adrianquintero＠gmail.com 20 Aug '19

20 Aug '19

Hi, I am trying to do a Hyperconverged setup using ansible. So far I have been able to run a playbook to setup gluster but have not been able to identify how to install the Hosted-Engine VM using ansible and tie all the pieces together. Can someone point me in the right direction to deploy a hyperconverged environment using ansible? I have succesfully deployed ovirt 4.3.5 (Hyperconverged) using the web UI. thanks.

3 4

ovirt node install
by Staniforth, Paul 20 Aug '19

20 Aug '19

Hello, on the latest version of the oVirt-node install running nodectl info gives the error Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/lib/python2.7/site-packages/nodectl/__main__.py", line 42, in <module> CliApplication() File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 200, in CliApplication return cmdmap.command(args) File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 118, in command return self.commands[command](**kwargs) File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 76, in info Info(self.imgbased, self.machine).write() File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 46, in __init__ self._fetch_information() File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 49, in _fetch_information self._get_bootloader_info() File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 62, in _get_bootloader_info bootinfo["entries"][k] = v.__dict__ AttributeError: 'list' object has no attribute '__dict__' Also what is the correct way to update from ovirt node 4.2 to 4.3? I used yum install https://resources.ovirt.org/pub/ovirt-4.3/rpm/el7/noarch/ovirt-node-ng-imag… I then did yum erase ovirt-release42 and rm /etc/yum.repos.d/ovirt-4.2* Regards, Paul S. To view the terms under which this email is distributed, please go to:- http://leedsbeckett.ac.uk/disclaimer/email/

3 5

numa pinning for high performance
by Vincent Royer 20 Aug '19

20 Aug '19

Is there a good place to learn about cpu pinning and the various settings? I've built a test VM and am trying to understand - I've given it 2 cpus, 4 cores each, 1 thread per core. - The VM is pinned to a specific host with passthrough cpu enabled - numa pinning, first 4 cores on physical socket 1 and the next 4 on physical socket 2 - The host has two sockets each with 20 core cpus I would expect to see a load applied on 8 of the 40 cores on the host during testing. Instead it seems like 32 of 40 cores are working at 100%. What are the effects of this on the performance of the vm? [image: image.png] [image: image.png] [image: image.png] [image: image.png] [image: image.png] [image: image.png]

3 3

Export as OVA fails
by anthonywest＠alfatron.com.au 20 Aug '19

20 Aug '19

Hi. We are running oVirt 4.3.4 with each of the hosts configured to use local storage. When I attempt to export a large virtual machine as an OVA the process begins but fails about half an hour later. On the oVirt Events page the following entries appear: Starting to export Vm Jessica as a Virtual Appliance Failed to export Vm Jessica as a Virtual Appliance to path /storage/Jessica.ova on Host Trueblood When I log onto the host Trueblood I see that a file called Jessica.ova.tmp has been created. The engine.log file contains the following errors: 2019-08-16 16:40:27,141+10 ERROR [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-engineScheduled-Thread-37) [80ae4c6] Ansible playbook execution failed: Timeout occurred while executing Ansible playbook. 2019-08-16 16:40:27,142+10 ERROR [org.ovirt.engine.core.bll.CreateOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-37) [80ae4c6] Failed to create OVA. Please check logs for more details: /var/log/ovirt-engine/ova/ovirt-export-ova-ansible-20190816161027-trueblood.alfatron.com.au-80ae4c6.log 2019-08-16 16:40:27,161+10 ERROR [org.ovirt.engine.core.bll.exportimport.ExportVmToOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-37) [80ae4c6] Failed to create OVA file 2019-08-16 16:40:27,162+10 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-37) [80ae4c6] Command 'ExportVmToOva' id: '12e98544-67ea-45b0-adb8-c50c8c190ecf' with children [a6c04b63-fe81-4d13-a69f-87b8ac55ceaa] failed when attempting to perform the next operation, marking as 'ACTIVE' 2019-08-16 16:40:27,162+10 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-37) [80ae4c6] EngineException: ENGINE (Failed with error ENGINE and code 5001): org.ovirt.engine.core.common.errors.EngineException: EngineException: ENGINE (Failed with error ENGINE and code 5001) at org.ovirt.engine.core.bll.exportimport.ExportOvaCommand.createOva(ExportOvaCommand.java:117) [bll.jar:] at org.ovirt.engine.core.bll.exportimport.ExportVmToOvaCommand.executeNextOperation(ExportVmToOvaCommand.java:224) [bll.jar:] at org.ovirt.engine.core.bll.exportimport.ExportVmToOvaCommand.performNextOperation(ExportVmToOvaCommand.java:216) [bll.jar:] at org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback.childCommandsExecutionEnded(SerialChildCommandsExecutionCallback.java:32) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:77) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:175) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) [bll.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_212] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_212] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_212] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78) 2019-08-16 16:40:28,304+10 ERROR [org.ovirt.engine.core.bll.exportimport.ExportVmToOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-10) [3f52073f-b7d2-4238-8ac1-4634678b11b4] Ending command 'org.ovirt.engine.core.bll.exportimport.ExportVmToOvaCommand' with failure. 2019-08-16 16:40:28,745+10 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-10) [41e553bf] EVENT_ID: IMPORTEXPORT_EXPORT_VM_TO_OVA_FAILED(1,225), Failed to export Vm Jessica as a Virtual Appliance to path /storage/Jessica.ova on Host Trueblood Any suggestions/help on solving this would be much appreciated. Anthony

2 1

Issues with oVirt-Engine start - oVirt 4.3.4
by Vrgotic, Marko 19 Aug '19

19 Aug '19

Dear oVirt, While working on a procedure to get the NFS v4 mount from Netapp, working on oVIrt, following steps came out to be the way to go in regards of setting it up for oVIrt SHE and VM Guests: * mkdir /mnt/rhevstore * mount -t nfs 10.20.30.40:/ovirt_hosted_engine /mnt/rhevstore * chown -R 36.36 /mnt/rhevstore * chmod -R 755 /mnt/rhevstore * umount /mnt/rhevstore This works fine, and it needs to be executed on each Hypervisor, before its provisioned into oVirt. However, just today I have discovered that command chmod -R 755 /mnt/rhevstore, if executed on new to be added Hypervisor, after oVirt is already running, it brings the oVirt Engine into broken state. The moment I executed the above on 3rd Hypervisor, before provisioning it into oVirt, following occurred: * Engine threw following error: * 2019-08-19 13:16:31,425Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-82) [] Failed in 'SpmStatusVDS' method * Connection was lost: * packet_write_wait: Connection to 10.210.11.10 port 22: Broken pipe * And VDSM on SHE Hosting hypervisor started logging errors like: * 2019-08-19 15:00:52,340+0000 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) * 2019-08-19 15:00:53,865+0000 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=periodic/2 running <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x7f59442c3d50> at 0x7f59442c3b90> timeout=15, duration=225.00 at 0x7f592476df90> task#=578 at 0x7f59442ef910>, traceback: * File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap * self.__bootstrap_inner() * File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner * self.run() * File: "/usr/lib64/python2.7/threading.py", line 765, in run * self.__target(*self.__args, **self.__kwargs) * File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run * ret = func(*args, **kwargs) * File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run * self._execute_task() * File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task * task() * File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__ * self._callable() * File: "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 186, in __call__ * self._func() * File: "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 481, in __call__ * stats = hostapi.get_stats(self._cif, self._samples.stats()) * File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 79, in get_stats * ret['haStats'] = _getHaInfo() * File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 177, in _getHaInfo * stats = instance.get_all_stats() * File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 94, in get_all_stats * stats = broker.get_stats_from_storage() * File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 143, in get_stats_from_storage * result = self._proxy.get_stats() * File: "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ * return self.__send(self.__name, args) * File: "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request * verbose=self.__verbose * File: "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request * return self.single_request(host, handler, request_body, verbose) * File: "/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request * response = h.getresponse(buffering=True) * File: "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse * response.begin() * File: "/usr/lib64/python2.7/httplib.py", line 444, in begin * version, status, reason = self._read_status() * File: "/usr/lib64/python2.7/httplib.py", line 400, in _read_status * line = self.fp.readline(_MAXLINE + 1) * File: "/usr/lib64/python2.7/socket.py", line 476, in readline * data = self._sock.recv(self._rbufsize) (executor:363) * 2019-08-19 15:00:54,103+0000 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:312) I am unable to boot the Engine VM – it end up in Status ForceStop Hosted-engine –vm-status shows: [root@ovirt-sj-02 ~]# hosted-engine --vm-status The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. But, storage is mounted and reachable, as well as ovirt-ha-agent running: [root@ovirt-sj-02 ~]# systemctl status ovirt-ha-agent ● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring Agent Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2019-08-19 14:57:07 UTC; 23s ago Main PID: 43388 (ovirt-ha-agent) Tasks: 2 CGroup: /system.slice/ovirt-ha-agent.service └─43388 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent Can somebody help me with what to do? — — — Met vriendelijke groet / Kind regards, Marko Vrgotic

1 2

Hosted Engine on seperate L2 network from nodes?
by Dan Poltawski 19 Aug '19

19 Aug '19

For some security requirements, I’ve been asked if it’s possible to segregate the hosted engine from the physical nodes, with specific firewalling for access to do node/ storage operations (I’m using managed block storage). Is this an approach others us, or is it better practice and just ensure the nodes and engine are all sharing the same network? Thanks, dan ________________________________ The Networking People (TNP) Limited. Registered office: Network House, Caton Rd, Lancaster, LA1 3PE. Registered in England & Wales with company number: 07667393 This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

2 1

Adding VLANs to a single-host, self-hosted-engine oVirt deployment?
by Derek Atkins 16 Aug '19

16 Aug '19

HI, I've got a single oVirt host running a self-contained hosted-engine deployment. When I set it up I did not use VLANs in my network. I am in the process of moving my equipment, and in part of this move I would like to introduce VLANs into my network infrastructure. The documentation seems to imply that to add virtual networks and/or VLANs to a host that I need to put it into maintenance mode, configure it in the engine, and then resync the network. However, I don't think I can do that with a single-host environment. If I put the host into local maint mode, it will try to offload all my VMs, including the engine, which obviously it cannot do because there is no other host to migrate them to. So what's the approach to add VLANs in this situation? I should add that this system started at 4.0, and I'm still only running 4.1 (although I do plan to upgrade to 4.2 as part of this move). I'm hesitant to upgrade further because of the impending removal of SDK-3 -- I am depending on a script that uses ovirt-shell which I keep being told is going away. If ovirt-shell is still in 4.3 then I might consider upgrading to that as well. :) Thanks. -derek -- Derek Atkins 617-623-3745 derek(a)ihtfp.com www.ihtfp.com Computer and Internet Security Consultant

2 3

Can't add a pre-existing (storage-only) gluster member as a compute-also node using self-hosted hyper-convergent engine
by thomas＠hoberg.net 16 Aug '19

16 Aug '19

Intro: Trying to add oVirt self-hosted engine to a pre-existing set of system already using Gluster storage seems to fail, because ‘hosted engine wizard’ seems to create its own peer pool for managed hosts, instead of joining the one given for storage. Context: I am testing on a set of four Atom Goldmont+ boxes, which are silent/low-power, cheap, fast-enough for edge workloads and even take 32GB of RAM these days. But for various, sometimes even good reasons, they are not a mainline platform and I attributed many problems I faced to the niche hardware--sometimes correctly. Because the three-node hyperconverged setup has very exacting requirements, hard to meet in pre-existing machines (and in my case many initial failures with too little insight), I created the storage gluster separately first and then used the “hosted engine” wizard to set up the hosted engine on one of the Atom nodes. I used CentOS7 (fresh install and latest updates) on the primary nodes, not the oVirt node image, because some of my targets are big HPC machines, that are supposed to run oVirt for support services in a small niche, while Nvidia-Docker/SLURM workloads dominate. I assumed that if I were to split the storage and the orchestration setup into distinct steps, it would give me both better insight and flexibility to expand/transform the storage and the compute without losing any of the self-hosted hyperconvergeance comfort. Problem: I had all kinds of problem to get the hosted engine to run all the way through on the Atoms, it typically stopped just shy of the final launch as a VM on the Gluster storage. I eventually stumbled across this message from Yedidyah Bar David: https://lists.ovirt.org/pipermail/users/2018-March/087923.html I then had a look at the engine database and found that indeed the compute nodes were all in a separate gluster newly created by the hosted-engine setup and evidently used for cluster synchronization, instead of joining the pool already used for the bulk of the storage. I don’t know if this would be considered an error that needs fixing, an issue that can be avoided using a manual configuration, or something else. I believe it could use some highlighting in the documentation.

1 2

Hyperconverged on a self-made Gluster: Considered normal or 'way-off'?
by thomas＠hoberg.net 16 Aug '19

16 Aug '19

I use three very low power target nodes (J5005 Intel Atoms, 24GB RAM, 250GB SSD) to experiment with hyperconverged oVirt, and some bigger machines to act as pure compute nodes, preferably CentOS based, potentially oVirt Node image based. I've used both CentOS7 and oVirt Node image bases (all latest versions and patches) and typically I use either a single partition and xfs root file system to avoid partitioning the constricted space on CentOS or the recommended partition layout with the oVirt node images, which still consumes the single primary SSD /dev/sda. In both cases I don't have another device or partition left for the three-node hyperconverged wizard's ansible scripts to play with. And since I want to understand better what is going on under the hood in any case I tried to separate the Gluster setup and the hosted-engine setup. Question: Is the impact of that decision perhaps far bigger than I imagined? So I went with the alternative A Hosted-Engine wizard, where the storage was already provisioned and I had set up 2+1 Gluster storage using bricks right on the xfs, assuming I would achieve something very similar to what the three node wizard would simplify, except without the LVM/VDO layer all done with Ansible. But I am running into all kinds of problems and what continues to irritate me is the note in the manuals, that clusters need to be either 'compute' or 'storage' but cannot be both--yet that's the very point of hyperconvergence... So basically I am wondering whether at this point there is a strict bifurifcation of oVirt between the hyperconverged variant and the 'Hosted-Engine' on existing storage, without any middle ground. Or is it ok to do a two-phased approach, where you separate out the Gluster aspect and the Hosted-Engine *but with physical servers that are doing both storage and compute*? Also, I am wondering about the 'Enable Virt Service' and 'Enable Gluster Service' tick boxes in the Cluster property pages: For my 'hosted engine' wizard generated Cluster 'gluster' was unticked, even if it did run on Gluster storage. And while ticking 'gluster' in addition had no visible effect, unticking either one of the two is now impossible. I could not find any documentation on the significance or effect of these boxes. While the prepared three node replicated storage Gluster for the /engine was running on CentOS nodes, the Hosted-Engine VM was then created on additional oVirt Node hosts using that Gluster storage (trying to do the same on CentOS nodes always fails, separate topic), with the assumption that I could then push the hosted-engine VM back to the gluster storage nodes and get rid of the temporary compute-only host used for hosted engine setup. I then tried to add the Gluster nodes as hosts (and hosted-engine backup nodes), that also failed. Whether it failed because they were CentOS based or because they were already used for storage (and there is this hidden exclusivity), I would like to have some clarity on. So to repeat once again my main question: Is the (one or three) node hyperconverged oVirt setup a completely different thing from the hosted-engine on Gluster? Are you supposed to be able to add extra compute hosts, storage etc. to be managed by a three-node HC based hosted-engine? Are you supposed to be able to expand the three-node HC also in terms of storage, if perhaps currently only in quorum-maintaining multiples? Is oVirt currently actually strictly mandating 100% segregation between compute and storage unless you use the wholly contained single/triple HC setups?

1 0

Disk Allocation policy changes after a snapshot
by Kevin Doyle 16 Aug '19

16 Aug '19

Hi I have linux VM with 2 disks one for OS is sparse the other is for a Database and is Preallocated. When I take a snapshot of the VM both disks change to sparse policy but the disks in the snapshot are 1 sparse and 1 allocated. Before the snapshot the VM was running fine, now it crashes when data is written to the database. When I delete the snapshot the disks go back to 1 sparse and 1 allocated. Has anyone else seen this happen. Ovirt is 4.3.2.1-1.el7 and it is running on a hostedEngine Many thanks Kevin

4 5

failed import ova vm
by David David 16 Aug '19

16 Aug '19

hi cant import vmware OVA Engine: 4.3.5.5-1.el7 OS Version: RHEL - 7 - 6.1810.2.el7.centos Kernel Version: 3.10.0 - 957.27.2.el7.x86_64 KVM Version: 2.12.0 - 18.el7_6.7.1 LIBVIRT Version: libvirt-4.5.0-10.el7_6.12 VDSM Version: vdsm-4.30.24-1.el7 WebUI -> Virtual Machines -> Import (select vmware OVA, select OVAs path, Load) - OK(details) - OK vdsm.log, engine.log in attachment

2 1

Inconsistent metadata on VM's disks
by Nardus Geldenhuys 16 Aug '19

16 Aug '19

Hi There Hope you are all well. We got this weird issue at the moment. Let me explain. We are on ovirt 4.3.5. We use Centos 7.6 and when we do an update on the VM's the VM's can not boot into multi user mode. It complains that the root mount can't be found. When you boot using ISO and rescue mode you can see the disks. When you chroot into the VM and run pvscan twice your VM can be rebooted and it is fixed. We don't know if this is Centos based or maybe something on ovirt. Screenshots below. Any help would be appreciated. [image: Screenshot from 2019-08-11 18-15-14.png] [image: Screenshot from 2019-08-12 15-10-53.png] [image: Screenshot from 2019-08-12 14-18-33.png] [image: Screenshot from 2019-08-09 21-54-44.png]

2 1

Looking to install additional packages on Ovirt Node
by krahmani＠ieee.org 15 Aug '19

15 Aug '19

Hello all, Hope everyone is doing well! I would simply like to ask how it would be possible to add additional packages to the Ovirt Centos node. I followed the instructions here: https://serverfault.com/questions/927652/is-it-ok-to-add-the-epel-repo-to-a… but I am still receiving an error that the "rear" package cannot be found (the package I am trying to install, ofcourse). Please help, I appreciate taking the time to read this. Thank you!

2 1

Re: VM --- is not responding.
by Strahil 14 Aug '19

14 Aug '19

Would you check the health status of the controllers : hpssacli ctrl all show status Best Regards, Strahil NikolovOn Aug 11, 2019 09:55, Edoardo Mazza <edo7411(a)gmail.com> wrote: > > The hosts are 3 ProLiant DL380 Gen10, 2 hosts with HPE Smart Array P816i-a SR Gen10 like controller and the other host with > HPE Smart Array P408i-a SR Gen10. The storage for ovirt enviroment is gluster and the last host is the arbiter in the gluster enviroment. > The S.M.A.R.T. healt status is ok for all host > Edoardo > > > > > > Il giorno gio 8 ago 2019 alle ore 16:19 Sandro Bonazzola <sbonazzo(a)redhat.com> ha scritto: >> >> >> >> Il giorno gio 8 ago 2019 alle ore 11:19 Edoardo Mazza <edo7411(a)gmail.com> ha scritto: >>> >>> Hi all, >>> It is more days that for same vm I received this error, but I don't underdand why. >>> The traffic of the virtual machine is not excessive, cpu and ram to, but for few minutes the vm is not responding. and in the messages log file of the vm I received the error under, yo can help me? >>> thanks >> >> >> can you check the S.M.A.R.T. health status of the disks? >> >> >>> >>> Edoardo >>> kernel: NMI watchdog: BUG: soft lockup - CPU#2 stuck for 25s! [kworker/2:0:26227] >>> Aug 8 02:51:11 vmmysql kernel: Modules linked in: binfmt_misc ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ >>> ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_ >>> nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_con >>> ntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter snd_hda_c >>> odec_generic iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel snd_hda_intel snd_hda_codec aesni_intel snd_hda_core lrw gf128mul >>> glue_helper ablk_helper snd_hwdep cryptd snd_seq snd_seq_device snd_pcm snd_timer snd soundcore virtio_rng sg virtio_balloon >>> i2c_piix4 parport_pc parport joydev pcspkr ip_tables xfs libcrc32c sd_mod >>> Aug 8 02:51:14 vmmysql kernel: crc_t10dif crct10dif_generic sr_mod cdrom virtio_net virtio_console virtio_scsi ata_generic p >>> ata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw qxl floppy drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci drm_panel_orientation_quirks virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod >>> Aug 8 02:51:14 vmmysql kernel: CPU: 2 PID: 26227 Comm: kworker/2:0 Kdump: loaded Tainted: G L ------------ 3.10.0-957.12.1.el7.x86_64 #1 >>> Aug 8 02:51:14 vmmysql kernel: Hardware name: oVirt oVirt Node, BIOS 1.11.0-2.el7 04/01/2014 >>> Aug 8 02:51:14 vmmysql kernel: Workqueue: events_freezable disk_events_workfn >>> Aug 8 02:51:14 vmmysql kernel: task: ffff9e25b6609040 ti: ffff9e27b1610000 task.ti: ffff9e27b1610000 >>> Aug 8 02:51:14 vmmysql kernel: RIP: 0010:[<ffffffffb8b6b355>] [<ffffffffb8b6b355>] _raw_spin_unlock_irqrestore+0x15/0x20 >>> Aug 8 02:51:14 vmmysql kernel: RSP: 0000:ffff9e27b1613a68 EFLAGS: 00000286 >>> Aug 8 02:51:14 vmmysql kernel: RAX: 0000000000000001 RBX: ffff9e27b1613a10 RCX: ffff9e27b72a3d05 >>> Aug 8 02:51:14 vmmysql kernel: RDX: ffff9e27b729a420 RSI: 0000000000000286 RDI: 0000000000000286 >>> Aug 8 02:51:14 vmmysql kernel: RBP: ffff9e27b1613a68 R08: 0000000000000001 R09: ffff9e25b67fc198 >>> Aug 8 02:51:14 vmmysql kernel: R10: ffff9e27b45bd8d8 R11: 0000000000000000 R12: ffff9e25b67fde80 >>> Aug 8 02:51:14 vmmysql kernel: R13: ffff9e25b67fc000 R14: ffff9e25b67fc158 R15: ffffffffc032f8e0 >>> Aug 8 02:51:14 vmmysql kernel: FS: 0000000000000000(0000) GS:ffff9e27b7280000(0000) knlGS:0000000000000000 >>> Aug 8 02:51:14 vmmysql kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>> Aug 8 02:51:14 vmmysql kernel: CR2: 00007f0c9e9b6008 CR3: 00

2 1

Either allow 2 CD-ROM's or selectable *.vfd Floppy Images from Storage via Run Once other than from deprcated ISO-Storage
by Ralf Schenk 13 Aug '19

13 Aug '19

Hello, when Installing Windows VM's on Ovirt we need either 2 CD-ROM's attached as ISO Files (Installer ISO and Virtio-Win-ISO) to be able to install to Virtio-(SCSI)-Disks. In Ovirt 4.3.4 it is not possible to attach 2 CD-ROM's to a VM. So we have to use Floppy Images (virtio-win-*.vfd) attached to install drivers within Installer. We need to use "Run Once" to attach flopppy disks. There are only *.vfd selectable which are located on ISO-Storage.Domain, which will be deprecated now or then. -> We won't be able to install Windows VM's from unmodified ISO Installer-CD's without ISO Storage Domain or making *.vfd Files selectable via "Run Once" When will that be available... ? Bye -- *Ralf Schenk* fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail *rs(a)databay.de* <mailto:rs@databay.de> *Databay AG* Jens-Otto-Krag-Straße 11 D-52146 Würselen *www.databay.de* <http://www.databay.de> Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen ------------------------------------------------------------------------

6 5

Re: oVirt 4.3.5 potential issue with NFS storage
by Shani Leviim 13 Aug '19

13 Aug '19

Basically, I meant to verify the access by ssh, but I want to verify something following your detailed reply: According to [1], in order to set a NetApp NFS server, the required steps should look like this: # mount NetApp_NFS:/path/to/export /mnt # chown -R 36.36 /mnt # chmod -R 755 /mnt # umount /mnt Which is quite similar to the steps you've mentioned, except the last step of unmounting: Unmount the 10.214.13.64:/ovirt_production I think that you had to unmount /mnt/rhevstore instead. Can you please verify? [1] https://access.redhat.com/solutions/660143 *Regards,* *Shani Leviim* On Sun, Aug 11, 2019 at 10:57 PM Vrgotic, Marko <M.Vrgotic(a)activevideo.com> wrote: > Hi Shani, > > Thank you for your reply, but > How do I do that? > Reason why I am asking is following: > Hosts 2,3,4 do not have that issue. Host 1 and 5 do. > What I learned previously is that when using Netapp based NFS, which we > are, it’s required to before provisioning SHE and/or just adding a Host to > a pool, it’s required to execute following steps: > > Create random dir on a host: > - mkdir /mnt/rhevstore > Mount netapp volume to the dir > - mount -o sec=sys -t nfs 10.214.13.64:/ovirt_production /mnt/rhevstore > Set ownership to vdsm:kvm (36:36): > - chown -R vdsm:kvm /mnt/rhevstore/* > Unmount the 10.214.13.64:/ovirt_production > > I do not expect the above ownership actions need to be done initially on > each host, before starting the deployment, otherwise it would be > practically impossible to expand the Host pool. > > All 5 hosts are provisioned in same way. How? I am using foreman to > provision these servers, so they are built of same kickstart hostgroup > template. > > I even installed ovirt-hosted-engine-setup package to make sure all > required packages, users and groups are in place before adding host to > oVirt via UI or Ansible. > > Is it possible that we if I am already using or heavily using the > mentioned volume via Hosts already added to oVirt pool, that ownership > actions executed,on host about to be added to the pool, will fail to > complete setting ownership on all required files on the volume? > > To repeat the question above: How do I make sure Host can read metadata > file of the storage volume? > > Kindly awaiting your reply. > > > All best, > Marko Vrgotic > Sent from my iPhone > > On 11 Aug 2019, at 01:19, Shani Leviim <sleviim(a)redhat.com> wrote: > > Hi Marko, > Is seems that there's a connectivity problem with host 10.210.13.64. > Can you please make sure the metadata under > /rhev/data-center/mnt/10.210.13.64:_ovirt__production/6effda5e-1a0d-4312-bf93-d97fa9eb5aee/dom_md/metadata > is accessible? > > > *Regards, * > > *Shani Leviim * > > > On Sat, Aug 10, 2019 at 2:57 AM Vrgotic, Marko <M.Vrgotic(a)activevideo.com> > wrote: > >> Log files from ovirt engine and ovirt-sj-05 vdsm attached. >> >> >> >> Its related to host named: ovirt-sj-05.ictv.com >> >> >> >> Kindly awaiting your reply. >> >> >> >> >> >> — — — >> Met vriendelijke groet / Kind regards, >> >> *Marko Vrgotic* >> >> >> >> >> >> >> >> *From: *"Vrgotic, Marko" <M.Vrgotic(a)activevideo.com> >> *Date: *Thursday, 8 August 2019 at 17:02 >> *To: *Shani Leviim <sleviim(a)redhat.com> >> *Cc: *"users(a)ovirt.org" <users(a)ovirt.org> >> *Subject: *Re: [ovirt-users] Re: oVirt 4.3.5 potential issue with NFS >> storage >> >> >> >> Hey Shanii, >> >> >> >> Thank you for the reply. >> >> Sure, I will attach the full logs asap. >> >> What do you mean by “flow you are doing”? >> >> >> >> Kindly awaiting your reply. >> >> >> >> Marko Vrgotic >> >> >> >> *From: *Shani Leviim <sleviim(a)redhat.com> >> *Date: *Thursday, 8 August 2019 at 00:01 >> *To: *"Vrgotic, Marko" <M.Vrgotic(a)activevideo.com> >> *Cc: *"users(a)ovirt.org" <users(a)ovirt.org> >> *Subject: *Re: [ovirt-users] Re: oVirt 4.3.5 potential issue with NFS >> storage >> >> >> >> Hi, >> >> Can you please clarify the flow you're doing? >> >> Also, can you please attach full vdsm and engine logs? >> >> >> *Regards,* >> >> *Shani Leviim* >> >> >> >> >> >> On Thu, Aug 8, 2019 at 6:25 AM Vrgotic, Marko <M.Vrgotic(a)activevideo.com> >> wrote: >> >> Log line form VDSM: >> >> >> >> “[root@ovirt-sj-05 ~]# tail -f /var/log/vdsm/vdsm.log | grep WARN >> >> 2019-08-07 09:40:03,556-0700 WARN (check/loop) [storage.check] Checker >> u'/rhev/data-center/mnt/10.210.13.64:_ovirt__production/bda97276-a399-448f-9113-017972f6b55a/dom_md/metadata' >> is blocked for 20.00 seconds (check:282) >> >> 2019-08-07 09:40:47,132-0700 WARN (monitor/bda9727) [storage.Monitor] >> Host id for domain bda97276-a399-448f-9113-017972f6b55a was released (id: >> 5) (monitor:445) >> >> 2019-08-07 09:44:53,564-0700 WARN (check/loop) [storage.check] Checker >> u'/rhev/data-center/mnt/10.210.13.64:_ovirt__production/bda97276-a399-448f-9113-017972f6b55a/dom_md/metadata' >> is blocked for 20.00 seconds (check:282) >> >> 2019-08-07 09:46:38,604-0700 WARN (monitor/bda9727) [storage.Monitor] >> Host id for domain bda97276-a399-448f-9113-017972f6b55a was released (id: >> 5) (monitor:445)” >> >> >> >> >> >> >> >> *From: *"Vrgotic, Marko" <M.Vrgotic(a)activevideo.com> >> *Date: *Wednesday, 7 August 2019 at 09:09 >> *To: *"users(a)ovirt.org" <users(a)ovirt.org> >> *Subject: *oVirt 4.3.5 potential issue with NFS storage >> >> >> >> Dear oVIrt, >> >> >> >> This is my third oVirt platform in the company, but first time I am >> seeing following logs: >> >> >> >> “2019-08-07 16:00:16,099Z INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-51) [1b85e637] Lock freed >> to object >> 'EngineLock:{exclusiveLocks='[2350ee82-94ed-4f90-9366-451e0104d1d6=PROVIDER]', >> sharedLocks=''}' >> >> 2019-08-07 16:00:25,618Z WARN >> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] >> (EE-ManagedThreadFactory-engine-Thread-37723) [] domain >> 'bda97276-a399-448f-9113-017972f6b55a:ovirt_production' in problem >> 'PROBLEMATIC'. vds: 'ovirt-sj-05.ictv.com' >> >> 2019-08-07 16:00:40,630Z INFO >> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] >> (EE-ManagedThreadFactory-engine-Thread-37735) [] Domain >> 'bda97276-a399-448f-9113-017972f6b55a:ovirt_production' recovered from >> problem. vds: 'ovirt-sj-05.ictv.com' >> >> 2019-08-07 16:00:40,652Z INFO >> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] >> (EE-ManagedThreadFactory-engine-Thread-37737) [] Domain >> 'bda97276-a399-448f-9113-017972f6b55a:ovirt_production' recovered from >> problem. vds: 'ovirt-sj-01.ictv.com' >> >> 2019-08-07 16:00:40,652Z INFO >> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] >> (EE-ManagedThreadFactory-engine-Thread-37737) [] Domain >> 'bda97276-a399-448f-9113-017972f6b55a:ovirt_production' has recovered from >> problem. No active host in the DC is reporting it as problematic, so >> clearing the domain recovery timer.” >> >> >> >> Can you help me understanding why is this being reported? >> >> >> >> This setup is: >> >> >> >> 5HOSTS, 3 in HA >> >> SelfHostedEngine >> >> Version 4.3.5 >> >> NFS based Netapp storage, version 4.1 >> >> “10.210.13.64:/ovirt_hosted_engine on /rhev/data-center/mnt/10.210.13.64:_ovirt__hosted__engine >> type nfs4 >> (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=10.210.11.14,local_lock=none,addr=10.210.13.64) >> >> >> >> 10.210.13.64:/ovirt_production on /rhev/data-center/mnt/10.210.13.64:_ovirt__production >> type nfs4 >> (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=10.210.11.14,local_lock=none,addr=10.210.13.64) >> >> tmpfs on /run/user/0 type tmpfs >> (rw,nosuid,nodev,relatime,seclabel,size=9878396k,mode=700)” >> >> >> >> First mount is SHE dedicated storage. >> >> Second mount “ovirt_produciton” is for other VM Guests. >> >> >> >> Kindly awaiting your reply. >> >> >> >> Marko Vrgotic >> >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/ICRKHD3GXTPQE… >> >>

2 1