Update single node environment from 4.3.3 to 4.3.5 problem
by Gianluca Cecchi
Hello,
after updating hosted engine from 4.3.3 to 4.3.5 and then the only host
composing the environment (plain CentOS 7.6) it seems it is not able to
start vdsm daemons
kernel installed with update is kernel-3.10.0-957.27.2.el7.x86_64
Same problem also if using previous running kernel
3.10.0-957.12.2.el7.x86_64
[root@ovirt01 vdsm]# uptime
00:50:08 up 25 min, 3 users, load average: 0.60, 0.67, 0.60
[root@ovirt01 vdsm]#
[root@ovirt01 vdsm]# systemctl status vdsmd -l
● vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/etc/systemd/system/vdsmd.service; enabled; vendor
preset: enabled)
Active: failed (Result: start-limit) since Fri 2019-08-23 00:37:27 CEST;
7s ago
Process: 25810 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
--pre-start (code=exited, status=1/FAILURE)
Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Failed to start Virtual
Desktop Server Manager.
Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Unit vdsmd.service entered
failed state.
Aug 23 00:37:27 ovirt01.mydomain systemd[1]: vdsmd.service failed.
Aug 23 00:37:27 ovirt01.mydomain systemd[1]: vdsmd.service holdoff time
over, scheduling restart.
Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Stopped Virtual Desktop Server
Manager.
Aug 23 00:37:27 ovirt01.mydomain systemd[1]: start request repeated too
quickly for vdsmd.service
Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Failed to start Virtual
Desktop Server Manager.
Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Unit vdsmd.service entered
failed state.
Aug 23 00:37:27 ovirt01.mydomain systemd[1]: vdsmd.service failed.
[root@ovirt01 vdsm]#
[root@ovirt01 vdsm]# pwd
/var/log/vdsm
[root@ovirt01 vdsm]# ll -t | head
total 118972
-rw-r--r--. 1 root root 3406465 Aug 23 00:25 supervdsm.log
-rw-r--r--. 1 root root 73621 Aug 23 00:25 upgrade.log
-rw-r--r--. 1 vdsm kvm 0 Aug 23 00:01 vdsm.log
-rw-r--r--. 1 vdsm kvm 538480 Aug 22 23:46 vdsm.log.1.xz
-rw-r--r--. 1 vdsm kvm 187486 Aug 22 23:46 mom.log
-rw-r--r--. 1 vdsm kvm 621320 Aug 22 22:01 vdsm.log.2.xz
-rw-r--r--. 1 root root 374464 Aug 22 22:00 supervdsm.log.1.xz
-rw-r--r--. 1 vdsm kvm 2097122 Aug 22 21:53 mom.log.1
-rw-r--r--. 1 vdsm kvm 636212 Aug 22 20:01 vdsm.log.3.xz
[root@ovirt01 vdsm]#
link to upgrade.log contents here:
https://drive.google.com/file/d/17jtX36oH1hlbNUAiVhdBkVDbd28QegXG/view?us...
link to supervdsm.log (in gzip format) here:
https://drive.google.com/file/d/1l61ePU-eFS_xVHEAHnJthzTTnTyzu0MP/view?us...
It seems that since update I get these kind of lines inside it...
restore-net::DEBUG::2019-08-22
23:56:38,591::cmdutils::133::root::(exec_cmd) /sbin/tc filter del dev eth0
pref 5000 (cwd None)
restore-net::DEBUG::2019-08-22
23:56:38,595::cmdutils::141::root::(exec_cmd) FAILED: <err> = 'RTNETLINK
answers: Invalid argument\nWe have an error talking to the kernel\n'; <rc>
= 2
[root@ovirt01 vdsm]# systemctl status supervdsmd -l
● supervdsmd.service - Auxiliary vdsm service for running helper functions
as root
Loaded: loaded (/usr/lib/systemd/system/supervdsmd.service; static;
vendor preset: enabled)
Active: active (running) since Fri 2019-08-23 00:25:17 CEST; 23min ago
Main PID: 4540 (supervdsmd)
Tasks: 3
CGroup: /system.slice/supervdsmd.service
└─4540 /usr/bin/python2 /usr/share/vdsm/supervdsmd --sockfile
/var/run/vdsm/svdsm.sock
Aug 23 00:25:17 ovirt01.mydomain systemd[1]: Started Auxiliary vdsm service
for running helper functions as root.
[root@ovirt01 vdsm]#
[root@ovirt01 vdsm]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
ovirtmgmt state UP group default qlen 1000
link/ether b8:ae:ed:7f:17:11 brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 00:c2:c6:a4:18:c5 brd ff:ff:ff:ff:ff:ff
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 36:21:c1:5e:70:aa brd ff:ff:ff:ff:ff:ff
5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 46:d8:db:81:41:4e brd ff:ff:ff:ff:ff:ff
22: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UP group default qlen 1000
link/ether b8:ae:ed:7f:17:11 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.211/24 brd 192.168.1.255 scope global ovirtmgmt
valid_lft forever preferred_lft forever
[root@ovirt01 vdsm]#
[root@ovirt01 vdsm]# ip route show
default via 192.168.1.1 dev ovirtmgmt
192.168.1.0/24 dev ovirtmgmt proto kernel scope link src 192.168.1.211
[root@ovirt01 vdsm]#
[root@ovirt01 vdsm]# brctl show
bridge name bridge id STP enabled interfaces
ovirtmgmt 8000.b8aeed7f1711 no eth0
[root@ovirt01 vdsm]#
[root@ovirt01 vdsm]# systemctl status openvswitch
● openvswitch.service - Open vSwitch
Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; enabled;
vendor preset: disabled)
Active: active (exited) since Fri 2019-08-23 00:25:09 CEST; 26min ago
Process: 3894 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
Main PID: 3894 (code=exited, status=0/SUCCESS)
Tasks: 0
CGroup: /system.slice/openvswitch.service
Aug 23 00:25:09 ovirt01.mydomain systemd[1]: Starting Open vSwitch...
Aug 23 00:25:09 ovirt01.mydomain systemd[1]: Started Open vSwitch.
[root@ovirt01 vdsm]# ovs-vsctl show
02539902-1788-4796-9cdf-cf11ce8436bb
Bridge br-int
fail_mode: secure
Port br-int
Interface br-int
type: internal
ovs_version: "2.11.0"
[root@ovirt01 vdsm]#
any hints? Thanks
Gianluca
4 years, 7 months
Evaluate oVirt to remove VMware
by dsalade＠gmail.com
Howdy All!!!
Somewhat new user to Linux and having some issues with oVirt node networking.
We are mainly a Windows/VMware medium shop but have a pretty good network.
Our VMware setup has redundant switches using multiple VLANs. This was fairly easy to setup with VMware.
We never had an issue with VMware, add 2 NICs to the ESXi host and then adding VLANs into the Virtual Switch.
Biggest issue I am having, should I be bonding, teaming and VLANing on top of that?
Tried setting up node with Engine inside which fails to deploy because of networking.
Then I tried a standalone Engine and could not add node to it - I believe networking as well as Power Management agent was the problem.
Would like the nodes/engine to be on one subnet (lets say 10.10.11.x) and VMs to be allowed to run there are well as 10 other VLANs.
I guess welcome to the wonderful world of Linux networking and learning some new technologies is in store for me.
I hope I am not being vague, like I said fairly new to Linux.
Thanks for any responses!
Allen
4 years, 7 months
VDSM Hooks during migration
by Vrgotic, Marko
Dear oVIrt,
Would you be so kind to help me/tell me or point me how to find which Hooks, and in which order, are triggered when VM is being migrated?
Kindly awaiting your reply.
— — —
Met vriendelijke groet / Kind regards,
Marko Vrgotic
4 years, 7 months
Re: VM --- is not responding.
by Strahil
Hm... It was supposes to show controller status.
Maybe the hpssacli you have is not supporting your raid cards.Check for newer version on HPE's support page.
Best Regards,
Strahil Nikolov
Best Regards,
Strahil NikolovOn Aug 14, 2019 11:40, Edoardo Mazza <edo7411(a)gmail.com> wrote:
>
> I installed hpssacli-2.40-13.0.x86_64.rpm and the result of "hpssacli ctrl all show status" is:
> Error: No controllers detected. Possible causes:.....
> The s.o. run on sd cards and the vm runs on array on traditional disk
> thanks
> Edoardo
>
> Il giorno lun 12 ago 2019 alle ore 05:59 Strahil <hunter86_bg(a)yahoo.com> ha scritto:
>>
>> Would you check the health status of the controllers :
>> hpssacli ctrl all show status
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> On Aug 11, 2019 09:55, Edoardo Mazza <edo7411(a)gmail.com> wrote:
>>>
>>> The hosts are 3 ProLiant DL380 Gen10, 2 hosts with HPE Smart Array P816i-a SR Gen10 like controller and the other host with
>>> HPE Smart Array P408i-a SR Gen10. The storage for ovirt enviroment is gluster and the last host is the arbiter in the gluster enviroment.
>>> The S.M.A.R.T. healt status is ok for all host
>>> Edoardo
>>>
>>>
>>>
>>>
>>>
>>> Il giorno gio 8 ago 2019 alle ore 16:19 Sandro Bonazzola <sbonazzo(a)redhat.com> ha scritto:
>>>>
>>>>
>>>>
>>>> Il giorno gio 8 ago 2019 alle ore 11:19 Edoardo Mazza <edo7411(a)gmail.com> ha scritto:
>>>>>
>>>>> Hi all,
>>>>> It is more days that for same vm I received this error, but I don't underdand why.
>>>>> The traffic of the virtual machine is not excessive, cpu and ram to, but for few minutes the vm is not responding. and in the messages log file of the vm I received the error under, yo can help me?
>>>>> thanks
>>>>
>>>>
>>>> can you check the S.M.A.R.T. health status of the disks?
>>>>
>>>>
>>>>>
>>>>> Edoardo
>>>>> kernel: NMI watchdog: BUG: soft lockup - CPU#2 stuck for 25s! [kworker/2:0:26227]
>>>>> Aug 8 02:51:11 vmmysql kernel: Modules linked in: binfmt_misc ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_
>>>>> ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_
>>>>> nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_con
>>>>> ntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter snd_hda_c
>>>>> odec_generic iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel snd_hda_intel snd_hda_codec aesni_intel snd_hda_core lrw gf128mul
>>>>> glue_helper ablk_helper snd_hwdep cryptd snd_seq snd_seq_device snd_pcm snd_timer snd soundcore virtio_rng sg virtio_balloon
>>>>> i2c_piix4 parport_pc parport joydev pcspkr ip_tables xfs libcrc32c sd_mod
>>>>> Aug 8 02:51:14 vmmysql kernel: crc_t10dif crct10dif_generic sr_mod cdrom virtio_net virtio_console virtio_scsi ata_generic p
>>>>> ata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw qxl floppy drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci drm_panel_orientation_quirks virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
>>>>> Aug 8 02:51:14 vmmysql kernel: CPU: 2 PID: 26227 Comm: kworker/2:0 Kdump: loaded Tainted: G L ------------ 3.10.0-957.12.1.el7.x86_64 #1
>>>>> Aug 8 02:51:14 vmmysql kernel: Hardware name: oVirt oVirt Node, BIOS 1.11.0-2.el7 04/01/2014
>>>>> Aug 8 02:51:14 vmmysql kernel: Workqueue: events_freezable disk_events_workfn
>>>>> Aug 8 02:51:14 vmmysql kernel: task: ffff9e25b6609040 ti: ffff9e27b1610000 task.ti: ffff9e27b1610000
>>>>> Aug 8 02:51:14 vmmysql kernel: RIP: 00
4 years, 7 months
Re: Need to enable STP on ovirt bridges
by Strahil
It seems that according to https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.0/... the ports of interest are:
16514/TCP
49152 - 49216/TCP
Best Regards,
Strahil NikolovOn Aug 25, 2019 08:48, Strahil <hunter86_bg(a)yahoo.com> wrote:
>
> Curtis,
>
> Do you have enough space to run tcpdump (port not 22) on both hosts and on the small VM you have done previously - and then start the migration?
>
> Best Regards,
> Strahil NikolovOn Aug 24, 2019 22:15, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote:
> >
> > I applied a 90Mbs QOS Rate Limit with 10 set for the shares to both
> > interfaces of 2 of the hosts. My hosts names are swm-01 and swm-02.
> >
> > Creating a small VM from a Cinder template and running it gave me a test VM.
> >
> > When I migrated from swm-01 to swm-02, swm-01 immediately became
> > unresponsive to pings, SSH'es, and to the ovirt interface which marked
> > it as "NonResponsive" soon after the VM finished. The VM did finish
> > migrating, however I'm unsure if that's a good migration or not.
> >
> > Thank you, Strahil.
> >
> > On Sat, Aug 24, 2019 at 12:39 PM Strahil <hunter86_bg(a)yahoo.com> wrote:
> > >
> > > What is your bandwidth threshold for the network used for VM migration ?
> > > Can you set a 90 mbit/s threshold (yes, less than 100mbit/s) and try to migrate a small (1 GB RAM) VM ?
> > >
> > > Do you see disconnects ?
> > >
> > > If no, try a little bit up (the threshold) and check again.
> > >
> > > Best Regards,
> > > Strahil NikolovOn Aug 23, 2019 23:19, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote:
> > > >
> > > > It took a while for my servers to come back on the network this time.
> > > > I think it's due to ovirt continuing to try to migrate the VMs around
> > > > like I requested. The 3 servers' names are "swm-01, swm-02 and
> > > > swm-03". Eventually (about 2-3 minutes ago) they all came back online.
> > > >
> > > > So I disabled and stopped the lldpad service.
> > > >
> > > > Nope. Started some more migrations and swm-02 and swm-03 disappeared
> > > > again. No ping, SSH hung, same as before - almost as soon as the
> > > > migration started.
> > > >
> > > > If you wall have any ideas what switch-level setting might be enabled,
> > > > let me know, cause I'm stumped. I can add it to the ticket that's
> > > > requesting the port configurations. I've already added the port
> > > > numbers and switch name that I got from CDP.
> > > >
> > > > Thanks again, I really appreciate the help!
> > > > cecjr
> > > >
> > > >
> > > >
> > > > On Fri, Aug 23, 2019 at 3:28 PM Dominik Holler <dholler(a)redhat.com> wrote:
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Aug 23, 2019 at 9:19 PM Dominik Holler <dholler(a)redhat.com> wrote:
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Fri, Aug 23, 2019 at 8:03 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote:
> > > > >>>
> > > > >>> This little cluster isn't in production or anything like that yet.
> > > > >>>
> > > > >>> So, I went ahead and used your ethtool commands to disable pause
> > > > >>> frames on both interfaces of each server. I then, chose a few VMs to
> > > > >>> migrate around at random.
> > > > >>>
> > > > >>> swm-02 and swm-03 both went out again. Unreachable. Can't ping, can't
> > > > >>> ssh, and the SSH session that I had open was unresponsive.
> > > > >>>
> > > > >>> Any other ideas?
> > > > >>>
> > > > >>
> > > > >> Sorry, no. Looks like two different NICs with different drivers and frimware goes down together.
> > > > >> This is a strong indication that the root cause is related to the switch.
> > > > >> Maybe you can get some information about the switch config by
> > > > >> 'lldptool get-tlv -n -i em1'
> > > > >>
> > > > >
> > > > > Another guess:
> > > > > After the optional 'lldptool get-tlv -n -i em1'
> > > > > 'systemctl stop lldpad'
> > > > > another try to migrate.
> > > > >
> > > > >
> > > > >>
> > > > >>
> > > > >>>
> > > > >>> On Fri, Aug 23, 2019 at 1:50 PM Dominik Holler <dholler(a)redhat.com> wrote:
> > > > >>> >
> > > > >>> >
> > > > >>> >
> > > > >>> > On Fri, Aug 23, 2019 at 6:45 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote:
> > > > >>> >>
> > > > >>> >> Unfortunately, I can't check on the switch. Trust me, I've tried.
> > > > >>> >> These servers are in a Co-Lo and I've put 5 tickets in asking about
> > > > >>> >> the port configuration. They just get ignored - but that's par for the
> > > > >>> >> coarse for IT here. Only about 2 out of 10 of our tickets get any
> > > > >>> >> response and usually the response doesn't help. Then
4 years, 7 months
attach untagged vlan internally on vm
by ernestclydeachua＠gmail.com
good day
currently i am testing oVirt on a single box and setup some tagged vms and non tagged vm.
the non tagged vm is a firewall but it has limitations on the number of nic so i cannot attach tagged vnic and wish to handdle vlan tagging on it
is it possible to pass untaged franes internally?
4 years, 7 months
Re: Need to enable STP on ovirt bridges
by Strahil
Curtis,
Do you have enough space to run tcpdump (port not 22) on both hosts and on the small VM you have done previously - and then start the migration?
Best Regards,
Strahil NikolovOn Aug 24, 2019 22:15, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote:
>
> I applied a 90Mbs QOS Rate Limit with 10 set for the shares to both
> interfaces of 2 of the hosts. My hosts names are swm-01 and swm-02.
>
> Creating a small VM from a Cinder template and running it gave me a test VM.
>
> When I migrated from swm-01 to swm-02, swm-01 immediately became
> unresponsive to pings, SSH'es, and to the ovirt interface which marked
> it as "NonResponsive" soon after the VM finished. The VM did finish
> migrating, however I'm unsure if that's a good migration or not.
>
> Thank you, Strahil.
>
> On Sat, Aug 24, 2019 at 12:39 PM Strahil <hunter86_bg(a)yahoo.com> wrote:
> >
> > What is your bandwidth threshold for the network used for VM migration ?
> > Can you set a 90 mbit/s threshold (yes, less than 100mbit/s) and try to migrate a small (1 GB RAM) VM ?
> >
> > Do you see disconnects ?
> >
> > If no, try a little bit up (the threshold) and check again.
> >
> > Best Regards,
> > Strahil NikolovOn Aug 23, 2019 23:19, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote:
> > >
> > > It took a while for my servers to come back on the network this time.
> > > I think it's due to ovirt continuing to try to migrate the VMs around
> > > like I requested. The 3 servers' names are "swm-01, swm-02 and
> > > swm-03". Eventually (about 2-3 minutes ago) they all came back online.
> > >
> > > So I disabled and stopped the lldpad service.
> > >
> > > Nope. Started some more migrations and swm-02 and swm-03 disappeared
> > > again. No ping, SSH hung, same as before - almost as soon as the
> > > migration started.
> > >
> > > If you wall have any ideas what switch-level setting might be enabled,
> > > let me know, cause I'm stumped. I can add it to the ticket that's
> > > requesting the port configurations. I've already added the port
> > > numbers and switch name that I got from CDP.
> > >
> > > Thanks again, I really appreciate the help!
> > > cecjr
> > >
> > >
> > >
> > > On Fri, Aug 23, 2019 at 3:28 PM Dominik Holler <dholler(a)redhat.com> wrote:
> > > >
> > > >
> > > >
> > > > On Fri, Aug 23, 2019 at 9:19 PM Dominik Holler <dholler(a)redhat.com> wrote:
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Aug 23, 2019 at 8:03 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote:
> > > >>>
> > > >>> This little cluster isn't in production or anything like that yet.
> > > >>>
> > > >>> So, I went ahead and used your ethtool commands to disable pause
> > > >>> frames on both interfaces of each server. I then, chose a few VMs to
> > > >>> migrate around at random.
> > > >>>
> > > >>> swm-02 and swm-03 both went out again. Unreachable. Can't ping, can't
> > > >>> ssh, and the SSH session that I had open was unresponsive.
> > > >>>
> > > >>> Any other ideas?
> > > >>>
> > > >>
> > > >> Sorry, no. Looks like two different NICs with different drivers and frimware goes down together.
> > > >> This is a strong indication that the root cause is related to the switch.
> > > >> Maybe you can get some information about the switch config by
> > > >> 'lldptool get-tlv -n -i em1'
> > > >>
> > > >
> > > > Another guess:
> > > > After the optional 'lldptool get-tlv -n -i em1'
> > > > 'systemctl stop lldpad'
> > > > another try to migrate.
> > > >
> > > >
> > > >>
> > > >>
> > > >>>
> > > >>> On Fri, Aug 23, 2019 at 1:50 PM Dominik Holler <dholler(a)redhat.com> wrote:
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> > On Fri, Aug 23, 2019 at 6:45 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote:
> > > >>> >>
> > > >>> >> Unfortunately, I can't check on the switch. Trust me, I've tried.
> > > >>> >> These servers are in a Co-Lo and I've put 5 tickets in asking about
> > > >>> >> the port configuration. They just get ignored - but that's par for the
> > > >>> >> coarse for IT here. Only about 2 out of 10 of our tickets get any
> > > >>> >> response and usually the response doesn't help. Then
4 years, 7 months
Re: Need to enable STP on ovirt bridges
by Strahil
Curtis,
Do you have enough space to run tcpdump (port not 22) on both hosts and on the small VM you have done previously - and then start the migration?
Best Regards,
Strahil NikolovOn Aug 24, 2019 22:15, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote:
>
> I applied a 90Mbs QOS Rate Limit with 10 set for the shares to both
> interfaces of 2 of the hosts. My hosts names are swm-01 and swm-02.
>
> Creating a small VM from a Cinder template and running it gave me a test VM.
>
> When I migrated from swm-01 to swm-02, swm-01 immediately became
> unresponsive to pings, SSH'es, and to the ovirt interface which marked
> it as "NonResponsive" soon after the VM finished. The VM did finish
> migrating, however I'm unsure if that's a good migration or not.
>
> Thank you, Strahil.
>
> On Sat, Aug 24, 2019 at 12:39 PM Strahil <hunter86_bg(a)yahoo.com> wrote:
> >
> > What is your bandwidth threshold for the network used for VM migration ?
> > Can you set a 90 mbit/s threshold (yes, less than 100mbit/s) and try to migrate a small (1 GB RAM) VM ?
> >
> > Do you see disconnects ?
> >
> > If no, try a little bit up (the threshold) and check again.
> >
> > Best Regards,
> > Strahil NikolovOn Aug 23, 2019 23:19, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote:
> > >
> > > It took a while for my servers to come back on the network this time.
> > > I think it's due to ovirt continuing to try to migrate the VMs around
> > > like I requested. The 3 servers' names are "swm-01, swm-02 and
> > > swm-03". Eventually (about 2-3 minutes ago) they all came back online.
> > >
> > > So I disabled and stopped the lldpad service.
> > >
> > > Nope. Started some more migrations and swm-02 and swm-03 disappeared
> > > again. No ping, SSH hung, same as before - almost as soon as the
> > > migration started.
> > >
> > > If you wall have any ideas what switch-level setting might be enabled,
> > > let me know, cause I'm stumped. I can add it to the ticket that's
> > > requesting the port configurations. I've already added the port
> > > numbers and switch name that I got from CDP.
> > >
> > > Thanks again, I really appreciate the help!
> > > cecjr
> > >
> > >
> > >
> > > On Fri, Aug 23, 2019 at 3:28 PM Dominik Holler <dholler(a)redhat.com> wrote:
> > > >
> > > >
> > > >
> > > > On Fri, Aug 23, 2019 at 9:19 PM Dominik Holler <dholler(a)redhat.com> wrote:
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Aug 23, 2019 at 8:03 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote:
> > > >>>
> > > >>> This little cluster isn't in production or anything like that yet.
> > > >>>
> > > >>> So, I went ahead and used your ethtool commands to disable pause
> > > >>> frames on both interfaces of each server. I then, chose a few VMs to
> > > >>> migrate around at random.
> > > >>>
> > > >>> swm-02 and swm-03 both went out again. Unreachable. Can't ping, can't
> > > >>> ssh, and the SSH session that I had open was unresponsive.
> > > >>>
> > > >>> Any other ideas?
> > > >>>
> > > >>
> > > >> Sorry, no. Looks like two different NICs with different drivers and frimware goes down together.
> > > >> This is a strong indication that the root cause is related to the switch.
> > > >> Maybe you can get some information about the switch config by
> > > >> 'lldptool get-tlv -n -i em1'
> > > >>
> > > >
> > > > Another guess:
> > > > After the optional 'lldptool get-tlv -n -i em1'
> > > > 'systemctl stop lldpad'
> > > > another try to migrate.
> > > >
> > > >
> > > >>
> > > >>
> > > >>>
> > > >>> On Fri, Aug 23, 2019 at 1:50 PM Dominik Holler <dholler(a)redhat.com> wrote:
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> > On Fri, Aug 23, 2019 at 6:45 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote:
> > > >>> >>
> > > >>> >> Unfortunately, I can't check on the switch. Trust me, I've tried.
> > > >>> >> These servers are in a Co-Lo and I've put 5 tickets in asking about
> > > >>> >> the port configuration. They just get ignored - but that's par for the
> > > >>> >> coarse for IT here. Only about 2 out of 10 of our tickets get any
> > > >>> >> response and usually the response doesn't help. Then
4 years, 7 months
Re: Need to enable STP on ovirt bridges
by Strahil
What is your bandwidth threshold for the network used for VM migration ?
Can you set a 90 mbit/s threshold (yes, less than 100mbit/s) and try to migrate a small (1 GB RAM) VM ?
Do you see disconnects ?
If no, try a little bit up (the threshold) and check again.
Best Regards,
Strahil NikolovOn Aug 23, 2019 23:19, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote:
>
> It took a while for my servers to come back on the network this time.
> I think it's due to ovirt continuing to try to migrate the VMs around
> like I requested. The 3 servers' names are "swm-01, swm-02 and
> swm-03". Eventually (about 2-3 minutes ago) they all came back online.
>
> So I disabled and stopped the lldpad service.
>
> Nope. Started some more migrations and swm-02 and swm-03 disappeared
> again. No ping, SSH hung, same as before - almost as soon as the
> migration started.
>
> If you wall have any ideas what switch-level setting might be enabled,
> let me know, cause I'm stumped. I can add it to the ticket that's
> requesting the port configurations. I've already added the port
> numbers and switch name that I got from CDP.
>
> Thanks again, I really appreciate the help!
> cecjr
>
>
>
> On Fri, Aug 23, 2019 at 3:28 PM Dominik Holler <dholler(a)redhat.com> wrote:
> >
> >
> >
> > On Fri, Aug 23, 2019 at 9:19 PM Dominik Holler <dholler(a)redhat.com> wrote:
> >>
> >>
> >>
> >> On Fri, Aug 23, 2019 at 8:03 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote:
> >>>
> >>> This little cluster isn't in production or anything like that yet.
> >>>
> >>> So, I went ahead and used your ethtool commands to disable pause
> >>> frames on both interfaces of each server. I then, chose a few VMs to
> >>> migrate around at random.
> >>>
> >>> swm-02 and swm-03 both went out again. Unreachable. Can't ping, can't
> >>> ssh, and the SSH session that I had open was unresponsive.
> >>>
> >>> Any other ideas?
> >>>
> >>
> >> Sorry, no. Looks like two different NICs with different drivers and frimware goes down together.
> >> This is a strong indication that the root cause is related to the switch.
> >> Maybe you can get some information about the switch config by
> >> 'lldptool get-tlv -n -i em1'
> >>
> >
> > Another guess:
> > After the optional 'lldptool get-tlv -n -i em1'
> > 'systemctl stop lldpad'
> > another try to migrate.
> >
> >
> >>
> >>
> >>>
> >>> On Fri, Aug 23, 2019 at 1:50 PM Dominik Holler <dholler(a)redhat.com> wrote:
> >>> >
> >>> >
> >>> >
> >>> > On Fri, Aug 23, 2019 at 6:45 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote:
> >>> >>
> >>> >> Unfortunately, I can't check on the switch. Trust me, I've tried.
> >>> >> These servers are in a Co-Lo and I've put 5 tickets in asking about
> >>> >> the port configuration. They just get ignored - but that's par for the
> >>> >> coarse for IT here. Only about 2 out of 10 of our tickets get any
> >>> >> response and usually the response doesn't help. Then the system they
> >>> >> use auto-closes the ticket. That was why I was suspecting STP before.
> >>> >>
> >>> >> I can do ethtool. I do have root on these servers, though. Are you
> >>> >> trying to get me to turn off link-speed auto-negotiation? Would you
> >>> >> like me to try that?
> >>> >>
> >>> >
> >>> > It is just a suspicion, that the reason is pause frames.
> >>> > Let's start on a NIC which is not used for ovirtmgmt, I guess em1.
> >>> > Does 'ethtool -S em1 | grep pause' show something?
> >>> > Does 'ethtool em1 | grep pause' indicates support for pause?
> >>> > The current config is shown by 'ethtool -a em1'.
> >>> > '-A autoneg' "Specifies whether pause autonegotiation should be enabled." according to ethtool doc.
> >>> > Assuming flow control is enabled by default, I would try to disable it via
> >>> > 'ethtool -A em1 autoneg off rx off tx off'
> >>> > and check if it is applied via
> >>> > 'ethtool -a em1'
> >>> > and check if the behavior under load changes.
> >>> >
> >>> >
> >>> >
> >>> >>
> >>> >> On Fri, Aug 23, 2019 at 12:24 PM Dominik Holler <dholler(a)redhat.com> wrote:
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > On Fri, Aug 23, 2019 at 5:49 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote:
> >>> >> >>
> >>> >> >> Sure! Right now, I only have a 5
4 years, 7 months
Moving ovirt engine disk to another storage volume
by Erick Perez - Quadrian Enterprises
Good morning,
I am running Ovirt 4.3.5 in Centos 7.6 with one virt node an a NFS
storage node. I did the self-hosted engine setup and i plan to add a
second virt host in a few days.
I need to do heavy maintenance on the storage node (VDO and mdadm
things) and would like to know how (or a link to an article) can I
move the ovirt engine disk to another storage.
Currentl the NFS storage has two volumens (volA,volB) and the physical
host have spare space too. Virtual machines are in VolB and the engine
is in VolA.
I would like to move the engine disk from VolA to VolB or to local storage.
BTW I am not sure if I should say "move the engine" or should I say
move "hosted_storage domain" domain
thanks in advance.
---------------------
Erick Perez
4 years, 7 months
