[vdsm] strange network test failure on FC23
Hi, Jenkins doesn't like my (trivial) https://gerrit.ovirt.org/#/c/49271/ which is about moving one log line (!). The failure is 00:08:54.680 ====================================================================== 00:08:54.680 FAIL: testEnablePromisc (ipwrapperTests.TestDrvinfo) 00:08:54.680 ---------------------------------------------------------------------- 00:08:54.680 Traceback (most recent call last): 00:08:54.680 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/ipwrapperTests.py", line 130, in testEnablePromisc 00:08:54.680 "Could not enable promiscuous mode.") 00:08:54.680 AssertionError: Could not enable promiscuous mode. 00:08:54.680 -------------------- >> begin captured logging << -------------------- 00:08:54.680 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /usr/sbin/brctl show (cwd None) 00:08:54.680 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 00:08:54.680 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link add name vdsm-HIRjJp type bridge (cwd None) 00:08:54.680 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 00:08:54.680 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link set dev vdsm-HIRjJp up (cwd None) 00:08:54.680 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 00:08:54.680 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link set dev vdsm-HIRjJp promisc on (cwd None) 00:08:54.680 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 00:08:54.680 --------------------- >> end captured logging << --------------------- Here in fullest: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/638/console The command like looks OK, and can't think any reason it could fail, except startup race. Using taskset, the ip command now takes a little longer to complete. Maybe -just wild guessing- the code isn't properly waiting for the command to complete? Otherwise not the slightest clue :) Bests, -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
--H8ygTp4AXg6deix2 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I'm retriggering on another slave to see if it fails there too, might be env related. But that would not discard the race issue (as it might be just a coincidence or the slave a bit faster) On 11/27 11:55, Francesco Romani wrote:
Hi, =20 Jenkins doesn't like my (trivial) https://gerrit.ovirt.org/#/c/49271/ =20 which is about moving one log line (!). =20 The failure is =20 00:08:54.680 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 00:08:54.680 FAIL: testEnablePromisc (ipwrapperTests.TestDrvinfo) 00:08:54.680 ------------------------------------------------------------=
00:08:54.680 Traceback (most recent call last): 00:08:54.680 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23= -x86_64/vdsm/tests/ipwrapperTests.py", line 130, in testEnablePromisc 00:08:54.680 "Could not enable promiscuous mode.") 00:08:54.680 AssertionError: Could not enable promiscuous mode. 00:08:54.680 -------------------- >> begin captured logging << ----------=
00:08:54.680 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /usr/sbin/brctl= show (cwd None) 00:08:54.680 root: DEBUG: SUCCESS: <err> =3D ''; <rc> =3D 0 00:08:54.680 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link a= dd name vdsm-HIRjJp type bridge (cwd None) 00:08:54.680 root: DEBUG: SUCCESS: <err> =3D ''; <rc> =3D 0 00:08:54.680 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link s= et dev vdsm-HIRjJp up (cwd None) 00:08:54.680 root: DEBUG: SUCCESS: <err> =3D ''; <rc> =3D 0 00:08:54.680 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link s= et dev vdsm-HIRjJp promisc on (cwd None) 00:08:54.680 root: DEBUG: SUCCESS: <err> =3D ''; <rc> =3D 0 00:08:54.680 --------------------- >> end captured logging << -----------=
=20 =20 Here in fullest: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc2= 3-x86_64/638/console =20 The command like looks OK, and can't think any reason it could fail, exce= pt startup race. Using taskset, the ip command now takes a little longer to complete. =20 Maybe -just wild guessing- the code isn't properly waiting for the comman= d to complete? Otherwise not the slightest clue :) =20 Bests, =20 --=20 Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --H8ygTp4AXg6deix2 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJWWJvkAAoJEEBxx+HSYmnDqToH/Ayeh5w/7ugCBEaC51DwoeyO nFU0GSWMaIiMnr05D0dUc4xBvmbhsrxUa2jPQi1bXvAV/w7yyg+I3GbhjOldE/kD EQJd4RH0svVQkUxRpog73NgIZawybeV9hEKYBf8xJlJlTkrtystlQHH3FEfFtsCx qWQ9ayXYrtI0y+4Z/sEr5olaDNPLBe0plEL4o7n85IAD8UIxN/ukBjvHWvpVJGwr XRU9gRZIJ8Y9Q5Iqoi7fYicrzmHXtQXG88AAexCj3UGZSC7RqEzdKQ83bbzAW/Vn GnTWt3jhFCb8UVidWr9phiVaWrBHQkR/EoDszSwYV6cXcTZHL0+2A5zwSjR/WBc= =HQp2 -----END PGP SIGNATURE----- --H8ygTp4AXg6deix2--
--n/aVsWSeQ4JHkrmm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I see though that is leaving a bunch of test interfaces in the slave: 2753: vdsmtest-gNhf3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqu= eue state UNKNOWN group default=20 link/ether 86:73:13:4c:e2:63 brd ff:ff:ff:ff:ff:ff 2767: vdsmtest-aX5So: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqu= eue state UNKNOWN group default=20 link/ether 9e:fa:75:3e:a3:e6 brd ff:ff:ff:ff:ff:ff 2768: vdsmtest-crso1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqu= eue state UNKNOWN group default=20 link/ether 22:ce:cb:5c:42:3b brd ff:ff:ff:ff:ff:ff 2772: vdsmtest-JDc5P: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqu= eue state UNKNOWN group default=20 link/ether ae:79:dc:e9:22:9a brd ff:ff:ff:ff:ff:ff Can we do a cleanup in the tests and remove those? That might collide with other tests and create failures. On 11/27 19:07, David Caro wrote:
=20 I'm retriggering on another slave to see if it fails there too, might be = env related. But that would not discard the race issue (as it might be just a coincidence or the slave a bit faster) =20 On 11/27 11:55, Francesco Romani wrote:
Hi, =20 Jenkins doesn't like my (trivial) https://gerrit.ovirt.org/#/c/49271/ =20 which is about moving one log line (!). =20 The failure is =20 00:08:54.680 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D 00:08:54.680 FAIL: testEnablePromisc (ipwrapperTests.TestDrvinfo) 00:08:54.680 ----------------------------------------------------------=
00:08:54.680 Traceback (most recent call last): 00:08:54.680 File "/home/jenkins/workspace/vdsm_master_check-patch-fc= 23-x86_64/vdsm/tests/ipwrapperTests.py", line 130, in testEnablePromisc 00:08:54.680 "Could not enable promiscuous mode.") 00:08:54.680 AssertionError: Could not enable promiscuous mode. 00:08:54.680 -------------------- >> begin captured logging << --------=
00:08:54.680 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /usr/sbin/brc= tl show (cwd None) 00:08:54.680 root: DEBUG: SUCCESS: <err> =3D ''; <rc> =3D 0 00:08:54.680 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link= add name vdsm-HIRjJp type bridge (cwd None) 00:08:54.680 root: DEBUG: SUCCESS: <err> =3D ''; <rc> =3D 0 00:08:54.680 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link= set dev vdsm-HIRjJp up (cwd None) 00:08:54.680 root: DEBUG: SUCCESS: <err> =3D ''; <rc> =3D 0 00:08:54.680 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link= set dev vdsm-HIRjJp promisc on (cwd None) 00:08:54.680 root: DEBUG: SUCCESS: <err> =3D ''; <rc> =3D 0 00:08:54.680 --------------------- >> end captured logging << ---------=
=20 =20 Here in fullest: http://jenkins.ovirt.org/job/vdsm_master_check-patch-f= c23-x86_64/638/console =20 The command like looks OK, and can't think any reason it could fail, ex= cept startup race. Using taskset, the ip command now takes a little longer to complete. =20 Maybe -just wild guessing- the code isn't properly waiting for the comm= and to complete? Otherwise not the slightest clue :) =20 Bests, =20 --=20 Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra =20 --=20 David Caro =20 Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D =20 Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --n/aVsWSeQ4JHkrmm Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJWWJxaAAoJEEBxx+HSYmnDmWkH/j2mM7ax6xtgByHqKGlqSAAd EbMcxCNvGEftaE+gmz34i3qtQ6jWYDkXNhk0BZh7q1EWrlHLjLEYM5y85Lx3EQqM l+EX8GdgGOOEU9aojRk8V4oVrV34lUvVd/mXoE5bSDE+Mg0XKFfEF8c5BnFpmq24 fBgMbwGfCkfIslElI+Oucei3IjOno1pVKnwVj35+IR9KjGQadT9gpl4nOHdpsDsd Sspr+pshyMRKTSkRbVMz0ylS9mjv18kTT10RULnEg6CFJUZgb0iwicV7nJp4yWMl I4weS9uh5Bz1pjEltwWlnd3gVXs8xKb5GWQWYgaHzpnGJKov622xsnRaW3HSfEI= =WHwH -----END PGP SIGNATURE----- --n/aVsWSeQ4JHkrmm--
Adding Dan, Ido On Fri, Nov 27, 2015 at 8:09 PM, David Caro <dcaro@redhat.com> wrote:
I see though that is leaving a bunch of test interfaces in the slave:
2753: vdsmtest-gNhf3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default link/ether 86:73:13:4c:e2:63 brd ff:ff:ff:ff:ff:ff 2767: vdsmtest-aX5So: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default link/ether 9e:fa:75:3e:a3:e6 brd ff:ff:ff:ff:ff:ff 2768: vdsmtest-crso1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default link/ether 22:ce:cb:5c:42:3b brd ff:ff:ff:ff:ff:ff 2772: vdsmtest-JDc5P: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default link/ether ae:79:dc:e9:22:9a brd ff:ff:ff:ff:ff:ff
Can we do a cleanup in the tests and remove those? That might collide with other tests and create failures.
On 11/27 19:07, David Caro wrote:
I'm retriggering on another slave to see if it fails there too, might be env related. But that would not discard the race issue (as it might be just a coincidence or the slave a bit faster)
On 11/27 11:55, Francesco Romani wrote:
Hi,
Jenkins doesn't like my (trivial) https://gerrit.ovirt.org/#/c/49271/
which is about moving one log line (!).
The failure is
00:08:54.680 ====================================================================== 00:08:54.680 FAIL: testEnablePromisc (ipwrapperTests.TestDrvinfo) 00:08:54.680 ---------------------------------------------------------------------- 00:08:54.680 Traceback (most recent call last): 00:08:54.680 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/ipwrapperTests.py", line 130, in testEnablePromisc 00:08:54.680 "Could not enable promiscuous mode.") 00:08:54.680 AssertionError: Could not enable promiscuous mode. 00:08:54.680 -------------------- >> begin captured logging << -------------------- 00:08:54.680 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /usr/sbin/brctl show (cwd None) 00:08:54.680 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 00:08:54.680 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link add name vdsm-HIRjJp type bridge (cwd None) 00:08:54.680 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 00:08:54.680 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link set dev vdsm-HIRjJp up (cwd None) 00:08:54.680 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 00:08:54.680 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/ip link set dev vdsm-HIRjJp promisc on (cwd None) 00:08:54.680 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 00:08:54.680 --------------------- >> end captured logging << ---------------------
Here in fullest: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc23-x86_64/638/console
The command like looks OK, and can't think any reason it could fail, except startup race. Using taskset, the ip command now takes a little longer to complete.
Maybe -just wild guessing- the code isn't properly waiting for the command to complete? Otherwise not the slightest clue :)
Bests,
-- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Fri, Nov 27, 2015 at 07:09:30PM +0100, David Caro wrote:
I see though that is leaving a bunch of test interfaces in the slave:
2753: vdsmtest-gNhf3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default link/ether 86:73:13:4c:e2:63 brd ff:ff:ff:ff:ff:ff 2767: vdsmtest-aX5So: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default link/ether 9e:fa:75:3e:a3:e6 brd ff:ff:ff:ff:ff:ff 2768: vdsmtest-crso1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default link/ether 22:ce:cb:5c:42:3b brd ff:ff:ff:ff:ff:ff 2772: vdsmtest-JDc5P: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default link/ether ae:79:dc:e9:22:9a brd ff:ff:ff:ff:ff:ff
Can we do a cleanup in the tests and remove those? That might collide with other tests and create failures.
These bridges are no longer created on master (thanks to Nir's http://gerrit.ovirt.org/44111) The should have been removed by the run_tests that created them, but this may not take place if it is killed (or dies) beforehand.
On Fri, Nov 27, 2015 at 6:55 PM, Francesco Romani <fromani@redhat.com> wrote:
Using taskset, the ip command now takes a little longer to complete.
Since we always use the same set of CPUs, I assume using a mask (for 0 & 1, just use 0x3, as the man suggests) might be a tiny of a fraction faster to execute taskset with, instead of the need to translate the numeric CPU list. However, the real concern is making sure CPUs 0 & 1 are not really too busy with stuff (including interrupt handling, etc.) Y.
On Sun, Nov 29, 2015 at 10:37 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Nov 27, 2015 at 6:55 PM, Francesco Romani <fromani@redhat.com> wrote:
Using taskset, the ip command now takes a little longer to complete.
Since we always use the same set of CPUs, I assume using a mask (for 0 & 1, just use 0x3, as the man suggests) might be a tiny of a fraction faster to execute taskset with, instead of the need to translate the numeric CPU list.
Creating the string "0-<last cpu index>" is one line in vdsm. The code handling this in taskset is written in C, so the parsing time is practically zero. Even if it was non-zero, this code run once when we run a child process, so the cost is insignificant.
However, the real concern is making sure CPUs 0 & 1 are not really too busy with stuff (including interrupt handling, etc.)
This code is used when we run a child process, to allow the child process to run on all cpus (in this case, cpu 0 and cpu 1). So I think there is no concern here. Vdsm itself is running by default on cpu 1, which should be less busy then cpu 0. The user can modify this configuration on the host, I guess we need to expose this on the engine side (cluster setting?). Also if vdsm is pinned to certain cpu, should user get a warning trying to pin a vm to this cpu? Michal, what do you think? Nir
On Sun, Nov 29, 2015 at 5:37 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Nov 29, 2015 at 10:37 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Nov 27, 2015 at 6:55 PM, Francesco Romani <fromani@redhat.com> wrote:
Using taskset, the ip command now takes a little longer to complete.
Since we always use the same set of CPUs, I assume using a mask (for 0 &
just use 0x3, as the man suggests) might be a tiny of a fraction faster to execute taskset with, instead of the need to translate the numeric CPU
1, list.
Creating the string "0-<last cpu index>" is one line in vdsm. The code handling this in taskset is written in C, so the parsing time is practically zero. Even if it was non-zero, this code run once when we run a child process, so the cost is insignificant.
I think it's easier to just to have it as a mask in a config item somewhere, without need to create it or parse it anywhere. For us and for the user.
However, the real concern is making sure CPUs 0 & 1 are not really too busy with stuff (including interrupt handling, etc.)
This code is used when we run a child process, to allow the child process to run on all cpus (in this case, cpu 0 and cpu 1). So I think there is no concern here.
Vdsm itself is running by default on cpu 1, which should be less busy then cpu 0.
I assume those are cores, which probably in a multi-socket will be in the first socket only. There's a good chance that the FC and or network/cards will also bind their interrupts to core0 & core 1 (check /proc/interrupts) on the same socket.
From my poor laptop (1s, 4c): 42: 1487104 9329 4042 3598 IR-PCI-MSI 512000-edge 0000:00:1f.2
(my SATA controller) 43: 14664923 34 18 13 IR-PCI-MSI 327680-edge xhci_hcd (my dock station connector) 45: 6754579 4437 2501 2419 IR-PCI-MSI 32768-edge i915 (GPU) 47: 187409 11627 1235 1259 IR-PCI-MSI 2097152-edge iwlwifi (NIC, wifi) Y.
The user can modify this configuration on the host, I guess we need to expose this on the engine side (cluster setting?).
Also if vdsm is pinned to certain cpu, should user get a warning trying to pin a vm to this cpu?
Michal, what do you think?
Nir
On Sun, Nov 29, 2015 at 6:01 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Sun, Nov 29, 2015 at 5:37 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Nov 29, 2015 at 10:37 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Nov 27, 2015 at 6:55 PM, Francesco Romani <fromani@redhat.com> wrote:
Using taskset, the ip command now takes a little longer to complete.
Since we always use the same set of CPUs, I assume using a mask (for 0
&
1, just use 0x3, as the man suggests) might be a tiny of a fraction faster to execute taskset with, instead of the need to translate the numeric CPU list.
Creating the string "0-<last cpu index>" is one line in vdsm. The code handling this in taskset is written in C, so the parsing time is practically zero. Even if it was non-zero, this code run once when we run a child process, so the cost is insignificant.
I think it's easier to just to have it as a mask in a config item somewhere, without need to create it or parse it anywhere. For us and for the user.
However, the real concern is making sure CPUs 0 & 1 are not really too busy with stuff (including interrupt handling, etc.)
This code is used when we run a child process, to allow the child process to run on all cpus (in this case, cpu 0 and cpu 1). So I think there is no concern here.
Vdsm itself is running by default on cpu 1, which should be less busy then cpu 0.
I assume those are cores, which probably in a multi-socket will be in the first socket only. There's a good chance that the FC and or network/cards will also bind
We have this option in /etc/vdsm/vdsm.conf: # Comma separated whitelist of CPU cores on which VDSM is allowed to # run. The default is "", meaning VDSM can be scheduled by the OS to # run on any core. Valid examples: "1", "0,1", "0,2,3" # cpu_affinity = 1 I think this is the easiest option for users. their
interrupts to core0 & core 1 (check /proc/interrupts) on the same socket. From my poor laptop (1s, 4c): 42: 1487104 9329 4042 3598 IR-PCI-MSI 512000-edge
0000:00:1f.2
(my SATA controller)
43: 14664923 34 18 13 IR-PCI-MSI 327680-edge
xhci_hcd (my dock station connector)
45: 6754579 4437 2501 2419 IR-PCI-MSI 32768-edge i915 (GPU)
47: 187409 11627 1235 1259 IR-PCI-MSI 2097152-edge
iwlwifi (NIC, wifi)
Interesting, here an example from a 8 cores machine running my vms: [nsoffer@jumbo ~]$ cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0: 31 0 0 0 0 0 0 0 IR-IO-APIC-edge timer 1: 2 0 0 1 0 0 0 0 IR-IO-APIC-edge i8042 8: 0 0 0 0 0 0 0 1 IR-IO-APIC-edge rtc0 9: 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi acpi 12: 3 0 0 0 0 0 1 0 IR-IO-APIC-edge i8042 16: 4 4 9 0 9 1 1 3 IR-IO-APIC 16-fasteoi ehci_hcd:usb3 23: 13 1 5 0 12 1 1 0 IR-IO-APIC 23-fasteoi ehci_hcd:usb4 24: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0 25: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar1 26: 3670 354 215 9062370 491 124 169 54 IR-PCI-MSI-edge 0000:00:1f.2 27: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 28: 166285414 0 3 0 4 0 0 0 IR-PCI-MSI-edge em1 29: 18 0 0 0 4 3 0 0 IR-PCI-MSI-edge mei_me 30: 1 151 17 0 3 169 26 94 IR-PCI-MSI-edge snd_hda_intel NMI: 2508 2296 2317 2356 867 918 912 903 Non-maskable interrupts LOC: 302996116 312923350 312295375 312089303 86282447 94046427 90847792 91761277 Local timer interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts PMI: 2508 2296 2317 2356 867 918 912 903 Performance monitoring interrupts IWI: 1 0 0 5 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries RES: 34480637 12953645 13139863 14309885 8881861 10110753 9709070 9703933 Rescheduling interrupts CAL: 7387779 7682087 7283716 7135792 2771105 1785528 1887493 1843734 Function call interrupts TLB: 11121 16458 17923 15216 8534 8173 8639 7837 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 7789 7789 7789 7789 7789 7789 7789 7789 Machine check polls HYP: 0 0 0 0 0 0 0 0 Hypervisor callback interrupts ERR: 0 MIS: 0 It seems that our default (CPU1) is fine. Francesco, what do you think? Nir
On 29 Nov 2015, at 17:34, Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Nov 29, 2015 at 6:01 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Sun, Nov 29, 2015 at 5:37 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Nov 29, 2015 at 10:37 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Nov 27, 2015 at 6:55 PM, Francesco Romani <fromani@redhat.com> wrote:
Using taskset, the ip command now takes a little longer to complete.
I fail to find the original reference for this. Why does it take longer? is it purely the additional taskset executable invocation? On busy system we do have these issues all the time, with lvm, etc…so I don’t think it’s significant
Since we always use the same set of CPUs, I assume using a mask (for 0 & 1, just use 0x3, as the man suggests) might be a tiny of a fraction faster to execute taskset with, instead of the need to translate the numeric CPU list.
Creating the string "0-<last cpu index>" is one line in vdsm. The code handling this in taskset is written in C, so the parsing time is practically zero. Even if it was non-zero, this code run once when we run a child process, so the cost is insignificant.
I think it's easier to just to have it as a mask in a config item somewhere, without need to create it or parse it anywhere. For us and for the user.
We have this option in /etc/vdsm/vdsm.conf:
# Comma separated whitelist of CPU cores on which VDSM is allowed to # run. The default is "", meaning VDSM can be scheduled by the OS to # run on any core. Valid examples: "1", "0,1", "0,2,3" # cpu_affinity = 1
I think this is the easiest option for users.
+1
However, the real concern is making sure CPUs 0 & 1 are not really too busy with stuff (including interrupt handling, etc.)
This code is used when we run a child process, to allow the child process to run on all cpus (in this case, cpu 0 and cpu 1). So I think there is no concern here.
Vdsm itself is running by default on cpu 1, which should be less busy then cpu 0.
I assume those are cores, which probably in a multi-socket will be in the first socket only. There's a good chance that the FC and or network/cards will also bind their interrupts to core0 & core 1 (check /proc/interrupts) on the same socket. From my poor laptop (1s, 4c): 42: 1487104 9329 4042 3598 IR-PCI-MSI 512000-edge 0000:00:1f.2
(my SATA controller)
43: 14664923 34 18 13 IR-PCI-MSI 327680-edge xhci_hcd (my dock station connector)
45: 6754579 4437 2501 2419 IR-PCI-MSI 32768-edge i915 (GPU)
47: 187409 11627 1235 1259 IR-PCI-MSI 2097152-edge iwlwifi (NIC, wifi)
Interesting, here an example from a 8 cores machine running my vms:
[nsoffer@jumbo ~]$ cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0: 31 0 0 0 0 0 0 0 IR-IO-APIC-edge timer 1: 2 0 0 1 0 0 0 0 IR-IO-APIC-edge i8042 8: 0 0 0 0 0 0 0 1 IR-IO-APIC-edge rtc0 9: 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi acpi 12: 3 0 0 0 0 0 1 0 IR-IO-APIC-edge i8042 16: 4 4 9 0 9 1 1 3 IR-IO-APIC 16-fasteoi ehci_hcd:usb3 23: 13 1 5 0 12 1 1 0 IR-IO-APIC 23-fasteoi ehci_hcd:usb4 24: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0 25: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar1 26: 3670 354 215 9062370 491 124 169 54 IR-PCI-MSI-edge 0000:00:1f.2 27: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 28: 166285414 0 3 0 4 0 0 0 IR-PCI-MSI-edge em1 29: 18 0 0 0 4 3 0 0 IR-PCI-MSI-edge mei_me 30: 1 151 17 0 3 169 26 94 IR-PCI-MSI-edge snd_hda_intel NMI: 2508 2296 2317 2356 867 918 912 903 Non-maskable interrupts LOC: 302996116 312923350 312295375 312089303 86282447 94046427 90847792 91761277 Local timer interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts PMI: 2508 2296 2317 2356 867 918 912 903 Performance monitoring interrupts IWI: 1 0 0 5 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries RES: 34480637 12953645 13139863 14309885 8881861 10110753 9709070 9703933 Rescheduling interrupts CAL: 7387779 7682087 7283716 7135792 2771105 1785528 1887493 1843734 Function call interrupts TLB: 11121 16458 17923 15216 8534 8173 8639 7837 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 7789 7789 7789 7789 7789 7789 7789 7789 Machine check polls HYP: 0 0 0 0 0 0 0 0 Hypervisor callback interrupts ERR: 0 MIS: 0
It seems that our default (CPU1) is fine.
I think it’s safe enough. Numbers above (and I checked the same on ppc with similar pattern) are for a reasonablt epty system. We can get a different picture when vdsm is busy. In general I think it’s indeed best to use the second online CPU for vdsm and all CPUs for child processes regarding exposing to users in UI - I think that’s way too low level. vdsm.conf is good enough Thanks, michal
Francesco, what do you think?
Nir
----- Original Message -----
From: "Michal Skrivanek" <mskrivan@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com>, "Francesco Romani" <fromani@redhat.com> Cc: "Yaniv Kaul" <ykaul@redhat.com>, "infra" <infra@ovirt.org>, "devel" <devel@ovirt.org> Sent: Monday, November 30, 2015 9:52:59 AM Subject: Re: [ovirt-devel] [vdsm] strange network test failure on FC23
On 29 Nov 2015, at 17:34, Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Nov 29, 2015 at 6:01 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Sun, Nov 29, 2015 at 5:37 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Nov 29, 2015 at 10:37 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Nov 27, 2015 at 6:55 PM, Francesco Romani <fromani@redhat.com> wrote:
Using taskset, the ip command now takes a little longer to complete.
I fail to find the original reference for this. Why does it take longer? is it purely the additional taskset executable invocation? On busy system we do have these issues all the time, with lvm, etc…so I don’t think it’s significant
Yep, that's only the overhead of taskset executable.
Since we always use the same set of CPUs, I assume using a mask (for 0 & 1, just use 0x3, as the man suggests) might be a tiny of a fraction faster to execute taskset with, instead of the need to translate the numeric CPU list.
Creating the string "0-<last cpu index>" is one line in vdsm. The code handling this in taskset is written in C, so the parsing time is practically zero. Even if it was non-zero, this code run once when we run a child process, so the cost is insignificant.
I think it's easier to just to have it as a mask in a config item somewhere, without need to create it or parse it anywhere. For us and for the user.
We have this option in /etc/vdsm/vdsm.conf:
# Comma separated whitelist of CPU cores on which VDSM is allowed to # run. The default is "", meaning VDSM can be scheduled by the OS to # run on any core. Valid examples: "1", "0,1", "0,2,3" # cpu_affinity = 1
I think this is the easiest option for users.
+1
+1, modulo the changes we need to fix https://bugzilla.redhat.com/show_bug.cgi?id=1286462 (patch is coming)
I assume those are cores, which probably in a multi-socket will be in the first socket only. There's a good chance that the FC and or network/cards will also bind their interrupts to core0 & core 1 (check /proc/interrupts) on the same socket. From my poor laptop (1s, 4c):
Yes, especially core0 (since 0 is nice defaults). This was the rationale behind the choice of cpu #1 in the first place.
It seems that our default (CPU1) is fine.
I think it’s safe enough. Numbers above (and I checked the same on ppc with similar pattern) are for a reasonablt epty system. We can get a different picture when vdsm is busy. In general I think it’s indeed best to use the second online CPU for vdsm and all CPUs for child processes
Agreed - except for cases like bz1286462 - but let's discuss this on gerrit/bz
regarding exposing to users in UI - I think that’s way too low level. vdsm.conf is good enough
Agreed. This is one thing that "just works". Bests, -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
participants (6)
-
Dan Kenigsberg -
David Caro -
Francesco Romani -
Michal Skrivanek -
Nir Soffer -
Yaniv Kaul