6:32 p.m.
--o99acAvKqrTZeiCU
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On 02/03, David Caro wrote:
=20
It took more than one hour :S
=20
Current status is that all the vms are up and running, all the services a=
re
working, but we have one host down, ovirt-srv02 is out of the pool of
hos=
ts,
with a strange issue resolving names.
=20
When running ping it can't resolve names, but with dig it works ok. That's
usually a misconfiguration in the nsswitch.conf file, but it's ok
I tried selinux and iptables.
=20
Did a strace of pinx, and I can see it does open a socket to the nameserv=
er and
sends the query, but nothing goes out the interface... (had tcpdump
open =
in
another screen)
=20
socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) =3D 4 <0.000786>
connect(4, {sa_family=3DAF_INET, sin_port=3Dhtons(53), sin_addr=3Dinet_ad=
dr("8.8.8.8")}, 16) =3D 0 <0.000028>
poll([{fd=3D4, events=3DPOLLOUT}], 1, 0) =3D 1 ([{fd=3D4,
revents=3DPO=
LLOUT}]) <0.000016>
sendto(4,
"ck\1\0\0\1\0\0\0\0\0\0\3www\6google\3com\0\0\1\0\1", 32, MSG_N=
OSIGNAL,
NULL, 0) =3D 32 <0.000052>
=20
Any idea is welcome, I'm going to get some sleep now that all the service=
s
are
back up...
Mystery solved! Thanks Max!
The issue was that the iproute2 module can edit multiple routing tables, and
the usual command 'ip route show' only shows the routes in the default kern=
el
table while newer vdsm adds a new table for the routing aside from it, and =
when
I modified the gateway and netmask in the routing tables of the hosts, the
extra side table was not updated. That lead to the strange behavior of ping=
udp
requests to the dns server being routed through the old gateway while icmp =
was
being routed though the new table.
You can find more info about routing tables and rules here:
http://linux-ip.net/html/routing-tables.html
http://linux-ip.net/html/routing-rpdb.html
and some examples here:
http://linux-ip.net/html/adv-multi-internet.html
Just for future reference, you can see all the routing for all the routing
tables with:
ip route show table all
=20
=20
ps. ovirt-srv02 was also the hosted engine master, and was the host that =
broke
during the upgrade the last time. Maybe it was related to this issue,
or =
this
issue is related to that...
=20
See you tomorrow!
=20
=20
On 02/02, David Caro wrote:
>=20
> Hi all,
>=20
> We are having a downtime on some of the vms and hosts on the phx lab. I=
t's
> caused by an unexpected issue with the engine and dhcp after
changing t=
he
> gateway of the machines to adapt to the new ip range.
>=20
> It's almost fixed, but we (I) should really get the environment to a re=
ally
> stable status again and finish the upgrade.
>=20
> There are still some issues, but most of them are already fixed, will f=
ix
the
> rest in less than one hour.
>=20
>=20
>=20
> --=20
> David Caro
>=20
> Red Hat S.L.
> Continuous Integration Engineer - EMEA ENG Virtualization R&D
>=20
> Tel.: +420 532 294 605
> Email: dcaro(a)redhat.com
> Web: www.redhat.com
> RHT Global #: 82-62605
=20
=20
=20
> _______________________________________________
> Infra mailing list
> Infra(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
=20
=20
--=20
David Caro
=20
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
=20
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--=20
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
--o99acAvKqrTZeiCU
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJU0OokAAoJEEBxx+HSYmnDCbAH/2OaIVm/vgMj48oiJE82LRfv
mgRxaU14IEzj4jZOE9b7xgjEZGpSDCdlpR2cINH6GSiE+RiygsSLETZE59R93y4Q
nFfdcWDq7LCFIxAWhSLbcRCh6ZwQVzotTFefOvw1mTTHOEFZVuK6XF+VUdnepBmC
WmJmgng2IZanjM/rCKU+ikr/Ql1NX6xjakXoxaiIkEXJIBgM82pyTdVuSelJ1z16
Krzbat8bzhByd7LL+HrxD77VXtYJEHmTDnpbkPjxtHdx2+wHUROr/ik62gPk/lAg
8mLwrIQKZhQbtDnFfz9L3cC/l84pLDbNedQE4HuhIOpr92HQl2y6Hm38Emgho7k=
=PYBI
-----END PGP SIGNATURE-----
--o99acAvKqrTZeiCU--