8:12 p.m.
--pf9I7BMVVzbSWLtt
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
We are having a major outage on phoenix lab, don't expect any vms/slaves to=
be
properly working yet.
Will update when solved or in an hour with the status.
--=20
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
--pf9I7BMVVzbSWLtt
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJU0QFzAAoJEEBxx+HSYmnDI6oH/im7CNuDufcx+wm6dVsFjv/J
kGfnSxDUZhcf8GqZgLotgXxhfM8LmSJorJkOAyCysZIWlAB+Hyzuw6eDuRoireur
2mULSrxdsCWzkn3/IHNKF+txmuREZ+zrwtR5ExiL5iAxK8smDiTYnUQ0+uAvlbKb
vQK7J8ApZYTXlwu7aggVaya5jVTFOPJny6eosec1hSTx8gLkuQcpuUWSP0AqbDxd
YWLgWqei0pa5Knz+/S35Rv/Kdw3m6gX5ZSgqWfyW+bMh+/Gf/0YsHrJ8LPvgBXXb
bgP59/P79zOkzDKwCKYeEkAQ7P3NwFBZIUrIOmoxVvex8AgdoLxEwXfuHgJT2Fs=
=jNwl
-----END PGP SIGNATURE-----
--pf9I7BMVVzbSWLtt--
9:48 p.m.
--+g7M9IMkV8truYOl
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Ok, update:
Not all the servers have been restored, most of the slave vms are up, and a=
ll
but one host are up.
Engine - Ok
storage -Ok
storage01 - Ok
storage02 - Ok
srv01 - DOWN
srv02 - OUT OF THE POOL (will add when 01 is up)
srv03 - OK
srv04 - OK
srv05 - OK
srv06 - OK
srv07 - OK
srv08 - OK
If you need any specific vm I can try to get it up on one of the running ho=
sts,
but I'd wait until the last host is up to start all of them.
Will update again when finished or in one hour.
On 02/03, David Caro wrote:
=20
We are having a major outage on phoenix lab, don't expect any vms/slaves =
to
be
properly working yet.
=20
Will update when solved or in an hour with the status.
=20
=20
--=20
David Caro
=20
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
=20
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--=20
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
--+g7M9IMkV8truYOl
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJU0RfoAAoJEEBxx+HSYmnDMs8IAKBOIsIFZMkWkplIXtfqoKfn
7+kPj+hgpAzIKOON3V68KNbGI0qVcma4ild4oISjIEmHHPqJLjsPxZWu2sF1iq6w
XSv1wG8M7KhCa8Gm+Pbz54LLCtxD4/S8HVkIBCti5kHxnLlJ8f3e0alcnTaKvntV
K1rTMrg5za4Xq2QfVxzX/nLlxnV45G+bpJl6qaSpVrIwvHukxLOOfskQWcGYVguH
ieUdt/ZGN0U+pad8fZWVwSuhkEFfQB0VMrv4PNjyAU5Nyp0vlhy74zP+6pup2ld/
cJxjAlCS98SNkYrBDD0VjT0klWLco4zsf9uwrfAx7JV9aNw3qyZZgg7+IYV3cyI=
=fVmR
-----END PGP SIGNATURE-----
--+g7M9IMkV8truYOl--
10:35 p.m.
--2/5bycvrmDh4d1IB
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
New update,
Host srv01 is up and running, but 02 and 03 have issues, they can't start up
any vms.
The error is in libvirt:
libvirtError: Child quit during startup handshake: Input/output error
Looking around I saw a thread in the users lists that fixed it with_
/usr/lib/systemd/systemd-vdsmd reconfigure force
That worked on srv01, but the others did not. So I'm trying to upgrade to f=
c20
one of them, the srv02, hoping the newer libvirt version will not have that
issue.
Those two hosts are the ones that are in the production data center, and it
has the foreman vm, so none of the slaves is working properly until that is=
solved.
Will update in ~one hour or when the problem is solved.
Being so late, if I get the production vms running in one host, I'll leave =
the
rest for tomorrow.
D
On 02/03, David Caro wrote:
=20
Ok, update:
=20
=20
Not all the servers have been restored, most of the slave vms are up, and=
all
but one host are up.
=20
Engine - Ok
storage -Ok
storage01 - Ok
storage02 - Ok
srv01 - DOWN
srv02 - OUT OF THE POOL (will add when 01 is up)
srv03 - OK
srv04 - OK
srv05 - OK
srv06 - OK
srv07 - OK
srv08 - OK
=20
=20
If you need any specific vm I can try to get it up on one of the running =
hosts,
but I'd wait until the last host is up to start all of them.
=20
=20
Will update again when finished or in one hour.
=20
=20
On 02/03, David Caro wrote:
>=20
> We are having a major outage on phoenix lab, don't expect any vms/slave=
s
to be
> properly working yet.
>=20
> Will update when solved or in an hour with the status.
>=20
>=20
> --=20
> David Caro
>=20
> Red Hat S.L.
> Continuous Integration Engineer - EMEA ENG Virtualization R&D
>=20
> Tel.: +420 532 294 605
> Email: dcaro(a)redhat.com
> Web: www.redhat.com
> RHT Global #: 82-62605
=20
=20
=20
> _______________________________________________
> Infra mailing list
> Infra(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
=20
=20
--=20
David Caro
=20
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
=20
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--=20
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
--2/5bycvrmDh4d1IB
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJU0SMeAAoJEEBxx+HSYmnDYCEIAIwQlGphK3w65SNnmRexN2Fe
eZta0vvIHqiLR7jXgzFux0WfwrgxQ6k4XiG0/4wvZzumXI2zzxTLyH3agqcqjPHu
JO+vzEg5hWLr/liHowj/4Qapi7HT5n51xwxdd7sCMOj9E6Q5kHkBvUkk2+8k4D3H
5mDEuwGGm1bqHF81SOLKR7fJbRh5rY85ShHtJyrrK+DYUxcRbO8D0P9JjcPGlEnL
DWVePF3QBGT73aumaUzC0NqpLI04pFJHECIQDHhgDgm5fL6QhMTZSG3RaXkk29NQ
/B2M0urf1D9pg3DbRaHBZ7kJdx1t+WCtJfZCrx48uMcRxqaKx9udBcADnDxhaXE=
=bZSt
-----END PGP SIGNATURE-----
--2/5bycvrmDh4d1IB--
11:50 p.m.
--WChQLJJJfbwij+9x
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Good news!
I got a vm working in the recently-upgraded fc20 host ovirt-srv02. The issue
with the vms seems to be that the default value for the numa setting is not
behaving correctly with libvirt. The fc19 vms just show the input/output er=
ror,
but the fc20 shows also the libvirt full string, and there you can see that=
it
complains about numa:
libvirtError: internal error: Process exited prior to exec: libvirt: error=
: internal error: NUMA memory tuning in 'preferred' mode only supports sin=
gle node
So what I've done is edit the vm, pin it to a node, set it as not migratable
(or whatever is the spelling) and changed the numa mode from preferred to
strict. Saved, and then edited the vm again, reverting the host pin and the
migration settings, but not changing the numa ones. That allowed me to boot=
one
of the vms so far (just tested).
Some ugly issues:
The know multipathd message in the logs... it's quite annoying and fills =
up
the logs.
Vdsm messed up the network a couple of times, once it removed all the ifc=
fg
files, and the other it restored old values in the rules/route files
Vdsm failed on vdsm-restore-net-config:89 with a non-existing key excepti=
on
insted of just showing an error message and continuing execution
I'll triage the above errors tomorrow and resend to the devels, for now just
sending to avoid forgetting about them.
Will continue booting the rest of production vms, do some simple sanity and
leave the rest for tomorrow.
On the good side, we have now one fc20 host on each cluster, and 3.5 on all=
the
production DC hosts! yay \o/
If anything comes up again I'll update in this thread, if not, tomorrow mor=
ning
I'll update when all the environment is working 100%
pd. Thanks Fabian and Max!!
On 02/03, David Caro wrote:
=20
New update,
=20
Host srv01 is up and running, but 02 and 03 have issues, they can't start=
up
any vms.
=20
The error is in libvirt:
=20
libvirtError: Child quit during startup handshake: Input/output error
=20
=20
Looking around I saw a thread in the users lists that fixed it with_
=20
/usr/lib/systemd/systemd-vdsmd reconfigure force
=20
That worked on srv01, but the others did not. So I'm trying to upgrade to=
fc20
one of them, the srv02, hoping the newer libvirt version will not
have th=
at
issue.
=20
Those two hosts are the ones that are in the production data center, and =
it
has the foreman vm, so none of the slaves is working properly until
that =
is solved.
=20
=20
Will update in ~one hour or when the problem is solved.
=20
Being so late, if I get the production vms running in one host, I'll leav=
e
the
rest for tomorrow.
=20
=20
D
=20
On 02/03, David Caro wrote:
>=20
> Ok, update:
>=20
>=20
> Not all the servers have been restored, most of the slave vms are up, a=
nd
all
> but one host are up.
>=20
> Engine - Ok
> storage -Ok
> storage01 - Ok
> storage02 - Ok
> srv01 - DOWN
> srv02 - OUT OF THE POOL (will add when 01 is up)
> srv03 - OK
> srv04 - OK
> srv05 - OK
> srv06 - OK
> srv07 - OK
> srv08 - OK
>=20
>=20
> If you need any specific vm I can try to get it up on one of the runnin=
g
hosts,
> but I'd wait until the last host is up to start all of
them.
>=20
>=20
> Will update again when finished or in one hour.
>=20
>=20
> On 02/03, David Caro wrote:
> >=20
> > We are having a major outage on phoenix lab, don't expect any vms/sla=
ves to be
> > properly working yet.
> >=20
> > Will update when solved or in an hour with the status.
> >=20
> >=20
> > --=20
> > David Caro
> >=20
> > Red Hat S.L.
> > Continuous Integration Engineer - EMEA ENG Virtualization R&D
> >=20
> > Tel.: +420 532 294 605
> > Email: dcaro(a)redhat.com
> > Web: www.redhat.com
> > RHT Global #: 82-62605
>=20
>=20
>=20
> > _______________________________________________
> > Infra mailing list
> > Infra(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
>=20
>=20
> --=20
> David Caro
>=20
> Red Hat S.L.
> Continuous Integration Engineer - EMEA ENG Virtualization R&D
>=20
> Tel.: +420 532 294 605
> Email: dcaro(a)redhat.com
> Web: www.redhat.com
> RHT Global #: 82-62605
=20
=20
=20
> _______________________________________________
> Infra mailing list
> Infra(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
=20
=20
--=20
David Caro
=20
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
=20
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--=20
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
--WChQLJJJfbwij+9x
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJU0TSGAAoJEEBxx+HSYmnDGV0H/0KgMp+lXs7IGgD2/jMVxwmI
9RFBnKVsPtOvsnUq6UbmUcT+z9MMeU/Ru/SqDeTJhfgUI4JB2nxBQVEKmgu7sbp3
i+qQfwc6N6Pt3UQvCBHJ3XX4CHc4KMXe0F8Y4OY3Hp4WHrn5h+2rD/BhLij/kKV8
YQDVZs78v8KwTcS22r4iK2elKahkWUWyodBBwKE5YFa2h3rxbn911dLmeoh9fwCX
9s+lacP0XD/8Cjbc2snQ/i9mjuB1iTw/o+9nzbl6ScTomH33ND5/8V7KRG1WFtf/
iQj4j68BWb7YYaxhyv4qE9QOwAOwwF5l/uIVfr5oDH4hYIWj4FfZxuBbRAVPeDQ=
=jquJ
-----END PGP SIGNATURE-----
--WChQLJJJfbwij+9x--
1:32 p.m.
----- Original Message -----
From: "David Caro" <dcaroest(a)redhat.com>
To: "Infra" <infra(a)ovirt.org>
Sent: Tuesday, February 3, 2015 10:50:14 PM
Subject: Re: Major outage
Good news!
I got a vm working in the recently-upgraded fc20 host ovirt-srv02. The issue
with the vms seems to be that the default value for the numa setting is not
behaving correctly with libvirt. The fc19 vms just show the input/output
error,
but the fc20 shows also the libvirt full string, and there you can see that
it
complains about numa:
libvirtError: internal error: Process exited prior to exec: libvirt: error :
internal error: NUMA memory tuning in 'preferred' mode only supports single
node
So what I've done is edit the vm, pin it to a node, set it as not migratable
(or whatever is the spelling) and changed the numa mode from preferred to
strict. Saved, and then edited the vm again, reverting the host pin and the
migration settings, but not changing the numa ones. That allowed me to boot
one
of the vms so far (just tested).
Some ugly issues:
The know multipathd message in the logs... it's quite annoying and fills up
the logs.
Vdsm messed up the network a couple of times, once it removed all the ifcfg
files, and the other it restored old values in the rules/route files
Vdsm failed on vdsm-restore-net-config:89 with a non-existing key exception
insted of just showing an error message and continuing execution
I'll triage the above errors tomorrow and resend to the devels, for now just
sending to avoid forgetting about them.
Will continue booting the rest of production vms, do some simple sanity and
leave the rest for tomorrow.
On the good side, we have now one fc20 host on each cluster, and 3.5 on all
the
production DC hosts! yay \o/
Great news!
adding some NUMA experts to see if they have an advise on optimizing it on the DC.
e.
If anything comes up again I'll update in this thread, if not, tomorrow
morning
I'll update when all the environment is working 100%
pd. Thanks Fabian and Max!!
On 02/03, David Caro wrote:
>
> New update,
>
> Host srv01 is up and running, but 02 and 03 have issues, they can't start
> up
> any vms.
>
> The error is in libvirt:
>
> libvirtError: Child quit during startup handshake: Input/output error
>
>
> Looking around I saw a thread in the users lists that fixed it with_
>
> /usr/lib/systemd/systemd-vdsmd reconfigure force
>
> That worked on srv01, but the others did not. So I'm trying to upgrade to
> fc20
> one of them, the srv02, hoping the newer libvirt version will not have that
> issue.
>
> Those two hosts are the ones that are in the production data center, and it
> has the foreman vm, so none of the slaves is working properly until that is
> solved.
>
>
> Will update in ~one hour or when the problem is solved.
>
> Being so late, if I get the production vms running in one host, I'll leave
> the
> rest for tomorrow.
>
>
> D
>
> On 02/03, David Caro wrote:
> >
> > Ok, update:
> >
> >
> > Not all the servers have been restored, most of the slave vms are up, and
> > all
> > but one host are up.
> >
> > Engine - Ok
> > storage -Ok
> > storage01 - Ok
> > storage02 - Ok
> > srv01 - DOWN
> > srv02 - OUT OF THE POOL (will add when 01 is up)
> > srv03 - OK
> > srv04 - OK
> > srv05 - OK
> > srv06 - OK
> > srv07 - OK
> > srv08 - OK
> >
> >
> > If you need any specific vm I can try to get it up on one of the running
> > hosts,
> > but I'd wait until the last host is up to start all of them.
> >
> >
> > Will update again when finished or in one hour.
> >
> >
> > On 02/03, David Caro wrote:
> > >
> > > We are having a major outage on phoenix lab, don't expect any
> > > vms/slaves to be
> > > properly working yet.
> > >
> > > Will update when solved or in an hour with the status.
> > >
> > >
> > > --
> > > David Caro
> > >
> > > Red Hat S.L.
> > > Continuous Integration Engineer - EMEA ENG Virtualization R&D
> > >
> > > Tel.: +420 532 294 605
> > > Email: dcaro(a)redhat.com
> > > Web: www.redhat.com
> > > RHT Global #: 82-62605
> >
> >
> >
> > > _______________________________________________
> > > Infra mailing list
> > > Infra(a)ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/infra
> >
> >
> > --
> > David Caro
> >
> > Red Hat S.L.
> > Continuous Integration Engineer - EMEA ENG Virtualization R&D
> >
> > Tel.: +420 532 294 605
> > Email: dcaro(a)redhat.com
> > Web: www.redhat.com
> > RHT Global #: 82-62605
>
>
>
> > _______________________________________________
> > Infra mailing list
> > Infra(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
>
>
> --
> David Caro
>
> Red Hat S.L.
> Continuous Integration Engineer - EMEA ENG Virtualization R&D
>
> Tel.: +420 532 294 605
> Email: dcaro(a)redhat.com
> Web: www.redhat.com
> RHT Global #: 82-62605
> _______________________________________________
> Infra mailing list
> Infra(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
--
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
3579
days inactive
3580
days old
4 comments
2 participants
participants (2)
-
David Caro -
Eyal Edri