4:18 p.m.
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--jqakiCaOmFP9f4FpGI29oNeauWoOxF3aM
Content-Type: multipart/mixed; boundary="66cCtMpFhTKBqqwQAsjplKCPdvE5mwege";
protected-headers="v1"
From: ~Stack~ <i.am.stack(a)gmail.com>
To: users <users(a)ovirt.org>
Message-ID: <4db29c1e-4031-aece-e736-855879c5c023(a)gmail.com>
Subject: oVirt management has lost its SSL.
--66cCtMpFhTKBqqwQAsjplKCPdvE5mwege
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
Greetings,
OS: Scientific Linux 7.4
oVirt: 4.1
Everything fully updated.
Everything was working great. I received my new network card today to
upgrade my ovirt management node (physical node; not self-hosted), took
the machine down, swapped the card, and brought it up to many many errors=
=2E
Here's the basic break-down of my discoveries.
1) My /etc/pki/ovirt-engine/.trustedstore was corrupt. I had lots of
messages in my engine.log about it being corrupt. Restored from backup,
and oVirt engine was really peeved for not having my domain cert in it
(tons of messages in the engine.log file)...figured out how to add my
domain cert and it seemed OK. Which led me to...
2) My /etc/pki/ovirt-engine/keys/engine.p12 and
/etc/pki/ovirt-engine/keys/apache.p12 are _gone_. Don't have them in my
backups either. This results in a massive java dump when I try to start
the engine service.
3) I noticed that I had
/etc/pki/ovirt-engine/keys/engine.p12.201711021302 which is a time stamp
corresponding to when I shut the node down. Then I noticed, that I was
missing dang near EVERY file in /etc/pki/ovirt-engine but I had an
equivalent file with the ".201711021302" extension. So a touch of bash
and I copied all of my "*.201711021302" files with the proper
user/group/permissions into their base name. Hooray! No more errors in
the log files and all services start!!
4) I open my web browser and head to my management host...and I get this
error:
Keystore was tampered with, or password was incorrect
Well...yeah. I had to fix it in step one. :-/
I'm not getting anything useful out of my Internet searching. I don't
know what went wrong or why, but my SSL is just borked.
Any suggestions? Thoughts? Ideas?
Is there a way to just blow away and start over with the SSL _without_
destroying the VM's (which fortunately they all seem to still be
functional!)?
Any help would be greatly appreciated.
Thanks!
~Stack~
--66cCtMpFhTKBqqwQAsjplKCPdvE5mwege--
--jqakiCaOmFP9f4FpGI29oNeauWoOxF3aM
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
--jqakiCaOmFP9f4FpGI29oNeauWoOxF3aM--
9:30 a.m.
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--6srqPwiDoNgJCMhM5k3HBhW43FakwQb0c
Content-Type: multipart/mixed; boundary="0T6GsGDl9lswqG1oKhuTrenOACKCmepeH";
protected-headers="v1"
From: ~Stack~ <i.am.stack(a)gmail.com>
To: users <users(a)ovirt.org>
Message-ID: <d0bca9f6-2251-f865-436e-9e82b24333b1(a)gmail.com>
Subject: Re: oVirt management has lost its SSL.
References: <4db29c1e-4031-aece-e736-855879c5c023(a)gmail.com>
In-Reply-To: <4db29c1e-4031-aece-e736-855879c5c023(a)gmail.com>
--0T6GsGDl9lswqG1oKhuTrenOACKCmepeH
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
Greetings,
Please, I would greatly appreciate some help/feedback. I'm not sure what
else to do.
I reverted the .trustedstore to the only backup I have, and there is one
key in it. That too gets flagged by oVirt as having been tampered with
(I'm guessing oVirt added something that isn't there any more). The
password is correct as I can verify it from the oVirt config file on the
command line.
I'm out of ideas on fixing this. What happens to my oVirt hypervisors
and VM's if I rebuild the management engine host from scratch?
Thanks!
~Stack~
On 11/02/2017 04:18 PM, ~Stack~ wrote:
the machine down, swapped the card, and brought it up to many many
erro=
rs.
backups either. This results in a massive java dump when I try to
start=
p
s
--0T6GsGDl9lswqG1oKhuTrenOACKCmepeH--
--6srqPwiDoNgJCMhM5k3HBhW43FakwQb0c
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
--6srqPwiDoNgJCMhM5k3HBhW43FakwQb0c--
12:15 p.m.
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--FplIUg6f4oTjf2uFF6d2cIwEkHoJ3XiFB
Content-Type: multipart/mixed; boundary="7L1K7tlmuTMbpGfdltWoF2kVahxXLS4w9";
protected-headers="v1"
From: ~Stack~ <i.am.stack(a)gmail.com>
To: users <users(a)ovirt.org>
Message-ID: <a8ae8ead-b6cc-c1d1-4dcb-785f4c41a57f(a)gmail.com>
Subject: Re: oVirt management has lost its SSL.
References: <4db29c1e-4031-aece-e736-855879c5c023(a)gmail.com>
<d0bca9f6-2251-f865-436e-9e82b24333b1(a)gmail.com>
In-Reply-To: <d0bca9f6-2251-f865-436e-9e82b24333b1(a)gmail.com>
--7L1K7tlmuTMbpGfdltWoF2kVahxXLS4w9
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
Greetings,
I'm seriously just grasping at straws here. I took a spare hard drive,
tossed it in the management host, and did a fresh install. It did not
like me trying to add it into the existing infrastructure. Tried to dump
the DB from the old to the new, update the passwords, and pretty much
ended up in the same place.
I did check the .trustedkeystore and it has the same 1 key as my
original back up. So that isn't the issue.
Still poking at it. Would love some thoughts/feedback.
Thanks!
~Stack~
On 11/03/2017 09:30 AM, ~Stack~ wrote:
Greetings,
=20
Please, I would greatly appreciate some help/feedback. I'm not sure wha=
t
else to do.
=20
I reverted the .trustedstore to the only backup I have, and there is on=
e
e
k
> the machine down, swapped the card, and brought it up to many
many err=
ors.
,
> and oVirt engine was really peeved for not having my domain cert
in it=
y
> backups either. This results in a massive java dump when I try to
star=
t
mp
> corresponding to when I shut the node down. Then I noticed, that
I was=
> and I copied all of my "*.201711021302" files with the
proper
> user/group/permissions into their base name. Hooray! No more errors in=
> the log files and all services start!!
>
> 4) I open my web browser and head to my management host...and I get th=
is
--7L1K7tlmuTMbpGfdltWoF2kVahxXLS4w9--
--FplIUg6f4oTjf2uFF6d2cIwEkHoJ3XiFB
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
--FplIUg6f4oTjf2uFF6d2cIwEkHoJ3XiFB--
12:41 p.m.
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--RIdq5PTfuKg5kKXBX1iAIR2goeFGXkjdE
Content-Type: multipart/mixed; boundary="KnJOald1h0BWIrxeKcmMEprQiQEeI3RIi";
protected-headers="v1"
From: ~Stack~ <i.am.stack(a)gmail.com>
To: Alexander Wels <awels(a)redhat.com>, users(a)ovirt.org
Message-ID: <322b0492-ac8f-f031-89a2-c9e88f704934(a)gmail.com>
Subject: Re: [ovirt-users] oVirt management has lost its SSL.
References: <4db29c1e-4031-aece-e736-855879c5c023(a)gmail.com>
<d0bca9f6-2251-f865-436e-9e82b24333b1(a)gmail.com>
<a8ae8ead-b6cc-c1d1-4dcb-785f4c41a57f(a)gmail.com> <3083657.k6sMFeiRm5@awels>
In-Reply-To: <3083657.k6sMFeiRm5@awels>
--KnJOald1h0BWIrxeKcmMEprQiQEeI3RIi
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
On 11/03/2017 12:23 PM, Alexander Wels wrote:
> tossed it in the management host, and did a fresh install. It did
not
> like me trying to add it into the existing infrastructure. Tried to du=
mp
Thanks for the suggestion. I've tried that. Twice. Still the same error.
"Keystore was tampered with, or password was incorrect."
=46rom digging around in the logs, it looks like it is trying to access
/etc/pki/ovirt-engine/.trustedstore but the password found in the ovirt
configs works just fine. So I know it is not a password issue.
I've been trying to figure out how that file is created so I can
possibly generate a new one, but no luck so far.
Thanks!
~Stack~
--KnJOald1h0BWIrxeKcmMEprQiQEeI3RIi--
--RIdq5PTfuKg5kKXBX1iAIR2goeFGXkjdE
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
--RIdq5PTfuKg5kKXBX1iAIR2goeFGXkjdE--
12:48 p.m.
On Friday, November 3, 2017 1:41:00 PM EDT ~Stack~ wrote:
AFAIC engine-setup will create the files needed. Try running engine-cleanup
and maybe it will remove everything needed and then running engine-setup
again?
But if all else fails you should be able to create a fresh engine, and after
you have added a host, you should be able to import the existing storage
domain (like you noted the VMs are still there).
1:17 p.m.
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--h3SR4RpK38rWMkS9h7ELfbT8hHMBAQnPq
Content-Type: multipart/mixed; boundary="MiSkDcn1FBPBUNr7Lcn8tsI7bFhrjKpek";
protected-headers="v1"
From: ~Stack~ <i.am.stack(a)gmail.com>
To: Alexander Wels <awels(a)redhat.com>
Cc: users(a)ovirt.org
Message-ID: <92281f4f-4245-32ba-e604-ef73cce78bd5(a)gmail.com>
Subject: Re: [ovirt-users] oVirt management has lost its SSL.
References: <4db29c1e-4031-aece-e736-855879c5c023(a)gmail.com>
<3083657.k6sMFeiRm5@awels> <322b0492-ac8f-f031-89a2-c9e88f704934(a)gmail.com>
<3618278.H5GOBpvrsq@awels>
In-Reply-To: <3618278.H5GOBpvrsq@awels>
--MiSkDcn1FBPBUNr7Lcn8tsI7bFhrjKpek
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
On 11/03/2017 12:48 PM, Alexander Wels wrote:
=20
AFAIC engine-setup will create the files needed. Try running engine-cle=
anup=20
and maybe it will remove everything needed and then running
engine-setu=
p=20
again?
=20
But if all else fails you should be able to create a fresh engine, and =
after=20
you have added a host, you should be able to import the existing
storag=
e=20
domain (like you noted the VMs are still there).
=20
=20
Greetings,
Thanks, but I've tried that too. Even though it did delete the keystore,
I ended up with the exact same error. :-(
I'm doing a fresh install right now. I've never done an import like this
before. I just connect the fresh install to one of my hosts and I can
import the others hosts/vms/configurations?
Thanks!
~Stack~
--MiSkDcn1FBPBUNr7Lcn8tsI7bFhrjKpek--
--h3SR4RpK38rWMkS9h7ELfbT8hHMBAQnPq
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
--h3SR4RpK38rWMkS9h7ELfbT8hHMBAQnPq--
2:47 p.m.
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--0PcAiSkaHrjjr6QFh7VrO0RDvQmbbBC2V
Content-Type: multipart/mixed; boundary="cA3CNKrjSQ9lfIk0caXaSXaUTXaE2jSaB";
protected-headers="v1"
From: ~Stack~ <i.am.stack(a)gmail.com>
To: users(a)ovirt.org
Message-ID: <a8c5336a-1c83-6c33-bb93-5c0641bc9f00(a)gmail.com>
Subject: Re: [ovirt-users] oVirt management has lost its SSL.
References: <4db29c1e-4031-aece-e736-855879c5c023(a)gmail.com>
<3083657.k6sMFeiRm5@awels> <322b0492-ac8f-f031-89a2-c9e88f704934(a)gmail.com>
<3618278.H5GOBpvrsq@awels> <92281f4f-4245-32ba-e604-ef73cce78bd5(a)gmail.com>
In-Reply-To: <92281f4f-4245-32ba-e604-ef73cce78bd5(a)gmail.com>
--cA3CNKrjSQ9lfIk0caXaSXaUTXaE2jSaB
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
On 11/03/2017 01:17 PM, ~Stack~ wrote:
On 11/03/2017 12:48 PM, Alexander Wels wrote:
> But if all else fails you should be able to create a fresh
engine, and=
after=20
> you have added a host, you should be able to import the existing
stora=
ge=20
,
I ended up with the exact same error. :-(
=20
I'm doing a fresh install right now. I've never done an import like thi=
s
Bender: Are we boned?
Leela: Yeah, we're boned
So I built a new management host from scratch. I added one of my hosts,
and immediately crashed the vm's running on that hypervisor (they all
just stopped responding). I don't know why they didn't fail over, but
they didn't. Oh well. At least the other hypervisor is up!
So I tried following this guide to import my storage domain from the
section "Disaster Recovery flows" for "Import file Storage Domain".
https://www.ovirt.org/develop/release-management/features/storage/imports=
toragedomain/
Yeah. That didn't work. It says it can't find any other domains to
import, but if I attempt to create a new one it says it can't because
there are existing domains!
Well, while I was poking at it the other VM's started acting up (crazy
high latency and the ovirt logs were really pissed at me). So I shut off
the ones that still responded, then shut down the other hypervisor. I
backed up the VM's on my NFS share, and created a new directory for the
data domain.
Guess I'm rebuilding my environment from scratch. I just hope I can get
some of the VM's to come back some how. :-/
~Stack~
--cA3CNKrjSQ9lfIk0caXaSXaUTXaE2jSaB--
--0PcAiSkaHrjjr6QFh7VrO0RDvQmbbBC2V
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
--0PcAiSkaHrjjr6QFh7VrO0RDvQmbbBC2V--
2758
days inactive
2761
days old
8 comments
3 participants
participants (3)
-
Alexander Wels -
Fred Rolland -
~Stack~