1:14 p.m.
--_000_41e0a4df7d7b4b04824f154982fe953fepsaero_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Hi All,
I'm still facing issues with my HE engine. Here are the steps that I took t=
o end up in this situation:
- Update Engine from 4.1.7 to 4.1.9
o That worked as expected
- Automatic Backup of Engine DB in the night
- Upgraded Engine from 4.1.9 to 4.2.1
o That worked fine
- Noticed Issues with the HA support for HE
o Cause was not having the latest ovirt-ha agent/broker version on hosts
- After updating the first host with the latest packages for the Ag=
ent/Broker engine was started twice
o As a result the Engine VM Disk was corrupted and there is no Backup of =
the Disk
o There is also no Backup of the Engine DB with version 4.2
- VM disk was repaired with fsck.ext4, but DB is corrupt
o Can't restore the Engine DB because the Backup DB from Engine V 4.1
- Rolled back all changes on Engine VM to 4.1.9 and imported Backup
o Checked for HA VMs to set as disabled and started the Engine
- Login is fine but the Engine is having trouble picking up and inf=
ormation from the Hosts
o No information on running VMs or hosts status
- Final Situation
o 2 Hosts have VMs still running and I can't stop those
o I still have the image of my corrupted Engine VM (v4.2)
Since there were no major changes after upgrading from 4.1 to 4.2, would it=
be possible to manually restore the 4.1 DB to the 4.2 Engine VM to this up=
and running again or are there modifications made to the DB on upgrading t=
hat are relevant for this ? All my work on rolling back to 4.1.9 with the D=
B restore failed as the Engine is not capable of picking up information fro=
m the hosts. Lessons learned is to always make a copy/snapshot of the engin=
e VM disk before upgrading anything. What are my options on getting back to=
a working environment ? Any help or hint is greatly appreciated.
Thank you,
Sven
--_000_41e0a4df7d7b4b04824f154982fe953fepsaero_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml"
xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word"
=
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml"
xmlns=3D"http:=
//www.w3.org/TR/REC-html40" <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Dus-ascii"=
<meta name=3D"Generator" content=3D"Microsoft
Word 15 (filtered medium)" <style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
span.E-MailFormatvorlage17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:371422817;
mso-list-type:hybrid;
mso-list-template-ids:1904265318 1097992454 67567619 67567621 67567617 675=
67619 67567621 67567617 67567619 67567621;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" / </xml><![endif]--><!--[if gte mso
9]><xml <o:shapelayout
v:ext=3D"edit" <o:idmap v:ext=3D"edit"
data=3D"1" /
</o:shapelayout></xml><![endif]--
</head <body lang=3D"DE"
link=3D"#0563C1" vlink=3D"#954F72" <div
class=3D"WordSection1" <p
class=3D"MsoNormal">Hi All, <o:p></o:p></p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal"><span
lang=3D"EN-US">I’m still facing issues w=
ith my HE engine. Here are the steps that I took to end up in this situatio=
n:
<o:p></o:p></span></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US"><o:p> </o:p></span></p <p class=3D"MsoListParagraph"
style=3D"text-indent:-18.0pt;mso-list:l0 leve=
l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span
style=3D"mso-list:=
Ignore">-<span style=3D"font:7.0pt "Times New
Roman"">  =
;
</span></span></span><![endif]><span
lang=3D"EN-US">Update Engine from 4.1.=
7 to 4.1.9
<o:p></o:p></span></p <p
class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0=
pt;mso-list:l0 level2 lfo1" <![if !supportLists]><span
lang=3D"EN-US" style=3D"font-family:"Courie=
r New""><span style=3D"mso-list:Ignore">o<span
style=3D"font:7.0pt &qu=
ot;Times New Roman"">
</span></span></span><![endif]><span
lang=3D"EN-US">That worked as expected=
<o:p></o:p></span></p <p
class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve=
l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span
style=3D"mso-list:=
Ignore">-<span style=3D"font:7.0pt "Times New
Roman"">  =
;
</span></span></span><![endif]><span
lang=3D"EN-US">Automatic Backup of Eng=
ine DB in the night<o:p></o:p></span></p <p
class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve=
l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span
style=3D"mso-list:=
Ignore">-<span style=3D"font:7.0pt "Times New
Roman"">  =
;
</span></span></span><![endif]><span
lang=3D"EN-US">Upgraded Engine from 4.=
1.9 to 4.2.1<o:p></o:p></span></p <p
class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0=
pt;mso-list:l0 level2 lfo1" <![if !supportLists]><span
lang=3D"EN-US" style=3D"font-family:"Courie=
r New""><span style=3D"mso-list:Ignore">o<span
style=3D"font:7.0pt &qu=
ot;Times New Roman"">
</span></span></span><![endif]><span
lang=3D"EN-US">That worked fine<o:p></=
o:p></span></p <p
class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve=
l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span
style=3D"mso-list:=
Ignore">-<span style=3D"font:7.0pt "Times New
Roman"">  =
;
</span></span></span><![endif]><span
lang=3D"EN-US">Noticed Issues with the=
HA support for HE<o:p></o:p></span></p <p
class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0=
pt;mso-list:l0 level2 lfo1" <![if !supportLists]><span
lang=3D"EN-US" style=3D"font-family:"Courie=
r New""><span style=3D"mso-list:Ignore">o<span
style=3D"font:7.0pt &qu=
ot;Times New Roman"">
</span></span></span><![endif]><span
lang=3D"EN-US">Cause was not having th=
e latest ovirt-ha agent/broker version on
hosts<o:p></o:p></span></p <p
class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve=
l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span
style=3D"mso-list:=
Ignore">-<span style=3D"font:7.0pt "Times New
Roman"">  =
;
</span></span></span><![endif]><span
lang=3D"EN-US">After updating the firs=
t host with the latest packages for the Agent/Broker engine was started twi=
ce
<o:p></o:p></span></p <p
class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0=
pt;mso-list:l0 level2 lfo1" <![if !supportLists]><span
lang=3D"EN-US" style=3D"font-family:"Courie=
r New""><span style=3D"mso-list:Ignore">o<span
style=3D"font:7.0pt &qu=
ot;Times New Roman"">
</span></span></span><![endif]><span
lang=3D"EN-US">As a result the Engine =
VM Disk was corrupted and there is no Backup of the
Disk<o:p></o:p></span><=
/p <p class=3D"MsoListParagraph"
style=3D"margin-left:72.0pt;text-indent:-18.0=
pt;mso-list:l0 level2 lfo1" <![if !supportLists]><span
lang=3D"EN-US" style=3D"font-family:"Courie=
r New""><span style=3D"mso-list:Ignore">o<span
style=3D"font:7.0pt &qu=
ot;Times New Roman"">
</span></span></span><![endif]><span
lang=3D"EN-US">There is also no Backup=
of the Engine DB with version 4.2<o:p></o:p></span></p <p class=3D"MsoListParagraph"
style=3D"text-indent:-18.0pt;mso-list:l0 leve=
l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span
style=3D"mso-list:=
Ignore">-<span style=3D"font:7.0pt "Times New
Roman"">  =
;
</span></span></span><![endif]><span
lang=3D"EN-US">VM disk was repaired wi=
th fsck.ext4, but DB is corrupt<o:p></o:p></span></p <p class=3D"MsoListParagraph"
style=3D"margin-left:72.0pt;text-indent:-18.0=
pt;mso-list:l0 level2 lfo1" <![if !supportLists]><span
lang=3D"EN-US" style=3D"font-family:"Courie=
r New""><span style=3D"mso-list:Ignore">o<span
style=3D"font:7.0pt &qu=
ot;Times New Roman"">
</span></span></span><![endif]><span
lang=3D"EN-US">Can’t restore the=
Engine DB because the Backup DB from Engine V
4.1<o:p></o:p></span></p <p
class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve=
l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span
style=3D"mso-list:=
Ignore">-<span style=3D"font:7.0pt "Times New
Roman"">  =
;
</span></span></span><![endif]><span
lang=3D"EN-US">Rolled back all changes=
on Engine VM to 4.1.9 and imported Backup
<o:p></o:p></span></p <p
class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0=
pt;mso-list:l0 level2 lfo1" <![if !supportLists]><span
lang=3D"EN-US" style=3D"font-family:"Courie=
r New""><span style=3D"mso-list:Ignore">o<span
style=3D"font:7.0pt &qu=
ot;Times New Roman"">
</span></span></span><![endif]><span
lang=3D"EN-US">Checked for HA VMs to s=
et as disabled and started the Engine<o:p></o:p></span></p <p class=3D"MsoListParagraph"
style=3D"text-indent:-18.0pt;mso-list:l0 leve=
l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span
style=3D"mso-list:=
Ignore">-<span style=3D"font:7.0pt "Times New
Roman"">  =
;
</span></span></span><![endif]><span
lang=3D"EN-US">Login is fine but the E=
ngine is having trouble picking up and information from the Hosts<o:p></o:p=
></span></p <p
class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0=
pt;mso-list:l0 level2 lfo1" <![if !supportLists]><span
lang=3D"EN-US" style=3D"font-family:"Courie=
r New""><span style=3D"mso-list:Ignore">o<span
style=3D"font:7.0pt &qu=
ot;Times New Roman"">
</span></span></span><![endif]><span
lang=3D"EN-US">No information on runni=
ng VMs or hosts status<o:p></o:p></span></p <p class=3D"MsoListParagraph"
style=3D"text-indent:-18.0pt;mso-list:l0 leve=
l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span
style=3D"mso-list:=
Ignore">-<span style=3D"font:7.0pt "Times New
Roman"">  =
;
</span></span></span><![endif]><span
lang=3D"EN-US">Final Situation<o:p></o=
:p></span></p <p
class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0=
pt;mso-list:l0 level2 lfo1" <![if !supportLists]><span
lang=3D"EN-US" style=3D"font-family:"Courie=
r New""><span style=3D"mso-list:Ignore">o<span
style=3D"font:7.0pt &qu=
ot;Times New Roman"">
</span></span></span><![endif]><span
lang=3D"EN-US">2 Hosts have VMs still =
running and I can’t stop those
<o:p></o:p></span></p <p
class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0=
pt;mso-list:l0 level2 lfo1" <![if !supportLists]><span
lang=3D"EN-US" style=3D"font-family:"Courie=
r New""><span style=3D"mso-list:Ignore">o<span
style=3D"font:7.0pt &qu=
ot;Times New Roman"">
</span></span></span><![endif]><span
lang=3D"EN-US">I still have the image =
of my corrupted Engine VM (v4.2)<o:p></o:p></span></p <p class=3D"MsoNormal"><span
lang=3D"EN-US"><o:p> </o:p></span></p <p class=3D"MsoNormal"><span
lang=3D"EN-US">Since there were no major chang=
es after upgrading from 4.1 to 4.2, would it be possible to manually restor=
e the 4.1 DB to the 4.2 Engine VM to this up and running again or are there=
modifications made to the DB on upgrading
that are relevant for this ? All my work on rolling back to 4.1.9 with the=
DB restore failed as the Engine is not capable of picking up information f=
rom the hosts. Lessons learned is to always make a copy/snapshot of the eng=
ine VM disk before upgrading anything.
What are my options on getting back to a working environment ? Any help or=
hint is greatly appreciated.<o:p></o:p></span></p <p class=3D"MsoNormal"><span
lang=3D"EN-US"><o:p> </o:p></span></p <p class=3D"MsoNormal"><span
lang=3D"EN-US">Thank you, <o:p></o:p></span></=
p <p class=3D"MsoNormal"><span
lang=3D"EN-US">Sven <o:p></o:p></span></=
p </div
</body </html
--_000_41e0a4df7d7b4b04824f154982fe953fepsaero_--
3:06 a.m.
On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik <Sven.Achtelik(a)eps.aero> wrote:
Hi All,
I’m still facing issues with my HE engine. Here are the steps that I took to
end up in this situation:
- Update Engine from 4.1.7 to 4.1.9
o That worked as expected
- Automatic Backup of Engine DB in the night
- Upgraded Engine from 4.1.9 to 4.2.1
o That worked fine
- Noticed Issues with the HA support for HE
o Cause was not having the latest ovirt-ha agent/broker version on hosts
- After updating the first host with the latest packages for the
Agent/Broker engine was started twice
o As a result the Engine VM Disk was corrupted and there is no Backup of
the Disk
o There is also no Backup of the Engine DB with version 4.2
- VM disk was repaired with fsck.ext4, but DB is corrupt
o Can’t restore the Engine DB because the Backup DB from Engine V 4.1
- Rolled back all changes on Engine VM to 4.1.9 and imported Backup
o Checked for HA VMs to set as disabled and started the Engine
- Login is fine but the Engine is having trouble picking up and
information from the Hosts
o No information on running VMs or hosts status
- Final Situation
o 2 Hosts have VMs still running and I can’t stop those
o I still have the image of my corrupted Engine VM (v4.2)
Since there were no major changes after upgrading from 4.1 to 4.2, would it
be possible to manually restore the 4.1 DB to the 4.2 Engine VM to this up
and running again or are there modifications made to the DB on upgrading
that are relevant for this ?
engine-backup requires restoring to the same version used to take the backup,
with a single exception - on 4.0, it can restore 3.6.
It's very easy to patch it to allow also 4.1->4.2, search inside it for
"VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever tested
this, so no idea might break. In 3.6->4.0 days, we did have to fix a few
other things, notably apache httpd and iptables->firewalld:
https://bugzilla.redhat.com/show_bug.cgi?id=1318580
All my work on rolling back to 4.1.9 with the
DB restore failed as the Engine is not capable of picking up information
from the hosts.
No idea why, but not sure it's related to your restore flow.
Lessons learned is to always make a copy/snapshot of the
engine VM disk before upgrading anything.
If it's a hosted-engine, this isn't supported - see my reply on the
list ~ 1 hour ago...
What are my options on getting
back to a working environment ? Any help or hint is greatly appreciated.
Restore again with either methods - what you tried, or patching engine-backup
and restore directly into 4.2 - and if the engine fails to talk to the hosts,
try to debug/fix this.
If you suspect corruption more severe that just the db, you can install a
fresh engine machine from scratch and restore to it. If it's a hosted-engine,
you'll need to deploy hosted-engine from scratch, check docs about hosted-engine
backup/restore.
Best regards,
--
Didi
3:50 a.m.
-----Ursprüngliche Nachricht-----
Von: Yedidyah Bar David [mailto:didi@redhat.com]
Gesendet: Mittwoch, 28. März 2018 10:06
An: Sven Achtelik
Cc: users(a)ovirt.org
Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before
upgrading to 4.2
On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik <Sven.Achtelik(a)eps.aero>
wrote:
> Hi All,
>
>
>
> I’m still facing issues with my HE engine. Here are the steps that I
> took to end up in this situation:
>
>
>
> - Update Engine from 4.1.7 to 4.1.9
>
> o That worked as expected
>
> - Automatic Backup of Engine DB in the night
>
> - Upgraded Engine from 4.1.9 to 4.2.1
>
> o That worked fine
>
> - Noticed Issues with the HA support for HE
>
> o Cause was not having the latest ovirt-ha agent/broker version on hosts
>
> - After updating the first host with the latest packages for the
> Agent/Broker engine was started twice
>
> o As a result the Engine VM Disk was corrupted and there is no Backup of
> the Disk
>
> o There is also no Backup of the Engine DB with version 4.2
>
> - VM disk was repaired with fsck.ext4, but DB is corrupt
>
> o Can’t restore the Engine DB because the Backup DB from Engine V 4.1
>
> - Rolled back all changes on Engine VM to 4.1.9 and imported Backup
>
> o Checked for HA VMs to set as disabled and started the Engine
>
> - Login is fine but the Engine is having trouble picking up and
> information from the Hosts
>
> o No information on running VMs or hosts status
>
> - Final Situation
>
> o 2 Hosts have VMs still running and I can’t stop those
>
> o I still have the image of my corrupted Engine VM (v4.2)
>
>
>
> Since there were no major changes after upgrading from 4.1 to 4.2,
> would it be possible to manually restore the 4.1 DB to the 4.2 Engine
> VM to this up and running again or are there modifications made to the
> DB on upgrading that are relevant for this ?
engine-backup requires restoring to the same version used to take the backup,
with a single exception - on 4.0, it can restore 3.6.
It's very easy to patch it to allow also 4.1->4.2, search inside it for
"VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever
tested this, so no idea might break. In 3.6->4.0 days, we did have to fix a few
other things, notably apache httpd and iptables->firewalld:
https://bugzilla.redhat.com/show_bug.cgi?id=1318580
> All my work on rolling back to 4.1.9 with the DB restore failed as the
> Engine is not capable of picking up information from the hosts.
No idea why, but not sure it's related to your restore flow.
> Lessons learned is to always make a copy/snapshot of the engine VM
> disk before upgrading anything.
If it's a hosted-engine, this isn't supported - see my reply on the list ~ 1
hour
ago...
> What are my options on getting
> back to a working environment ? Any help or hint is greatly appreciated.
Restore again with either methods - what you tried, or patching engine-
backup and restore directly into 4.2 - and if the engine fails to talk to the hosts,
try to debug/fix this.
If you suspect corruption more severe that just the db, you can install a fresh
engine machine from scratch and restore to it. If it's a hosted-engine, you'll
need to deploy hosted-engine from scratch, check docs about hosted-engine
backup/restore.
Will the setup of hosted engine from scratch require a new storage domain for the new
engine or can I use the one that is already there ? What about the VMs running on my
hosts, will they be effected by that ? It might be best to start with a fresh VM.
Best regards,
--
Didi
11:27 a.m.
-----Ursprüngliche Nachricht-----
Von: Yedidyah Bar David [mailto:didi@redhat.com]
Gesendet: Mittwoch, 28. März 2018 10:06
An: Sven Achtelik
Cc: users(a)ovirt.org
Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before
upgrading to 4.2
On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik <Sven.Achtelik(a)eps.aero>
wrote:
> Hi All,
>
>
>
> I’m still facing issues with my HE engine. Here are the steps that I
> took to end up in this situation:
>
>
>
> - Update Engine from 4.1.7 to 4.1.9
>
> o That worked as expected
>
> - Automatic Backup of Engine DB in the night
>
> - Upgraded Engine from 4.1.9 to 4.2.1
>
> o That worked fine
>
> - Noticed Issues with the HA support for HE
>
> o Cause was not having the latest ovirt-ha agent/broker version on hosts
>
> - After updating the first host with the latest packages for the
> Agent/Broker engine was started twice
>
> o As a result the Engine VM Disk was corrupted and there is no Backup of
> the Disk
>
> o There is also no Backup of the Engine DB with version 4.2
>
> - VM disk was repaired with fsck.ext4, but DB is corrupt
>
> o Can’t restore the Engine DB because the Backup DB from Engine V 4.1
>
> - Rolled back all changes on Engine VM to 4.1.9 and imported Backup
>
> o Checked for HA VMs to set as disabled and started the Engine
>
> - Login is fine but the Engine is having trouble picking up and
> information from the Hosts
>
> o No information on running VMs or hosts status
>
> - Final Situation
>
> o 2 Hosts have VMs still running and I can’t stop those
>
> o I still have the image of my corrupted Engine VM (v4.2)
>
>
>
> Since there were no major changes after upgrading from 4.1 to 4.2,
> would it be possible to manually restore the 4.1 DB to the 4.2 Engine
> VM to this up and running again or are there modifications made to the
> DB on upgrading that are relevant for this ?
engine-backup requires restoring to the same version used to take the backup,
with a single exception - on 4.0, it can restore 3.6.
It's very easy to patch it to allow also 4.1->4.2, search inside it for
"VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever
tested this, so no idea might break. In 3.6->4.0 days, we did have to fix a few
other things, notably apache httpd and iptables->firewalld:
https://bugzilla.redhat.com/show_bug.cgi?id=1318580
> All my work on rolling back to 4.1.9 with the DB restore failed as the
> Engine is not capable of picking up information from the hosts.
No idea why, but not sure it's related to your restore flow.
> Lessons learned is to always make a copy/snapshot of the engine VM
> disk before upgrading anything.
If it's a hosted-engine, this isn't supported - see my reply on the list ~ 1
hour
ago...
> What are my options on getting
> back to a working environment ? Any help or hint is greatly appreciated.
Restore again with either methods - what you tried, or patching engine-
backup and restore directly into 4.2 - and if the engine fails to talk to the hosts,
try to debug/fix this.
If you suspect corruption more severe that just the db, you can install a fresh
engine machine from scratch and restore to it. If it's a hosted-engine, you'll
need to deploy hosted-engine from scratch, check docs about hosted-engine
backup/restore.
I read through those documents and it seems that I would need an extra Host/Hardware which
I don't have.
https://ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring...
So how would I be able to get a new setup working when I would like to use the
Engine-VM-Image ? At this point it sounds like I would have to manually reinstall the
machine that is left over and running. I'm lost at this point.
Best regards,
--
Didi
4 a.m.
Hi All,
I got my stuff up and running again. I works like described in the manual and I used some
extra hardware to jumpstart this. I'm now back on my hosted Engine 4.1.9 with 3 Hosts
running it. The Engine is running on the appliance that is pulled by the deployment tool
and after having everything stable again I thought of upgrading to 4.2. Thing is that this
is just not working with the appliance because of some issue when upgrading Postgres
inside. Looking at the logs I found this:
----------------------------
Creating dump of database schemas
engine
ovirt_engine_history
postgres
template1
ok
encodings for database "postgres" do not match: old "SQL_ASCII", new
"UTF8"
Failure, exiting
------------------------------
After some research I found something here
https://bugzilla.redhat.com/show_bug.cgi?id=1525976, but I'm not sure what to do with
that Information. I used the appliance and didn't do anything manual in the complete
process and I'm wondering why I'm getting this issue now ? Could someone advice on
how to proceed ?
Thank you,
Sven
-----Ursprüngliche Nachricht-----
Von: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] Im Auftrag
von Sven Achtelik
Gesendet: Mittwoch, 28. März 2018 18:28
An: Yedidyah Bar David
Cc: users(a)ovirt.org
Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before
upgrading to 4.2
> -----Ursprüngliche Nachricht-----
> Von: Yedidyah Bar David [mailto:didi@redhat.com]
> Gesendet: Mittwoch, 28. März 2018 10:06
> An: Sven Achtelik
> Cc: users(a)ovirt.org
> Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup
> before upgrading to 4.2
>
> On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik
> <Sven.Achtelik(a)eps.aero>
> wrote:
> > Hi All,
> >
> >
> >
> > I’m still facing issues with my HE engine. Here are the steps that I
> > took to end up in this situation:
> >
> >
> >
> > - Update Engine from 4.1.7 to 4.1.9
> >
> > o That worked as expected
> >
> > - Automatic Backup of Engine DB in the night
> >
> > - Upgraded Engine from 4.1.9 to 4.2.1
> >
> > o That worked fine
> >
> > - Noticed Issues with the HA support for HE
> >
> > o Cause was not having the latest ovirt-ha agent/broker version on hosts
> >
> > - After updating the first host with the latest packages for the
> > Agent/Broker engine was started twice
> >
> > o As a result the Engine VM Disk was corrupted and there is no Backup of
> > the Disk
> >
> > o There is also no Backup of the Engine DB with version 4.2
> >
> > - VM disk was repaired with fsck.ext4, but DB is corrupt
> >
> > o Can’t restore the Engine DB because the Backup DB from Engine V 4.1
> >
> > - Rolled back all changes on Engine VM to 4.1.9 and imported Backup
> >
> > o Checked for HA VMs to set as disabled and started the Engine
> >
> > - Login is fine but the Engine is having trouble picking up and
> > information from the Hosts
> >
> > o No information on running VMs or hosts status
> >
> > - Final Situation
> >
> > o 2 Hosts have VMs still running and I can’t stop those
> >
> > o I still have the image of my corrupted Engine VM (v4.2)
> >
> >
> >
> > Since there were no major changes after upgrading from 4.1 to 4.2,
> > would it be possible to manually restore the 4.1 DB to the 4.2
> > Engine VM to this up and running again or are there modifications
> > made to the DB on upgrading that are relevant for this ?
>
> engine-backup requires restoring to the same version used to take the
> backup, with a single exception - on 4.0, it can restore 3.6.
>
> It's very easy to patch it to allow also 4.1->4.2, search inside it
> for "VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever
> tested this, so no idea might break. In 3.6->4.0 days, we did have to
> fix a few other things, notably apache httpd and iptables->firewalld:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1318580
>
> > All my work on rolling back to 4.1.9 with the DB restore failed as
> > the Engine is not capable of picking up information from the hosts.
>
> No idea why, but not sure it's related to your restore flow.
>
> > Lessons learned is to always make a copy/snapshot of the engine VM
> > disk before upgrading anything.
>
> If it's a hosted-engine, this isn't supported - see my reply on the
> list ~ 1 hour ago...
>
> > What are my options on getting
> > back to a working environment ? Any help or hint is greatly appreciated.
>
> Restore again with either methods - what you tried, or patching
> engine- backup and restore directly into 4.2 - and if the engine fails
> to talk to the hosts, try to debug/fix this.
>
> If you suspect corruption more severe that just the db, you can
> install a fresh engine machine from scratch and restore to it. If it's
> a hosted-engine, you'll need to deploy hosted-engine from scratch,
> check docs about hosted-engine backup/restore.
I read through those documents and it seems that I would need an extra
Host/Hardware which I don't have. https://ovirt.org/documentation/self-
hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-
Hosted_Environment/
So how would I be able to get a new setup working when I would like to use the
Engine-VM-Image ? At this point it sounds like I would have to manually
reinstall the machine that is left over and running. I'm lost at this point.
>
> Best regards,
> --
> Didi
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
3:48 a.m.
Hi Sven,
I've found this one: https://gist.github.com/turboladen/6790847
I think it should help :)
*Regards,*
*Shani Leviim*
On Fri, Apr 13, 2018 at 12:00 PM, Sven Achtelik <Sven.Achtelik(a)eps.aero>
wrote:
Hi All,
I got my stuff up and running again. I works like described in the manual
and I used some extra hardware to jumpstart this. I'm now back on my hosted
Engine 4.1.9 with 3 Hosts running it. The Engine is running on the
appliance that is pulled by the deployment tool and after having everything
stable again I thought of upgrading to 4.2. Thing is that this is just not
working with the appliance because of some issue when upgrading Postgres
inside. Looking at the logs I found this:
----------------------------
Creating dump of database schemas
engine
ovirt_engine_history
postgres
template1
ok
encodings for database "postgres" do not match: old "SQL_ASCII",
new
"UTF8"
Failure, exiting
------------------------------
After some research I found something here https://bugzilla.redhat.com/
show_bug.cgi?id=1525976, but I'm not sure what to do with that
Information. I used the appliance and didn't do anything manual in the
complete process and I'm wondering why I'm getting this issue now ? Could
someone advice on how to proceed ?
Thank you,
Sven
> -----Ursprüngliche Nachricht-----
> Von: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] Im Auftrag
> von Sven Achtelik
> Gesendet: Mittwoch, 28. März 2018 18:28
> An: Yedidyah Bar David
> Cc: users(a)ovirt.org
> Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before
> upgrading to 4.2
>
>
>
> > -----Ursprüngliche Nachricht-----
> > Von: Yedidyah Bar David [mailto:didi@redhat.com]
> > Gesendet: Mittwoch, 28. März 2018 10:06
> > An: Sven Achtelik
> > Cc: users(a)ovirt.org
> > Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup
> > before upgrading to 4.2
> >
> > On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik
> > <Sven.Achtelik(a)eps.aero>
> > wrote:
> > > Hi All,
> > >
> > >
> > >
> > > I’m still facing issues with my HE engine. Here are the steps that I
> > > took to end up in this situation:
> > >
> > >
> > >
> > > - Update Engine from 4.1.7 to 4.1.9
> > >
> > > o That worked as expected
> > >
> > > - Automatic Backup of Engine DB in the night
> > >
> > > - Upgraded Engine from 4.1.9 to 4.2.1
> > >
> > > o That worked fine
> > >
> > > - Noticed Issues with the HA support for HE
> > >
> > > o Cause was not having the latest ovirt-ha agent/broker version on
hosts
> > >
> > > - After updating the first host with the latest packages for
the
> > > Agent/Broker engine was started twice
> > >
> > > o As a result the Engine VM Disk was corrupted and there is no
Backup of
> > > the Disk
> > >
> > > o There is also no Backup of the Engine DB with version 4.2
> > >
> > > - VM disk was repaired with fsck.ext4, but DB is corrupt
> > >
> > > o Can’t restore the Engine DB because the Backup DB from Engine V
4.1
> > >
> > > - Rolled back all changes on Engine VM to 4.1.9 and imported
Backup
> > >
> > > o Checked for HA VMs to set as disabled and started the Engine
> > >
> > > - Login is fine but the Engine is having trouble picking up
and
> > > information from the Hosts
> > >
> > > o No information on running VMs or hosts status
> > >
> > > - Final Situation
> > >
> > > o 2 Hosts have VMs still running and I can’t stop those
> > >
> > > o I still have the image of my corrupted Engine VM (v4.2)
> > >
> > >
> > >
> > > Since there were no major changes after upgrading from 4.1 to 4.2,
> > > would it be possible to manually restore the 4.1 DB to the 4.2
> > > Engine VM to this up and running again or are there modifications
> > > made to the DB on upgrading that are relevant for this ?
> >
> > engine-backup requires restoring to the same version used to take the
> > backup, with a single exception - on 4.0, it can restore 3.6.
> >
> > It's very easy to patch it to allow also 4.1->4.2, search inside it
> > for "VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone
ever
> > tested this, so no idea might break. In 3.6->4.0 days, we did have to
> > fix a few other things, notably apache httpd and iptables->firewalld:
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=1318580
> >
> > > All my work on rolling back to 4.1.9 with the DB restore failed as
> > > the Engine is not capable of picking up information from the hosts.
> >
> > No idea why, but not sure it's related to your restore flow.
> >
> > > Lessons learned is to always make a copy/snapshot of the engine VM
> > > disk before upgrading anything.
> >
> > If it's a hosted-engine, this isn't supported - see my reply on the
> > list ~ 1 hour ago...
> >
> > > What are my options on getting
> > > back to a working environment ? Any help or hint is greatly
appreciated.
> >
> > Restore again with either methods - what you tried, or patching
> > engine- backup and restore directly into 4.2 - and if the engine fails
> > to talk to the hosts, try to debug/fix this.
> >
> > If you suspect corruption more severe that just the db, you can
> > install a fresh engine machine from scratch and restore to it. If it's
> > a hosted-engine, you'll need to deploy hosted-engine from scratch,
> > check docs about hosted-engine backup/restore.
>
> I read through those documents and it seems that I would need an extra
> Host/Hardware which I don't have. https://ovirt.org/documentation/self-
> hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-
> Hosted_Environment/
>
> So how would I be able to get a new setup working when I would like to
use the
> Engine-VM-Image ? At this point it sounds like I would have to manually
> reinstall the machine that is left over and running. I'm lost at this
point.
> >
> > Best regards,
> > --
> > Didi
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
1:55 a.m.
On Fri, Apr 13, 2018 at 12:00 PM, Sven Achtelik <Sven.Achtelik(a)eps.aero> wrote:
Hi All,
I got my stuff up and running again. I works like described in the manual and I used some
extra hardware to jumpstart this. I'm now back on my hosted Engine 4.1.9 with 3 Hosts
running it. The Engine is running on the appliance that is pulled by the deployment tool
and after having everything stable again I thought of upgrading to 4.2. Thing is that this
is just not working with the appliance because of some issue when upgrading Postgres
inside. Looking at the logs I found this:
----------------------------
Creating dump of database schemas
engine
ovirt_engine_history
postgres
template1
ok
encodings for database "postgres" do not match: old "SQL_ASCII", new
"UTF8"
Failure, exiting
------------------------------
After some research I found something here
https://bugzilla.redhat.com/show_bug.cgi?id=1525976, but I'm not sure what to do with
that Information. I used the appliance and didn't do anything manual in the complete
process and I'm wondering why I'm getting this issue now ? Could someone advice on
how to proceed ?
Looks like:
https://bugzilla.redhat.com/1528371
Which version do you upgrade to?
If to one that should be covered by above bug, please attach your
setup log to it. Thanks.
I was on vacation last week and will be in next one too. Adding Simone.
Best regards,
Thank you,
Sven
> -----Ursprüngliche Nachricht-----
> Von: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] Im Auftrag
> von Sven Achtelik
> Gesendet: Mittwoch, 28. März 2018 18:28
> An: Yedidyah Bar David
> Cc: users(a)ovirt.org
> Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before
> upgrading to 4.2
>
>
>
> > -----Ursprüngliche Nachricht-----
> > Von: Yedidyah Bar David [mailto:didi@redhat.com]
> > Gesendet: Mittwoch, 28. März 2018 10:06
> > An: Sven Achtelik
> > Cc: users(a)ovirt.org
> > Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup
> > before upgrading to 4.2
> >
> > On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik
> > <Sven.Achtelik(a)eps.aero>
> > wrote:
> > > Hi All,
> > >
> > >
> > >
> > > I’m still facing issues with my HE engine. Here are the steps that I
> > > took to end up in this situation:
> > >
> > >
> > >
> > > - Update Engine from 4.1.7 to 4.1.9
> > >
> > > o That worked as expected
> > >
> > > - Automatic Backup of Engine DB in the night
> > >
> > > - Upgraded Engine from 4.1.9 to 4.2.1
> > >
> > > o That worked fine
> > >
> > > - Noticed Issues with the HA support for HE
> > >
> > > o Cause was not having the latest ovirt-ha agent/broker version on hosts
> > >
> > > - After updating the first host with the latest packages for the
> > > Agent/Broker engine was started twice
> > >
> > > o As a result the Engine VM Disk was corrupted and there is no Backup of
> > > the Disk
> > >
> > > o There is also no Backup of the Engine DB with version 4.2
> > >
> > > - VM disk was repaired with fsck.ext4, but DB is corrupt
> > >
> > > o Can’t restore the Engine DB because the Backup DB from Engine V 4.1
> > >
> > > - Rolled back all changes on Engine VM to 4.1.9 and imported
Backup
> > >
> > > o Checked for HA VMs to set as disabled and started the Engine
> > >
> > > - Login is fine but the Engine is having trouble picking up and
> > > information from the Hosts
> > >
> > > o No information on running VMs or hosts status
> > >
> > > - Final Situation
> > >
> > > o 2 Hosts have VMs still running and I can’t stop those
> > >
> > > o I still have the image of my corrupted Engine VM (v4.2)
> > >
> > >
> > >
> > > Since there were no major changes after upgrading from 4.1 to 4.2,
> > > would it be possible to manually restore the 4.1 DB to the 4.2
> > > Engine VM to this up and running again or are there modifications
> > > made to the DB on upgrading that are relevant for this ?
> >
> > engine-backup requires restoring to the same version used to take the
> > backup, with a single exception - on 4.0, it can restore 3.6.
> >
> > It's very easy to patch it to allow also 4.1->4.2, search inside it
> > for "VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever
> > tested this, so no idea might break. In 3.6->4.0 days, we did have to
> > fix a few other things, notably apache httpd and iptables->firewalld:
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=1318580
> >
> > > All my work on rolling back to 4.1.9 with the DB restore failed as
> > > the Engine is not capable of picking up information from the hosts.
> >
> > No idea why, but not sure it's related to your restore flow.
> >
> > > Lessons learned is to always make a copy/snapshot of the engine VM
> > > disk before upgrading anything.
> >
> > If it's a hosted-engine, this isn't supported - see my reply on the
> > list ~ 1 hour ago...
> >
> > > What are my options on getting
> > > back to a working environment ? Any help or hint is greatly appreciated.
> >
> > Restore again with either methods - what you tried, or patching
> > engine- backup and restore directly into 4.2 - and if the engine fails
> > to talk to the hosts, try to debug/fix this.
> >
> > If you suspect corruption more severe that just the db, you can
> > install a fresh engine machine from scratch and restore to it. If it's
> > a hosted-engine, you'll need to deploy hosted-engine from scratch,
> > check docs about hosted-engine backup/restore.
>
> I read through those documents and it seems that I would need an extra
> Host/Hardware which I don't have. https://ovirt.org/documentation/self-
> hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-
> Hosted_Environment/
>
> So how would I be able to get a new setup working when I would like to use the
> Engine-VM-Image ? At this point it sounds like I would have to manually
> reinstall the machine that is left over and running. I'm lost at this point.
> >
> > Best regards,
> > --
> > Didi
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
--
Didi
5:09 a.m.
I added the 4.2 Repos and this what the yum update installed the following packages
[root@ovirt-engine ~]# yum list "ovirt-*-setup*"
Loaded plugins: fastestmirror, versionlock
Determining fastest mirrors
* base: centos.host-engine.com
* extras: mirror.riverfrontnetworks.com
* ovirt-4.1-epel: mirrors.lug.mtu.edu
* ovirt-4.2: resources.ovirt.org
* ovirt-4.2-epel: mirrors.lug.mtu.edu
* updates: mirrors.advancedhosters.com
Installed Packages
ovirt-engine-dwh-setup.noarch
4.2.2.2-1.el7.centos
@ovirt-4.2
ovirt-engine-setup.noarch
4.2.2.6-1.el7.centos
@ovirt-4.2
ovirt-engine-setup-base.noarch
4.2.2.6-1.el7.centos
@ovirt-4.2
ovirt-engine-setup-plugin-ovirt-engine.noarch
4.2.2.6-1.el7.centos
@ovirt-4.2
ovirt-engine-setup-plugin-ovirt-engine-common.noarch
4.2.2.6-1.el7.centos
@ovirt-4.2
ovirt-engine-setup-plugin-vmconsole-proxy-helper.noarch
4.2.2.6-1.el7.centos
@ovirt-4.2
ovirt-engine-setup-plugin-websocket-proxy.noarch
4.2.2.6-1.el7.centos
@ovirt-4.2
ovirt-imageio-proxy-setup.noarch
1.2.2-0.el7.centos
@ovirt-4.2
Available Packages
ovirt-ansible-engine-setup.noarch
1.1.0-1.el7.centos
ovirt-4.2
ovirt-engine-extension-aaa-ldap-setup.noarch
1.3.7-1.el7.centos
ovirt-4.2
ovirt-engine-setup-plugin-dockerc.noarch
4.1.9.1-1.el7.centos
ovirt-4.1
ovirt-engine-setup-plugin-live.noarch
4.1.0-1.el7.centos
ovirt-4.1
ovirt-hosted-engine-setup.noarch
-----Ursprüngliche Nachricht-----
Von: Yedidyah Bar David [mailto:didi@redhat.com]
Gesendet: Freitag, 20. April 2018 08:55
An: Sven Achtelik; Simone Tiraboschi
Cc: users(a)ovirt.org
Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before
upgrading to 4.2
On Fri, Apr 13, 2018 at 12:00 PM, Sven Achtelik <Sven.Achtelik(a)eps.aero>
wrote:
> Hi All,
>
> I got my stuff up and running again. I works like described in the manual and I
used some extra hardware to jumpstart this. I'm now back on my hosted Engine
4.1.9 with 3 Hosts running it. The Engine is running on the appliance that is
pulled by the deployment tool and after having everything stable again I
thought of upgrading to 4.2. Thing is that this is just not working with the
appliance because of some issue when upgrading Postgres inside. Looking at
the logs I found this:
> ----------------------------
> Creating dump of database schemas
> engine
> ovirt_engine_history
> postgres
> template1
> ok
>
> encodings for database "postgres" do not match: old
"SQL_ASCII", new
"UTF8"
> Failure, exiting
> ------------------------------
>
> After some research I found something here
https://bugzilla.redhat.com/show_bug.cgi?id=1525976, but I'm not sure what
to do with that Information. I used the appliance and didn't do anything manual
in the complete process and I'm wondering why I'm getting this issue now ?
Could someone advice on how to proceed ?
Looks like:
https://bugzilla.redhat.com/1528371
Which version do you upgrade to?
If to one that should be covered by above bug, please attach your setup log to it.
Thanks.
I was on vacation last week and will be in next one too. Adding Simone.
Best regards,
>
> Thank you,
> Sven
>
>> -----Ursprüngliche Nachricht-----
>> Von: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] Im
>> Auftrag von Sven Achtelik
>> Gesendet: Mittwoch, 28. März 2018 18:28
>> An: Yedidyah Bar David
>> Cc: users(a)ovirt.org
>> Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup
>> before upgrading to 4.2
>>
>>
>>
>> > -----Ursprüngliche Nachricht-----
>> > Von: Yedidyah Bar David [mailto:didi@redhat.com]
>> > Gesendet: Mittwoch, 28. März 2018 10:06
>> > An: Sven Achtelik
>> > Cc: users(a)ovirt.org
>> > Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup
>> > before upgrading to 4.2
>> >
>> > On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik
>> > <Sven.Achtelik(a)eps.aero>
>> > wrote:
>> > > Hi All,
>> > >
>> > >
>> > >
>> > > I’m still facing issues with my HE engine. Here are the steps
>> > > that I took to end up in this situation:
>> > >
>> > >
>> > >
>> > > - Update Engine from 4.1.7 to 4.1.9
>> > >
>> > > o That worked as expected
>> > >
>> > > - Automatic Backup of Engine DB in the night
>> > >
>> > > - Upgraded Engine from 4.1.9 to 4.2.1
>> > >
>> > > o That worked fine
>> > >
>> > > - Noticed Issues with the HA support for HE
>> > >
>> > > o Cause was not having the latest ovirt-ha agent/broker version on
hosts
>> > >
>> > > - After updating the first host with the latest packages for
the
>> > > Agent/Broker engine was started twice
>> > >
>> > > o As a result the Engine VM Disk was corrupted and there is no
Backup
of
>> > > the Disk
>> > >
>> > > o There is also no Backup of the Engine DB with version 4.2
>> > >
>> > > - VM disk was repaired with fsck.ext4, but DB is corrupt
>> > >
>> > > o Can’t restore the Engine DB because the Backup DB from Engine V
4.1
>> > >
>> > > - Rolled back all changes on Engine VM to 4.1.9 and imported
Backup
>> > >
>> > > o Checked for HA VMs to set as disabled and started the Engine
>> > >
>> > > - Login is fine but the Engine is having trouble picking up
and
>> > > information from the Hosts
>> > >
>> > > o No information on running VMs or hosts status
>> > >
>> > > - Final Situation
>> > >
>> > > o 2 Hosts have VMs still running and I can’t stop those
>> > >
>> > > o I still have the image of my corrupted Engine VM (v4.2)
>> > >
>> > >
>> > >
>> > > Since there were no major changes after upgrading from 4.1 to
>> > > 4.2, would it be possible to manually restore the 4.1 DB to the
>> > > 4.2 Engine VM to this up and running again or are there
>> > > modifications made to the DB on upgrading that are relevant for this
?
>> >
>> > engine-backup requires restoring to the same version used to take
>> > the backup, with a single exception - on 4.0, it can restore 3.6.
>> >
>> > It's very easy to patch it to allow also 4.1->4.2, search inside
it
>> > for "VALID_BACKUP_RESTORE_PAIRS". However, I do not think
anyone
>> > ever tested this, so no idea might break. In 3.6->4.0 days, we did
>> > have to fix a few other things, notably apache httpd and iptables-
>firewalld:
>> >
>> > https://bugzilla.redhat.com/show_bug.cgi?id=1318580
>> >
>> > > All my work on rolling back to 4.1.9 with the DB restore failed
>> > > as the Engine is not capable of picking up information from the
hosts.
>> >
>> > No idea why, but not sure it's related to your restore flow.
>> >
>> > > Lessons learned is to always make a copy/snapshot of the engine
>> > > VM disk before upgrading anything.
>> >
>> > If it's a hosted-engine, this isn't supported - see my reply on
the
>> > list ~ 1 hour ago...
>> >
>> > > What are my options on getting
>> > > back to a working environment ? Any help or hint is greatly
appreciated.
>> >
>> > Restore again with either methods - what you tried, or patching
>> > engine- backup and restore directly into 4.2 - and if the engine
>> > fails to talk to the hosts, try to debug/fix this.
>> >
>> > If you suspect corruption more severe that just the db, you can
>> > install a fresh engine machine from scratch and restore to it. If
>> > it's a hosted-engine, you'll need to deploy hosted-engine from
>> > scratch, check docs about hosted-engine backup/restore.
>>
>> I read through those documents and it seems that I would need an
>> extra Host/Hardware which I don't have.
>> https://ovirt.org/documentation/self-
>> hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-
>> Hosted_Environment/
>>
>> So how would I be able to get a new setup working when I would like
>> to use the Engine-VM-Image ? At this point it sounds like I would
>> have to manually reinstall the machine that is left over and running. I'm
lost
at this point.
>> >
>> > Best regards,
>> > --
>> > Didi
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
--
Didi
4:25 a.m.
Hi All,
I'm still failing on this one. I tried setting the locale different, but that
doesn't seem to do the trick. When looking at the logfile
/var/lib/pgsql/initdb_rh-postgresql95-postgresql.log I can see that the setup process is
somehow getting the information and setting the Encoding to UTF8.
----------------------------------------
/var/lib/pgsql/initdb_rh-postgresql95-postgresql.log
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
----------------------------------------
This leads into the issue that is showing up in
/var/lib/pgsql/upgrade_rh-postgresql95-postgresql.log
----------------------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for reg* system OID user data types ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for invalid "line" user columns ok
Creating dump of global objects ok
Creating dump of database schemas
engine
ovirt_engine_history
postgres
template1
ok
encodings for database "postgres" do not match: old "SQL_ASCII", new
"UTF8"
Failure, exiting
---------------------------------------
Even changing the locale.conf to en_US without the UTF8 doesn't change anything. Is
this information cached somewhere and needs to be reread before this will work ?
Any advice on moving forward is appreciated.
Thank you,
Sven
-----Ursprüngliche Nachricht-----
Von: Yedidyah Bar David [mailto:didi@redhat.com]
Gesendet: Freitag, 20. April 2018 08:55
An: Sven Achtelik <Sven.Achtelik(a)eps.aero>; Simone Tiraboschi
<stirabos(a)redhat.com>
Cc: users(a)ovirt.org
Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before
upgrading to 4.2
On Fri, Apr 13, 2018 at 12:00 PM, Sven Achtelik <Sven.Achtelik(a)eps.aero>
wrote:
> Hi All,
>
> I got my stuff up and running again. I works like described in the manual and I
used some extra hardware to jumpstart this. I'm now back on my hosted Engine
4.1.9 with 3 Hosts running it. The Engine is running on the appliance that is
pulled by the deployment tool and after having everything stable again I
thought of upgrading to 4.2. Thing is that this is just not working with the
appliance because of some issue when upgrading Postgres inside. Looking at
the logs I found this:
> ----------------------------
> Creating dump of database schemas
> engine
> ovirt_engine_history
> postgres
> template1
> ok
>
> encodings for database "postgres" do not match: old
"SQL_ASCII", new
"UTF8"
> Failure, exiting
> ------------------------------
>
> After some research I found something here
https://bugzilla.redhat.com/show_bug.cgi?id=1525976, but I'm not sure what
to do with that Information. I used the appliance and didn't do anything manual
in the complete process and I'm wondering why I'm getting this issue now ?
Could someone advice on how to proceed ?
Looks like:
https://bugzilla.redhat.com/1528371
Which version do you upgrade to?
If to one that should be covered by above bug, please attach your setup log to it.
Thanks.
I was on vacation last week and will be in next one too. Adding Simone.
Best regards,
>
> Thank you,
> Sven
>
>> -----Ursprüngliche Nachricht-----
>> Von: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] Im
>> Auftrag von Sven Achtelik
>> Gesendet: Mittwoch, 28. März 2018 18:28
>> An: Yedidyah Bar David
>> Cc: users(a)ovirt.org
>> Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup
>> before upgrading to 4.2
>>
>>
>>
>> > -----Ursprüngliche Nachricht-----
>> > Von: Yedidyah Bar David [mailto:didi@redhat.com]
>> > Gesendet: Mittwoch, 28. März 2018 10:06
>> > An: Sven Achtelik
>> > Cc: users(a)ovirt.org
>> > Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup
>> > before upgrading to 4.2
>> >
>> > On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik
>> > <Sven.Achtelik(a)eps.aero>
>> > wrote:
>> > > Hi All,
>> > >
>> > >
>> > >
>> > > I’m still facing issues with my HE engine. Here are the steps
>> > > that I took to end up in this situation:
>> > >
>> > >
>> > >
>> > > - Update Engine from 4.1.7 to 4.1.9
>> > >
>> > > o That worked as expected
>> > >
>> > > - Automatic Backup of Engine DB in the night
>> > >
>> > > - Upgraded Engine from 4.1.9 to 4.2.1
>> > >
>> > > o That worked fine
>> > >
>> > > - Noticed Issues with the HA support for HE
>> > >
>> > > o Cause was not having the latest ovirt-ha agent/broker version on
hosts
>> > >
>> > > - After updating the first host with the latest packages for
the
>> > > Agent/Broker engine was started twice
>> > >
>> > > o As a result the Engine VM Disk was corrupted and there is no
Backup
of
>> > > the Disk
>> > >
>> > > o There is also no Backup of the Engine DB with version 4.2
>> > >
>> > > - VM disk was repaired with fsck.ext4, but DB is corrupt
>> > >
>> > > o Can’t restore the Engine DB because the Backup DB from Engine V
4.1
>> > >
>> > > - Rolled back all changes on Engine VM to 4.1.9 and imported
Backup
>> > >
>> > > o Checked for HA VMs to set as disabled and started the Engine
>> > >
>> > > - Login is fine but the Engine is having trouble picking up
and
>> > > information from the Hosts
>> > >
>> > > o No information on running VMs or hosts status
>> > >
>> > > - Final Situation
>> > >
>> > > o 2 Hosts have VMs still running and I can’t stop those
>> > >
>> > > o I still have the image of my corrupted Engine VM (v4.2)
>> > >
>> > >
>> > >
>> > > Since there were no major changes after upgrading from 4.1 to
>> > > 4.2, would it be possible to manually restore the 4.1 DB to the
>> > > 4.2 Engine VM to this up and running again or are there
>> > > modifications made to the DB on upgrading that are relevant for this
?
>> >
>> > engine-backup requires restoring to the same version used to take
>> > the backup, with a single exception - on 4.0, it can restore 3.6.
>> >
>> > It's very easy to patch it to allow also 4.1->4.2, search inside
it
>> > for "VALID_BACKUP_RESTORE_PAIRS". However, I do not think
anyone
>> > ever tested this, so no idea might break. In 3.6->4.0 days, we did
>> > have to fix a few other things, notably apache httpd and iptables-
>firewalld:
>> >
>> > https://bugzilla.redhat.com/show_bug.cgi?id=1318580
>> >
>> > > All my work on rolling back to 4.1.9 with the DB restore failed
>> > > as the Engine is not capable of picking up information from the
hosts.
>> >
>> > No idea why, but not sure it's related to your restore flow.
>> >
>> > > Lessons learned is to always make a copy/snapshot of the engine
>> > > VM disk before upgrading anything.
>> >
>> > If it's a hosted-engine, this isn't supported - see my reply on
the
>> > list ~ 1 hour ago...
>> >
>> > > What are my options on getting
>> > > back to a working environment ? Any help or hint is greatly
appreciated.
>> >
>> > Restore again with either methods - what you tried, or patching
>> > engine- backup and restore directly into 4.2 - and if the engine
>> > fails to talk to the hosts, try to debug/fix this.
>> >
>> > If you suspect corruption more severe that just the db, you can
>> > install a fresh engine machine from scratch and restore to it. If
>> > it's a hosted-engine, you'll need to deploy hosted-engine from
>> > scratch, check docs about hosted-engine backup/restore.
>>
>> I read through those documents and it seems that I would need an
>> extra Host/Hardware which I don't have.
>> https://ovirt.org/documentation/self-
>> hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-
>> Hosted_Environment/
>>
>> So how would I be able to get a new setup working when I would like
>> to use the Engine-VM-Image ? At this point it sounds like I would
>> have to manually reinstall the machine that is left over and running. I'm
lost
at this point.
>> >
>> > Best regards,
>> > --
>> > Didi
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
--
Didi
2177
days inactive
2215
days old
8 comments
3 participants
participants (3)
-
Shani Leviim -
Sven Achtelik -
Yedidyah Bar David