Recovering oVirt-Engine with a backup before upgrading to 4.2
--_000_41e0a4df7d7b4b04824f154982fe953fepsaero_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi All, I'm still facing issues with my HE engine. Here are the steps that I took t= o end up in this situation: - Update Engine from 4.1.7 to 4.1.9 o That worked as expected - Automatic Backup of Engine DB in the night - Upgraded Engine from 4.1.9 to 4.2.1 o That worked fine - Noticed Issues with the HA support for HE o Cause was not having the latest ovirt-ha agent/broker version on hosts - After updating the first host with the latest packages for the Ag= ent/Broker engine was started twice o As a result the Engine VM Disk was corrupted and there is no Backup of = the Disk o There is also no Backup of the Engine DB with version 4.2 - VM disk was repaired with fsck.ext4, but DB is corrupt o Can't restore the Engine DB because the Backup DB from Engine V 4.1 - Rolled back all changes on Engine VM to 4.1.9 and imported Backup o Checked for HA VMs to set as disabled and started the Engine - Login is fine but the Engine is having trouble picking up and inf= ormation from the Hosts o No information on running VMs or hosts status - Final Situation o 2 Hosts have VMs still running and I can't stop those o I still have the image of my corrupted Engine VM (v4.2) Since there were no major changes after upgrading from 4.1 to 4.2, would it= be possible to manually restore the 4.1 DB to the 4.2 Engine VM to this up= and running again or are there modifications made to the DB on upgrading t= hat are relevant for this ? All my work on rolling back to 4.1.9 with the D= B restore failed as the Engine is not capable of picking up information fro= m the hosts. Lessons learned is to always make a copy/snapshot of the engin= e VM disk before upgrading anything. What are my options on getting back to= a working environment ? Any help or hint is greatly appreciated. Thank you, Sven --_000_41e0a4df7d7b4b04824f154982fe953fepsaero_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
</span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0=
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:Wingdings; panose-1:5 0 0 0 0 0 0 0 0 0;} @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-fareast-language:EN-US;} a:link, span.MsoHyperlink {mso-style-priority:99; color:#0563C1; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph {mso-style-priority:34; margin-top:0cm; margin-right:0cm; margin-bottom:0cm; margin-left:36.0pt; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-fareast-language:EN-US;} span.E-MailFormatvorlage17 {mso-style-type:personal-compose; font-family:"Calibri","sans-serif"; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-family:"Calibri","sans-serif"; mso-fareast-language:EN-US;} @page WordSection1 {size:612.0pt 792.0pt; margin:70.85pt 70.85pt 2.0cm 70.85pt;} div.WordSection1 {page:WordSection1;} /* List Definitions */ @list l0 {mso-list-id:371422817; mso-list-type:hybrid; mso-list-template-ids:1904265318 1097992454 67567619 67567621 67567617 675= 67619 67567621 67567617 67567619 67567621;} @list l0:level1 {mso-level-start-at:0; mso-level-number-format:bullet; mso-level-text:-; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:"Calibri","sans-serif"; mso-fareast-font-family:Calibri;} @list l0:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:"Courier New";} @list l0:level3 {mso-level-number-format:bullet; mso-level-text:\F0A7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:Wingdings;} @list l0:level4 {mso-level-number-format:bullet; mso-level-text:\F0B7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:Symbol;} @list l0:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:"Courier New";} @list l0:level6 {mso-level-number-format:bullet; mso-level-text:\F0A7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:Wingdings;} @list l0:level7 {mso-level-number-format:bullet; mso-level-text:\F0B7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:Symbol;} @list l0:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:"Courier New";} @list l0:level9 {mso-level-number-format:bullet; mso-level-text:\F0A7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:Wingdings;} ol {margin-bottom:0cm;} ul {margin-bottom:0cm;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> </head> <body lang=3D"DE" link=3D"#0563C1" vlink=3D"#954F72"> <div class=3D"WordSection1"> <p class=3D"MsoNormal">Hi All, <o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">I’m still facing issues w= ith my HE engine. Here are the steps that I took to end up in this situatio= n: <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p> <p class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve= l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span style=3D"mso-list:= Ignore">-<span style=3D"font:7.0pt "Times New Roman""> = ; </span></span></span><![endif]><span lang=3D"EN-US">Update Engine from 4.1.= 7 to 4.1.9 <o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0= pt;mso-list:l0 level2 lfo1"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"Courie= r New""><span style=3D"mso-list:Ignore">o<span style=3D"font:7.0pt &qu= ot;Times New Roman""> </span></span></span><![endif]><span lang=3D"EN-US">That worked as expected= <o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve= l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span style=3D"mso-list:= Ignore">-<span style=3D"font:7.0pt "Times New Roman""> = ; </span></span></span><![endif]><span lang=3D"EN-US">Automatic Backup of Eng= ine DB in the night<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve= l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span style=3D"mso-list:= Ignore">-<span style=3D"font:7.0pt "Times New Roman""> = ; </span></span></span><![endif]><span lang=3D"EN-US">Upgraded Engine from 4.= 1.9 to 4.2.1<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0= pt;mso-list:l0 level2 lfo1"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"Courie= r New""><span style=3D"mso-list:Ignore">o<span style=3D"font:7.0pt &qu= ot;Times New Roman""> </span></span></span><![endif]><span lang=3D"EN-US">That worked fine<o:p></= o:p></span></p> <p class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve= l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span style=3D"mso-list:= Ignore">-<span style=3D"font:7.0pt "Times New Roman""> = ; </span></span></span><![endif]><span lang=3D"EN-US">Noticed Issues with the= HA support for HE<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0= pt;mso-list:l0 level2 lfo1"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"Courie= r New""><span style=3D"mso-list:Ignore">o<span style=3D"font:7.0pt &qu= ot;Times New Roman""> </span></span></span><![endif]><span lang=3D"EN-US">Cause was not having th= e latest ovirt-ha agent/broker version on hosts<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve= l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span style=3D"mso-list:= Ignore">-<span style=3D"font:7.0pt "Times New Roman""> = ; </span></span></span><![endif]><span lang=3D"EN-US">After updating the firs= t host with the latest packages for the Agent/Broker engine was started twi= ce <o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0= pt;mso-list:l0 level2 lfo1"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"Courie= r New""><span style=3D"mso-list:Ignore">o<span style=3D"font:7.0pt &qu= ot;Times New Roman""> </span></span></span><![endif]><span lang=3D"EN-US">As a result the Engine = VM Disk was corrupted and there is no Backup of the Disk<o:p></o:p></span><= /p> <p class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0= pt;mso-list:l0 level2 lfo1"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"Courie= r New""><span style=3D"mso-list:Ignore">o<span style=3D"font:7.0pt &qu= ot;Times New Roman""> </span></span></span><![endif]><span lang=3D"EN-US">There is also no Backup= of the Engine DB with version 4.2<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve= l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span style=3D"mso-list:= Ignore">-<span style=3D"font:7.0pt "Times New Roman""> = ; </span></span></span><![endif]><span lang=3D"EN-US">VM disk was repaired wi= th fsck.ext4, but DB is corrupt<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0= pt;mso-list:l0 level2 lfo1"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"Courie= r New""><span style=3D"mso-list:Ignore">o<span style=3D"font:7.0pt &qu= ot;Times New Roman""> </span></span></span><![endif]><span lang=3D"EN-US">Can’t restore the= Engine DB because the Backup DB from Engine V 4.1<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve= l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span style=3D"mso-list:= Ignore">-<span style=3D"font:7.0pt "Times New Roman""> = ; </span></span></span><![endif]><span lang=3D"EN-US">Rolled back all changes= on Engine VM to 4.1.9 and imported Backup <o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0= pt;mso-list:l0 level2 lfo1"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"Courie= r New""><span style=3D"mso-list:Ignore">o<span style=3D"font:7.0pt &qu= ot;Times New Roman""> </span></span></span><![endif]><span lang=3D"EN-US">Checked for HA VMs to s= et as disabled and started the Engine<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve= l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span style=3D"mso-list:= Ignore">-<span style=3D"font:7.0pt "Times New Roman""> = ; </span></span></span><![endif]><span lang=3D"EN-US">Login is fine but the E= ngine is having trouble picking up and information from the Hosts<o:p></o:p= pt;mso-list:l0 level2 lfo1"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"Courie= r New""><span style=3D"mso-list:Ignore">o<span style=3D"font:7.0pt &qu= ot;Times New Roman""> </span></span></span><![endif]><span lang=3D"EN-US">No information on runni= ng VMs or hosts status<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve= l1 lfo1"><![if !supportLists]><span lang=3D"EN-US"><span style=3D"mso-list:= Ignore">-<span style=3D"font:7.0pt "Times New Roman""> = ; </span></span></span><![endif]><span lang=3D"EN-US">Final Situation<o:p></o= :p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0= pt;mso-list:l0 level2 lfo1"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"Courie= r New""><span style=3D"mso-list:Ignore">o<span style=3D"font:7.0pt &qu= ot;Times New Roman""> </span></span></span><![endif]><span lang=3D"EN-US">2 Hosts have VMs still = running and I can’t stop those <o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:72.0pt;text-indent:-18.0= pt;mso-list:l0 level2 lfo1"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"Courie= r New""><span style=3D"mso-list:Ignore">o<span style=3D"font:7.0pt &qu= ot;Times New Roman""> </span></span></span><![endif]><span lang=3D"EN-US">I still have the image = of my corrupted Engine VM (v4.2)<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Since there were no major chang= es after upgrading from 4.1 to 4.2, would it be possible to manually restor= e the 4.1 DB to the 4.2 Engine VM to this up and running again or are there= modifications made to the DB on upgrading that are relevant for this ? All my work on rolling back to 4.1.9 with the= DB restore failed as the Engine is not capable of picking up information f= rom the hosts. Lessons learned is to always make a copy/snapshot of the eng= ine VM disk before upgrading anything. What are my options on getting back to a working environment ? Any help or= hint is greatly appreciated.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Thank you, <o:p></o:p></span></= p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Sven <o:p></o:p></span></= p> </div> </body> </html> --_000_41e0a4df7d7b4b04824f154982fe953fepsaero_--
On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik <Sven.Achtelik@eps.aero> wrote:
Hi All,
I’m still facing issues with my HE engine. Here are the steps that I took to end up in this situation:
- Update Engine from 4.1.7 to 4.1.9
o That worked as expected
- Automatic Backup of Engine DB in the night
- Upgraded Engine from 4.1.9 to 4.2.1
o That worked fine
- Noticed Issues with the HA support for HE
o Cause was not having the latest ovirt-ha agent/broker version on hosts
- After updating the first host with the latest packages for the Agent/Broker engine was started twice
o As a result the Engine VM Disk was corrupted and there is no Backup of the Disk
o There is also no Backup of the Engine DB with version 4.2
- VM disk was repaired with fsck.ext4, but DB is corrupt
o Can’t restore the Engine DB because the Backup DB from Engine V 4.1
- Rolled back all changes on Engine VM to 4.1.9 and imported Backup
o Checked for HA VMs to set as disabled and started the Engine
- Login is fine but the Engine is having trouble picking up and information from the Hosts
o No information on running VMs or hosts status
- Final Situation
o 2 Hosts have VMs still running and I can’t stop those
o I still have the image of my corrupted Engine VM (v4.2)
Since there were no major changes after upgrading from 4.1 to 4.2, would it be possible to manually restore the 4.1 DB to the 4.2 Engine VM to this up and running again or are there modifications made to the DB on upgrading that are relevant for this ?
engine-backup requires restoring to the same version used to take the backup, with a single exception - on 4.0, it can restore 3.6. It's very easy to patch it to allow also 4.1->4.2, search inside it for "VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever tested this, so no idea might break. In 3.6->4.0 days, we did have to fix a few other things, notably apache httpd and iptables->firewalld: https://bugzilla.redhat.com/show_bug.cgi?id=1318580
All my work on rolling back to 4.1.9 with the DB restore failed as the Engine is not capable of picking up information from the hosts.
No idea why, but not sure it's related to your restore flow.
Lessons learned is to always make a copy/snapshot of the engine VM disk before upgrading anything.
If it's a hosted-engine, this isn't supported - see my reply on the list ~ 1 hour ago...
What are my options on getting back to a working environment ? Any help or hint is greatly appreciated.
Restore again with either methods - what you tried, or patching engine-backup and restore directly into 4.2 - and if the engine fails to talk to the hosts, try to debug/fix this. If you suspect corruption more severe that just the db, you can install a fresh engine machine from scratch and restore to it. If it's a hosted-engine, you'll need to deploy hosted-engine from scratch, check docs about hosted-engine backup/restore. Best regards, -- Didi
-----Ursprüngliche Nachricht----- Von: Yedidyah Bar David [mailto:didi@redhat.com] Gesendet: Mittwoch, 28. März 2018 10:06 An: Sven Achtelik Cc: users@ovirt.org Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2
On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik <Sven.Achtelik@eps.aero> wrote:
Hi All,
I’m still facing issues with my HE engine. Here are the steps that I took to end up in this situation:
- Update Engine from 4.1.7 to 4.1.9
o That worked as expected
- Automatic Backup of Engine DB in the night
- Upgraded Engine from 4.1.9 to 4.2.1
o That worked fine
- Noticed Issues with the HA support for HE
o Cause was not having the latest ovirt-ha agent/broker version on hosts
- After updating the first host with the latest packages for the Agent/Broker engine was started twice
o As a result the Engine VM Disk was corrupted and there is no Backup of the Disk
o There is also no Backup of the Engine DB with version 4.2
- VM disk was repaired with fsck.ext4, but DB is corrupt
o Can’t restore the Engine DB because the Backup DB from Engine V 4.1
- Rolled back all changes on Engine VM to 4.1.9 and imported Backup
o Checked for HA VMs to set as disabled and started the Engine
- Login is fine but the Engine is having trouble picking up and information from the Hosts
o No information on running VMs or hosts status
- Final Situation
o 2 Hosts have VMs still running and I can’t stop those
o I still have the image of my corrupted Engine VM (v4.2)
Since there were no major changes after upgrading from 4.1 to 4.2, would it be possible to manually restore the 4.1 DB to the 4.2 Engine VM to this up and running again or are there modifications made to the DB on upgrading that are relevant for this ?
engine-backup requires restoring to the same version used to take the backup, with a single exception - on 4.0, it can restore 3.6.
It's very easy to patch it to allow also 4.1->4.2, search inside it for "VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever tested this, so no idea might break. In 3.6->4.0 days, we did have to fix a few other things, notably apache httpd and iptables->firewalld:
https://bugzilla.redhat.com/show_bug.cgi?id=1318580
All my work on rolling back to 4.1.9 with the DB restore failed as the Engine is not capable of picking up information from the hosts.
No idea why, but not sure it's related to your restore flow.
Lessons learned is to always make a copy/snapshot of the engine VM disk before upgrading anything.
If it's a hosted-engine, this isn't supported - see my reply on the list ~ 1 hour ago...
What are my options on getting back to a working environment ? Any help or hint is greatly appreciated.
Restore again with either methods - what you tried, or patching engine- backup and restore directly into 4.2 - and if the engine fails to talk to the hosts, try to debug/fix this.
If you suspect corruption more severe that just the db, you can install a fresh engine machine from scratch and restore to it. If it's a hosted-engine, you'll need to deploy hosted-engine from scratch, check docs about hosted-engine backup/restore.
Will the setup of hosted engine from scratch require a new storage domain for the new engine or can I use the one that is already there ? What about the VMs running on my hosts, will they be effected by that ? It might be best to start with a fresh VM.
Best regards, -- Didi
-----Ursprüngliche Nachricht----- Von: Yedidyah Bar David [mailto:didi@redhat.com] Gesendet: Mittwoch, 28. März 2018 10:06 An: Sven Achtelik Cc: users@ovirt.org Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2
On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik <Sven.Achtelik@eps.aero> wrote:
Hi All,
I’m still facing issues with my HE engine. Here are the steps that I took to end up in this situation:
- Update Engine from 4.1.7 to 4.1.9
o That worked as expected
- Automatic Backup of Engine DB in the night
- Upgraded Engine from 4.1.9 to 4.2.1
o That worked fine
- Noticed Issues with the HA support for HE
o Cause was not having the latest ovirt-ha agent/broker version on hosts
- After updating the first host with the latest packages for the Agent/Broker engine was started twice
o As a result the Engine VM Disk was corrupted and there is no Backup of the Disk
o There is also no Backup of the Engine DB with version 4.2
- VM disk was repaired with fsck.ext4, but DB is corrupt
o Can’t restore the Engine DB because the Backup DB from Engine V 4.1
- Rolled back all changes on Engine VM to 4.1.9 and imported Backup
o Checked for HA VMs to set as disabled and started the Engine
- Login is fine but the Engine is having trouble picking up and information from the Hosts
o No information on running VMs or hosts status
- Final Situation
o 2 Hosts have VMs still running and I can’t stop those
o I still have the image of my corrupted Engine VM (v4.2)
Since there were no major changes after upgrading from 4.1 to 4.2, would it be possible to manually restore the 4.1 DB to the 4.2 Engine VM to this up and running again or are there modifications made to the DB on upgrading that are relevant for this ?
engine-backup requires restoring to the same version used to take the backup, with a single exception - on 4.0, it can restore 3.6.
It's very easy to patch it to allow also 4.1->4.2, search inside it for "VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever tested this, so no idea might break. In 3.6->4.0 days, we did have to fix a few other things, notably apache httpd and iptables->firewalld:
https://bugzilla.redhat.com/show_bug.cgi?id=1318580
All my work on rolling back to 4.1.9 with the DB restore failed as the Engine is not capable of picking up information from the hosts.
No idea why, but not sure it's related to your restore flow.
Lessons learned is to always make a copy/snapshot of the engine VM disk before upgrading anything.
If it's a hosted-engine, this isn't supported - see my reply on the list ~ 1 hour ago...
What are my options on getting back to a working environment ? Any help or hint is greatly appreciated.
Restore again with either methods - what you tried, or patching engine- backup and restore directly into 4.2 - and if the engine fails to talk to the hosts, try to debug/fix this.
If you suspect corruption more severe that just the db, you can install a fresh engine machine from scratch and restore to it. If it's a hosted-engine, you'll need to deploy hosted-engine from scratch, check docs about hosted-engine backup/restore.
I read through those documents and it seems that I would need an extra Host/Hardware which I don't have. https://ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an... So how would I be able to get a new setup working when I would like to use the Engine-VM-Image ? At this point it sounds like I would have to manually reinstall the machine that is left over and running. I'm lost at this point.
Best regards, -- Didi
Hi All, I got my stuff up and running again. I works like described in the manual and I used some extra hardware to jumpstart this. I'm now back on my hosted Engine 4.1.9 with 3 Hosts running it. The Engine is running on the appliance that is pulled by the deployment tool and after having everything stable again I thought of upgrading to 4.2. Thing is that this is just not working with the appliance because of some issue when upgrading Postgres inside. Looking at the logs I found this: ---------------------------- Creating dump of database schemas engine ovirt_engine_history postgres template1 ok encodings for database "postgres" do not match: old "SQL_ASCII", new "UTF8" Failure, exiting ------------------------------ After some research I found something here https://bugzilla.redhat.com/show_bug.cgi?id=1525976, but I'm not sure what to do with that Information. I used the appliance and didn't do anything manual in the complete process and I'm wondering why I'm getting this issue now ? Could someone advice on how to proceed ? Thank you, Sven
-----Ursprüngliche Nachricht----- Von: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] Im Auftrag von Sven Achtelik Gesendet: Mittwoch, 28. März 2018 18:28 An: Yedidyah Bar David Cc: users@ovirt.org Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2
-----Ursprüngliche Nachricht----- Von: Yedidyah Bar David [mailto:didi@redhat.com] Gesendet: Mittwoch, 28. März 2018 10:06 An: Sven Achtelik Cc: users@ovirt.org Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2
On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik <Sven.Achtelik@eps.aero> wrote:
Hi All,
I’m still facing issues with my HE engine. Here are the steps that I took to end up in this situation:
- Update Engine from 4.1.7 to 4.1.9
o That worked as expected
- Automatic Backup of Engine DB in the night
- Upgraded Engine from 4.1.9 to 4.2.1
o That worked fine
- Noticed Issues with the HA support for HE
o Cause was not having the latest ovirt-ha agent/broker version on hosts
- After updating the first host with the latest packages for the Agent/Broker engine was started twice
o As a result the Engine VM Disk was corrupted and there is no Backup of the Disk
o There is also no Backup of the Engine DB with version 4.2
- VM disk was repaired with fsck.ext4, but DB is corrupt
o Can’t restore the Engine DB because the Backup DB from Engine V 4.1
- Rolled back all changes on Engine VM to 4.1.9 and imported Backup
o Checked for HA VMs to set as disabled and started the Engine
- Login is fine but the Engine is having trouble picking up and information from the Hosts
o No information on running VMs or hosts status
- Final Situation
o 2 Hosts have VMs still running and I can’t stop those
o I still have the image of my corrupted Engine VM (v4.2)
Since there were no major changes after upgrading from 4.1 to 4.2, would it be possible to manually restore the 4.1 DB to the 4.2 Engine VM to this up and running again or are there modifications made to the DB on upgrading that are relevant for this ?
engine-backup requires restoring to the same version used to take the backup, with a single exception - on 4.0, it can restore 3.6.
It's very easy to patch it to allow also 4.1->4.2, search inside it for "VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever tested this, so no idea might break. In 3.6->4.0 days, we did have to fix a few other things, notably apache httpd and iptables->firewalld:
https://bugzilla.redhat.com/show_bug.cgi?id=1318580
All my work on rolling back to 4.1.9 with the DB restore failed as the Engine is not capable of picking up information from the hosts.
No idea why, but not sure it's related to your restore flow.
Lessons learned is to always make a copy/snapshot of the engine VM disk before upgrading anything.
If it's a hosted-engine, this isn't supported - see my reply on the list ~ 1 hour ago...
What are my options on getting back to a working environment ? Any help or hint is greatly appreciated.
Restore again with either methods - what you tried, or patching engine- backup and restore directly into 4.2 - and if the engine fails to talk to the hosts, try to debug/fix this.
If you suspect corruption more severe that just the db, you can install a fresh engine machine from scratch and restore to it. If it's a hosted-engine, you'll need to deploy hosted-engine from scratch, check docs about hosted-engine backup/restore.
I read through those documents and it seems that I would need an extra Host/Hardware which I don't have. https://ovirt.org/documentation/self- hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self- Hosted_Environment/
So how would I be able to get a new setup working when I would like to use the Engine-VM-Image ? At this point it sounds like I would have to manually reinstall the machine that is left over and running. I'm lost at this point.
Best regards, -- Didi
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Sven, I've found this one: https://gist.github.com/turboladen/6790847 I think it should help :) *Regards,* *Shani Leviim* On Fri, Apr 13, 2018 at 12:00 PM, Sven Achtelik <Sven.Achtelik@eps.aero> wrote:
Hi All,
I got my stuff up and running again. I works like described in the manual and I used some extra hardware to jumpstart this. I'm now back on my hosted Engine 4.1.9 with 3 Hosts running it. The Engine is running on the appliance that is pulled by the deployment tool and after having everything stable again I thought of upgrading to 4.2. Thing is that this is just not working with the appliance because of some issue when upgrading Postgres inside. Looking at the logs I found this: ---------------------------- Creating dump of database schemas engine ovirt_engine_history postgres template1 ok
encodings for database "postgres" do not match: old "SQL_ASCII", new "UTF8" Failure, exiting ------------------------------
After some research I found something here https://bugzilla.redhat.com/ show_bug.cgi?id=1525976, but I'm not sure what to do with that Information. I used the appliance and didn't do anything manual in the complete process and I'm wondering why I'm getting this issue now ? Could someone advice on how to proceed ?
Thank you, Sven
-----Ursprüngliche Nachricht----- Von: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] Im Auftrag von Sven Achtelik Gesendet: Mittwoch, 28. März 2018 18:28 An: Yedidyah Bar David Cc: users@ovirt.org Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2
-----Ursprüngliche Nachricht----- Von: Yedidyah Bar David [mailto:didi@redhat.com] Gesendet: Mittwoch, 28. März 2018 10:06 An: Sven Achtelik Cc: users@ovirt.org Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2
Hi All,
I’m still facing issues with my HE engine. Here are the steps that I took to end up in this situation:
- Update Engine from 4.1.7 to 4.1.9
o That worked as expected
- Automatic Backup of Engine DB in the night
- Upgraded Engine from 4.1.9 to 4.2.1
o That worked fine
- Noticed Issues with the HA support for HE
o Cause was not having the latest ovirt-ha agent/broker version on hosts
- After updating the first host with the latest packages for
On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik <Sven.Achtelik@eps.aero> wrote: the
Agent/Broker engine was started twice
o As a result the Engine VM Disk was corrupted and there is no Backup of the Disk
o There is also no Backup of the Engine DB with version 4.2
- VM disk was repaired with fsck.ext4, but DB is corrupt
o Can’t restore the Engine DB because the Backup DB from Engine V 4.1
- Rolled back all changes on Engine VM to 4.1.9 and imported Backup
o Checked for HA VMs to set as disabled and started the Engine
- Login is fine but the Engine is having trouble picking up and information from the Hosts
o No information on running VMs or hosts status
- Final Situation
o 2 Hosts have VMs still running and I can’t stop those
o I still have the image of my corrupted Engine VM (v4.2)
Since there were no major changes after upgrading from 4.1 to 4.2, would it be possible to manually restore the 4.1 DB to the 4.2 Engine VM to this up and running again or are there modifications made to the DB on upgrading that are relevant for this ?
engine-backup requires restoring to the same version used to take the backup, with a single exception - on 4.0, it can restore 3.6.
It's very easy to patch it to allow also 4.1->4.2, search inside it for "VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever tested this, so no idea might break. In 3.6->4.0 days, we did have to fix a few other things, notably apache httpd and iptables->firewalld:
https://bugzilla.redhat.com/show_bug.cgi?id=1318580
All my work on rolling back to 4.1.9 with the DB restore failed as the Engine is not capable of picking up information from the hosts.
No idea why, but not sure it's related to your restore flow.
Lessons learned is to always make a copy/snapshot of the engine VM disk before upgrading anything.
If it's a hosted-engine, this isn't supported - see my reply on the list ~ 1 hour ago...
What are my options on getting back to a working environment ? Any help or hint is greatly appreciated.
Restore again with either methods - what you tried, or patching engine- backup and restore directly into 4.2 - and if the engine fails to talk to the hosts, try to debug/fix this.
If you suspect corruption more severe that just the db, you can install a fresh engine machine from scratch and restore to it. If it's a hosted-engine, you'll need to deploy hosted-engine from scratch, check docs about hosted-engine backup/restore.
I read through those documents and it seems that I would need an extra Host/Hardware which I don't have. https://ovirt.org/documentation/self- hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self- Hosted_Environment/
So how would I be able to get a new setup working when I would like to use the Engine-VM-Image ? At this point it sounds like I would have to manually reinstall the machine that is left over and running. I'm lost at this point.
Best regards, -- Didi
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Fri, Apr 13, 2018 at 12:00 PM, Sven Achtelik <Sven.Achtelik@eps.aero> wrote:
Hi All,
I got my stuff up and running again. I works like described in the manual and I used some extra hardware to jumpstart this. I'm now back on my hosted Engine 4.1.9 with 3 Hosts running it. The Engine is running on the appliance that is pulled by the deployment tool and after having everything stable again I thought of upgrading to 4.2. Thing is that this is just not working with the appliance because of some issue when upgrading Postgres inside. Looking at the logs I found this: ---------------------------- Creating dump of database schemas engine ovirt_engine_history postgres template1 ok
encodings for database "postgres" do not match: old "SQL_ASCII", new "UTF8" Failure, exiting ------------------------------
After some research I found something here https://bugzilla.redhat.com/show_bug.cgi?id=1525976, but I'm not sure what to do with that Information. I used the appliance and didn't do anything manual in the complete process and I'm wondering why I'm getting this issue now ? Could someone advice on how to proceed ?
Looks like: https://bugzilla.redhat.com/1528371 Which version do you upgrade to? If to one that should be covered by above bug, please attach your setup log to it. Thanks. I was on vacation last week and will be in next one too. Adding Simone. Best regards,
Thank you, Sven
-----Ursprüngliche Nachricht----- Von: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] Im Auftrag von Sven Achtelik Gesendet: Mittwoch, 28. März 2018 18:28 An: Yedidyah Bar David Cc: users@ovirt.org Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2
-----Ursprüngliche Nachricht----- Von: Yedidyah Bar David [mailto:didi@redhat.com] Gesendet: Mittwoch, 28. März 2018 10:06 An: Sven Achtelik Cc: users@ovirt.org Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2
On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik <Sven.Achtelik@eps.aero> wrote:
Hi All,
I’m still facing issues with my HE engine. Here are the steps that I took to end up in this situation:
- Update Engine from 4.1.7 to 4.1.9
o That worked as expected
- Automatic Backup of Engine DB in the night
- Upgraded Engine from 4.1.9 to 4.2.1
o That worked fine
- Noticed Issues with the HA support for HE
o Cause was not having the latest ovirt-ha agent/broker version on hosts
- After updating the first host with the latest packages for the Agent/Broker engine was started twice
o As a result the Engine VM Disk was corrupted and there is no Backup of the Disk
o There is also no Backup of the Engine DB with version 4.2
- VM disk was repaired with fsck.ext4, but DB is corrupt
o Can’t restore the Engine DB because the Backup DB from Engine V 4.1
- Rolled back all changes on Engine VM to 4.1.9 and imported Backup
o Checked for HA VMs to set as disabled and started the Engine
- Login is fine but the Engine is having trouble picking up and information from the Hosts
o No information on running VMs or hosts status
- Final Situation
o 2 Hosts have VMs still running and I can’t stop those
o I still have the image of my corrupted Engine VM (v4.2)
Since there were no major changes after upgrading from 4.1 to 4.2, would it be possible to manually restore the 4.1 DB to the 4.2 Engine VM to this up and running again or are there modifications made to the DB on upgrading that are relevant for this ?
engine-backup requires restoring to the same version used to take the backup, with a single exception - on 4.0, it can restore 3.6.
It's very easy to patch it to allow also 4.1->4.2, search inside it for "VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever tested this, so no idea might break. In 3.6->4.0 days, we did have to fix a few other things, notably apache httpd and iptables->firewalld:
https://bugzilla.redhat.com/show_bug.cgi?id=1318580
All my work on rolling back to 4.1.9 with the DB restore failed as the Engine is not capable of picking up information from the hosts.
No idea why, but not sure it's related to your restore flow.
Lessons learned is to always make a copy/snapshot of the engine VM disk before upgrading anything.
If it's a hosted-engine, this isn't supported - see my reply on the list ~ 1 hour ago...
What are my options on getting back to a working environment ? Any help or hint is greatly appreciated.
Restore again with either methods - what you tried, or patching engine- backup and restore directly into 4.2 - and if the engine fails to talk to the hosts, try to debug/fix this.
If you suspect corruption more severe that just the db, you can install a fresh engine machine from scratch and restore to it. If it's a hosted-engine, you'll need to deploy hosted-engine from scratch, check docs about hosted-engine backup/restore.
I read through those documents and it seems that I would need an extra Host/Hardware which I don't have. https://ovirt.org/documentation/self- hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self- Hosted_Environment/
So how would I be able to get a new setup working when I would like to use the Engine-VM-Image ? At this point it sounds like I would have to manually reinstall the machine that is left over and running. I'm lost at this point.
Best regards, -- Didi
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Didi
I added the 4.2 Repos and this what the yum update installed the following packages [root@ovirt-engine ~]# yum list "ovirt-*-setup*" Loaded plugins: fastestmirror, versionlock Determining fastest mirrors * base: centos.host-engine.com * extras: mirror.riverfrontnetworks.com * ovirt-4.1-epel: mirrors.lug.mtu.edu * ovirt-4.2: resources.ovirt.org * ovirt-4.2-epel: mirrors.lug.mtu.edu * updates: mirrors.advancedhosters.com Installed Packages ovirt-engine-dwh-setup.noarch 4.2.2.2-1.el7.centos @ovirt-4.2 ovirt-engine-setup.noarch 4.2.2.6-1.el7.centos @ovirt-4.2 ovirt-engine-setup-base.noarch 4.2.2.6-1.el7.centos @ovirt-4.2 ovirt-engine-setup-plugin-ovirt-engine.noarch 4.2.2.6-1.el7.centos @ovirt-4.2 ovirt-engine-setup-plugin-ovirt-engine-common.noarch 4.2.2.6-1.el7.centos @ovirt-4.2 ovirt-engine-setup-plugin-vmconsole-proxy-helper.noarch 4.2.2.6-1.el7.centos @ovirt-4.2 ovirt-engine-setup-plugin-websocket-proxy.noarch 4.2.2.6-1.el7.centos @ovirt-4.2 ovirt-imageio-proxy-setup.noarch 1.2.2-0.el7.centos @ovirt-4.2 Available Packages ovirt-ansible-engine-setup.noarch 1.1.0-1.el7.centos ovirt-4.2 ovirt-engine-extension-aaa-ldap-setup.noarch 1.3.7-1.el7.centos ovirt-4.2 ovirt-engine-setup-plugin-dockerc.noarch 4.1.9.1-1.el7.centos ovirt-4.1 ovirt-engine-setup-plugin-live.noarch 4.1.0-1.el7.centos ovirt-4.1 ovirt-hosted-engine-setup.noarch
-----Ursprüngliche Nachricht----- Von: Yedidyah Bar David [mailto:didi@redhat.com] Gesendet: Freitag, 20. April 2018 08:55 An: Sven Achtelik; Simone Tiraboschi Cc: users@ovirt.org Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2
Hi All,
I got my stuff up and running again. I works like described in the manual and I used some extra hardware to jumpstart this. I'm now back on my hosted Engine 4.1.9 with 3 Hosts running it. The Engine is running on the appliance that is
On Fri, Apr 13, 2018 at 12:00 PM, Sven Achtelik <Sven.Achtelik@eps.aero> wrote: pulled by the deployment tool and after having everything stable again I thought of upgrading to 4.2. Thing is that this is just not working with the appliance because of some issue when upgrading Postgres inside. Looking at the logs I found this:
---------------------------- Creating dump of database schemas engine ovirt_engine_history postgres template1 ok
encodings for database "postgres" do not match: old "SQL_ASCII", new "UTF8" Failure, exiting ------------------------------
After some research I found something here https://bugzilla.redhat.com/show_bug.cgi?id=1525976, but I'm not sure what to do with that Information. I used the appliance and didn't do anything manual in the complete process and I'm wondering why I'm getting this issue now ? Could someone advice on how to proceed ?
Looks like:
https://bugzilla.redhat.com/1528371
Which version do you upgrade to?
If to one that should be covered by above bug, please attach your setup log to it. Thanks.
I was on vacation last week and will be in next one too. Adding Simone.
Best regards,
Thank you, Sven
-----Ursprüngliche Nachricht----- Von: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] Im Auftrag von Sven Achtelik Gesendet: Mittwoch, 28. März 2018 18:28 An: Yedidyah Bar David Cc: users@ovirt.org Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2
-----Ursprüngliche Nachricht----- Von: Yedidyah Bar David [mailto:didi@redhat.com] Gesendet: Mittwoch, 28. März 2018 10:06 An: Sven Achtelik Cc: users@ovirt.org Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2
On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik <Sven.Achtelik@eps.aero> wrote:
Hi All,
I’m still facing issues with my HE engine. Here are the steps that I took to end up in this situation:
- Update Engine from 4.1.7 to 4.1.9
o That worked as expected
- Automatic Backup of Engine DB in the night
- Upgraded Engine from 4.1.9 to 4.2.1
o That worked fine
- Noticed Issues with the HA support for HE
o Cause was not having the latest ovirt-ha agent/broker version on hosts
- After updating the first host with the latest packages for the Agent/Broker engine was started twice
o As a result the Engine VM Disk was corrupted and there is no Backup
of
the Disk
o There is also no Backup of the Engine DB with version 4.2
- VM disk was repaired with fsck.ext4, but DB is corrupt
o Can’t restore the Engine DB because the Backup DB from Engine V 4.1
- Rolled back all changes on Engine VM to 4.1.9 and imported Backup
o Checked for HA VMs to set as disabled and started the Engine
- Login is fine but the Engine is having trouble picking up and information from the Hosts
o No information on running VMs or hosts status
- Final Situation
o 2 Hosts have VMs still running and I can’t stop those
o I still have the image of my corrupted Engine VM (v4.2)
Since there were no major changes after upgrading from 4.1 to 4.2, would it be possible to manually restore the 4.1 DB to the 4.2 Engine VM to this up and running again or are there modifications made to the DB on upgrading that are relevant for this ?
engine-backup requires restoring to the same version used to take the backup, with a single exception - on 4.0, it can restore 3.6.
It's very easy to patch it to allow also 4.1->4.2, search inside it for "VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever tested this, so no idea might break. In 3.6->4.0 days, we did have to fix a few other things, notably apache httpd and iptables- firewalld:
https://bugzilla.redhat.com/show_bug.cgi?id=1318580
All my work on rolling back to 4.1.9 with the DB restore failed as the Engine is not capable of picking up information from the hosts.
No idea why, but not sure it's related to your restore flow.
Lessons learned is to always make a copy/snapshot of the engine VM disk before upgrading anything.
If it's a hosted-engine, this isn't supported - see my reply on the list ~ 1 hour ago...
What are my options on getting back to a working environment ? Any help or hint is greatly appreciated.
Restore again with either methods - what you tried, or patching engine- backup and restore directly into 4.2 - and if the engine fails to talk to the hosts, try to debug/fix this.
If you suspect corruption more severe that just the db, you can install a fresh engine machine from scratch and restore to it. If it's a hosted-engine, you'll need to deploy hosted-engine from scratch, check docs about hosted-engine backup/restore.
I read through those documents and it seems that I would need an extra Host/Hardware which I don't have. https://ovirt.org/documentation/self- hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self- Hosted_Environment/
So how would I be able to get a new setup working when I would like to use the Engine-VM-Image ? At this point it sounds like I would have to manually reinstall the machine that is left over and running. I'm lost at this point.
Best regards, -- Didi
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Didi
Hi All, I'm still failing on this one. I tried setting the locale different, but that doesn't seem to do the trick. When looking at the logfile /var/lib/pgsql/initdb_rh-postgresql95-postgresql.log I can see that the setup process is somehow getting the information and setting the Encoding to UTF8. ---------------------------------------- /var/lib/pgsql/initdb_rh-postgresql95-postgresql.log The files belonging to this database system will be owned by user "postgres". This user must also own the server process. The database cluster will be initialized with locale "en_US.UTF-8". The default database encoding has accordingly been set to "UTF8". The default text search configuration will be set to "english". ---------------------------------------- This leads into the issue that is showing up in /var/lib/pgsql/upgrade_rh-postgresql95-postgresql.log ---------------------------------------- Checking cluster versions ok Checking database user is the install user ok Checking database connection settings ok Checking for prepared transactions ok Checking for reg* system OID user data types ok Checking for contrib/isn with bigint-passing mismatch ok Checking for invalid "line" user columns ok Creating dump of global objects ok Creating dump of database schemas engine ovirt_engine_history postgres template1 ok encodings for database "postgres" do not match: old "SQL_ASCII", new "UTF8" Failure, exiting --------------------------------------- Even changing the locale.conf to en_US without the UTF8 doesn't change anything. Is this information cached somewhere and needs to be reread before this will work ? Any advice on moving forward is appreciated. Thank you, Sven
-----Ursprüngliche Nachricht----- Von: Yedidyah Bar David [mailto:didi@redhat.com] Gesendet: Freitag, 20. April 2018 08:55 An: Sven Achtelik <Sven.Achtelik@eps.aero>; Simone Tiraboschi <stirabos@redhat.com> Cc: users@ovirt.org Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2
Hi All,
I got my stuff up and running again. I works like described in the manual and I used some extra hardware to jumpstart this. I'm now back on my hosted Engine 4.1.9 with 3 Hosts running it. The Engine is running on the appliance that is
On Fri, Apr 13, 2018 at 12:00 PM, Sven Achtelik <Sven.Achtelik@eps.aero> wrote: pulled by the deployment tool and after having everything stable again I thought of upgrading to 4.2. Thing is that this is just not working with the appliance because of some issue when upgrading Postgres inside. Looking at the logs I found this:
---------------------------- Creating dump of database schemas engine ovirt_engine_history postgres template1 ok
encodings for database "postgres" do not match: old "SQL_ASCII", new "UTF8" Failure, exiting ------------------------------
After some research I found something here https://bugzilla.redhat.com/show_bug.cgi?id=1525976, but I'm not sure what to do with that Information. I used the appliance and didn't do anything manual in the complete process and I'm wondering why I'm getting this issue now ? Could someone advice on how to proceed ?
Looks like:
https://bugzilla.redhat.com/1528371
Which version do you upgrade to?
If to one that should be covered by above bug, please attach your setup log to it. Thanks.
I was on vacation last week and will be in next one too. Adding Simone.
Best regards,
Thank you, Sven
-----Ursprüngliche Nachricht----- Von: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] Im Auftrag von Sven Achtelik Gesendet: Mittwoch, 28. März 2018 18:28 An: Yedidyah Bar David Cc: users@ovirt.org Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2
-----Ursprüngliche Nachricht----- Von: Yedidyah Bar David [mailto:didi@redhat.com] Gesendet: Mittwoch, 28. März 2018 10:06 An: Sven Achtelik Cc: users@ovirt.org Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2
On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik <Sven.Achtelik@eps.aero> wrote:
Hi All,
I’m still facing issues with my HE engine. Here are the steps that I took to end up in this situation:
- Update Engine from 4.1.7 to 4.1.9
o That worked as expected
- Automatic Backup of Engine DB in the night
- Upgraded Engine from 4.1.9 to 4.2.1
o That worked fine
- Noticed Issues with the HA support for HE
o Cause was not having the latest ovirt-ha agent/broker version on hosts
- After updating the first host with the latest packages for the Agent/Broker engine was started twice
o As a result the Engine VM Disk was corrupted and there is no Backup
of
the Disk
o There is also no Backup of the Engine DB with version 4.2
- VM disk was repaired with fsck.ext4, but DB is corrupt
o Can’t restore the Engine DB because the Backup DB from Engine V 4.1
- Rolled back all changes on Engine VM to 4.1.9 and imported Backup
o Checked for HA VMs to set as disabled and started the Engine
- Login is fine but the Engine is having trouble picking up and information from the Hosts
o No information on running VMs or hosts status
- Final Situation
o 2 Hosts have VMs still running and I can’t stop those
o I still have the image of my corrupted Engine VM (v4.2)
Since there were no major changes after upgrading from 4.1 to 4.2, would it be possible to manually restore the 4.1 DB to the 4.2 Engine VM to this up and running again or are there modifications made to the DB on upgrading that are relevant for this ?
engine-backup requires restoring to the same version used to take the backup, with a single exception - on 4.0, it can restore 3.6.
It's very easy to patch it to allow also 4.1->4.2, search inside it for "VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever tested this, so no idea might break. In 3.6->4.0 days, we did have to fix a few other things, notably apache httpd and iptables- firewalld:
https://bugzilla.redhat.com/show_bug.cgi?id=1318580
All my work on rolling back to 4.1.9 with the DB restore failed as the Engine is not capable of picking up information from the hosts.
No idea why, but not sure it's related to your restore flow.
Lessons learned is to always make a copy/snapshot of the engine VM disk before upgrading anything.
If it's a hosted-engine, this isn't supported - see my reply on the list ~ 1 hour ago...
What are my options on getting back to a working environment ? Any help or hint is greatly appreciated.
Restore again with either methods - what you tried, or patching engine- backup and restore directly into 4.2 - and if the engine fails to talk to the hosts, try to debug/fix this.
If you suspect corruption more severe that just the db, you can install a fresh engine machine from scratch and restore to it. If it's a hosted-engine, you'll need to deploy hosted-engine from scratch, check docs about hosted-engine backup/restore.
I read through those documents and it seems that I would need an extra Host/Hardware which I don't have. https://ovirt.org/documentation/self- hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self- Hosted_Environment/
So how would I be able to get a new setup working when I would like to use the Engine-VM-Image ? At this point it sounds like I would have to manually reinstall the machine that is left over and running. I'm lost at this point.
Best regards, -- Didi
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Didi
participants (3)
-
Shani Leviim -
Sven Achtelik -
Yedidyah Bar David