9 p.m.
Hy!
I've just finished the process in the subject, and want to share a few things with
others, and also want to ask a question.
Long story "short" (it's my IT infra@home, I've oVirt in bigger setups
at work): I had 3 machines before: two desktop, and one Dell R710 server. One of the
desktop running FreeNAS with an SSD RAID1, the other desktop and my Dell machine was my
two hypervisors. Luckily, i've just got two HP DL380 G6, which become my hypervisors,
and my Dell machine became the TrueNAS storage (redundant PSU, more core, more RAM, more
disk :) ).
When I started the procedure, I used the latest oVirt 4.3, but it was the time to upgrade
the version, and also want to migrage all my Data (with the self hosted engine too) to the
TrueNAS Dell machine (but... I've only have the two SSD on my old FreeNAS, so it has
to be moved)
After I replaced the hypervisors, and migrated all my VM data to the new storage to new
disks (iSCSI->iSCSI, live and cold storage migration) it was the time, to shut off the
FreeNAS, and start the HE redeploy.
The main steps, what I take:
- Undeployed the hosted-engine from the host with ID: 2 (the ID came from hosted-engine
--vm-status command, the machine name is: Jupiter)
- Moved all my VMs to this host, only the HE remained on the machine with ID: 1 (name:
Saturn)
- Removed the remained stuff with: "hosted-engine --clean-metadata --host-id=2
--force-clean", so the Saturn was the only machine capable of running the HE
- Set Global maintenance mode=true
- Stop the ovirt-engine in the old HE machine
- Create a backup with "engine-backup --mode=backup --file=engine.backup
--log=engine-backup.log" and copy it to an another machine (my desktop actually)
- "hosted-engine --vm-shutdown"
- Saturn shutdown, 4.4.6 installer in (it was written to DVD before, don't want to
create another) Saturn complete reinstall.
- FreeNAS shutdown, SSD moved to TrueNAS, RAID 1/zVol create, iSCSI export...
- After the base network was created, and the backup was moved back to the new Saturn, I
started the new host deploy with: "hosted-engine --deploy
--restore-from-file=engine.backup"
Catch 1 and 2: If you do this, you must know the old DC and the Cluster name! write it
down somewhere! or you need to get out the PSQL dbs from the backup, and extract from
it... (like i had to)
The process goes almost flawless, but at some point, I've got a deployment error
(code: 519) which tells me, the Saturn have missing networks, and I can connect to
https://saturn.mope.local:6900/ovirt-engine
Catch 3: this URL not worked, since it was not the HE's URL, and I cannot login
because the FQDN checking... This may need some further improvement ;) After some
thinking, finally I've made a socks proxy with SSH (1080 forwarded to Saturn) and I
was able to login to the locally running HE and make the network changes, which was
required to continue the deploy process... Also (since the old FreeNAS box was on my desk,
the HE and the two hypervisor was unable to connect to my old SSD iSCSI, so I have to
remove it... (cannot put on maintenance, but I was able to delete it from
Storage->Domains tab) After this, Saturn came up, got the "green arrow", so I
removed the lock file which the deploy give me, and the deploy continued...
After this, I selected the new SSD RAID1 on my Dell iSCSI box, and the deploy was finally
able to copy the HE to the iSCSI, so far so good :)
Finally, I've got my new 4.4.6 setup, with Self hosted HE on my new TrueNAS@Dell. (and
all my VMs running on Jupiter at this time, without any errors)
The next step was to live migrate off Jupiter to Saturn, DVD in, Jupiter remove from the
Cluster, and reinstall to 4.4.6, and readd Jupiter (this time, with HE deployed again)
After I've put back Jupiter, and made the required initial setup with the network
(VLANs pulled to the bonds, iSCSI IPs set, etc) the cluster was up and running.
The next step was to upgrade the hypervisors to the latest image with rolling update, it
was worked as before, so the time come to move the cluster and DC compatilbility level
from 4.3 -> 4.6... This forced me to reboot all my VMs, as it written when I made this
change, but this worked too. At last: I've restarted the HE with hosted-engine
--vm-shutdown / --vm-start, because it has some "!" at the BIOS version...
And this is my question actually: after the restart, the BIOS version of the HE machine
remained the old one, and still have the "!" which states: "The
chipset/firmware type doen not match the cluster chipset/firmware type"
Doen anyone know, how can the HE BIOS updated after compatibility leve increase?
Sorry, if my mail should be goes to the users list, but because of "Catch 3" I
think is's better here.
Also I want to write this down, maybe someone find usefull, and this procedure can be used
in the Docs too, if you want!
Regards:
Peter
2:24 a.m.
New subject: oVirt 4.3 -> 4.4 upgrade and Self hosted HE storage migration
Hi,
On Sun, Sep 5, 2021 at 9:19 AM <erdosi.peter(a)kifu.gov.hu> wrote:
Hy!
I've just finished the process in the subject, and want to share a few things with
others, and also want to ask a question.
Thanks :-)
Long story "short" (it's my IT infra@home, I've oVirt in bigger setups
at work): I had 3 machines before: two desktop, and one Dell R710 server. One of the
desktop running FreeNAS with an SSD RAID1, the other desktop and my Dell machine was my
two hypervisors. Luckily, i've just got two HP DL380 G6, which become my hypervisors,
and my Dell machine became the TrueNAS storage (redundant PSU, more core, more RAM, more
disk :) ).
When I started the procedure, I used the latest oVirt 4.3, but it was the time to upgrade
the version, and also want to migrage all my Data (with the self hosted engine too) to the
TrueNAS Dell machine (but... I've only have the two SSD on my old FreeNAS, so it has
to be moved)
After I replaced the hypervisors, and migrated all my VM data to the new storage to new
disks (iSCSI->iSCSI, live and cold storage migration) it was the time, to shut off the
FreeNAS, and start the HE redeploy.
The main steps, what I take:
- Undeployed the hosted-engine from the host with ID: 2 (the ID came from hosted-engine
--vm-status command, the machine name is: Jupiter)
- Moved all my VMs to this host, only the HE remained on the machine with ID: 1 (name:
Saturn)
- Removed the remained stuff with: "hosted-engine --clean-metadata --host-id=2
--force-clean", so the Saturn was the only machine capable of running the HE
- Set Global maintenance mode=true
- Stop the ovirt-engine in the old HE machine
- Create a backup with "engine-backup --mode=backup --file=engine.backup
--log=engine-backup.log" and copy it to an another machine (my desktop actually)
- "hosted-engine --vm-shutdown"
- Saturn shutdown, 4.4.6 installer in (it was written to DVD before, don't want to
create another) Saturn complete reinstall.
- FreeNAS shutdown, SSD moved to TrueNAS, RAID 1/zVol create, iSCSI export...
- After the base network was created, and the backup was moved back to the new Saturn, I
started the new host deploy with: "hosted-engine --deploy
--restore-from-file=engine.backup"
Catch 1 and 2: If you do this, you must know the old DC and the Cluster name!
Are you sure you _must_? I think it will create new DC/cluster for you
if you type new names. Obviously, this might not be what you want.
So probably what you mean is something like:
If I restore from backup, I think the tool should check the DC/cluster
names in the backup, and let me select among them, perhaps even try to
intelligently guess the one that should be default (or just pick one
at random).
I agree this makes sense, you are welcome to file an RFE bug.
Not sure it will be prioritized - patches are welcome!
write it down somewhere! or you need to get out the PSQL dbs from the
backup, and extract from it... (like i had to)
Would also be useful if you detail your commands as a workaround, if
you file such a bug.
The process goes almost flawless, but at some point, I've got a
deployment error (code: 519) which tells me, the Saturn have missing networks, and I can
connect to https://saturn.mope.local:6900/ovirt-engine
Catch 3: this URL not worked, since it was not the HE's URL, and I cannot login
because the FQDN checking...
The deploy process temporarily configures the engine to allow using
the host's FQDN to connect to the engine, and forwards port 6900 for
you.
There might have been other complications I do not understand that
prevented this from working well.
What happened when you tried?
If you think this is a real bug, and can reliably reproduce it, please
file one in bugzilla. Thanks!
This may need some further improvement ;) After some thinking,
finally I've made a socks proxy with SSH (1080 forwarded to Saturn) and I was able to
login to the locally running HE and make the network changes, which was required to
continue the deploy process... Also (since the old FreeNAS box was on my desk, the HE and
the two hypervisor was unable to connect to my old SSD iSCSI, so I have to remove it...
I understand this was somewhat annoying, but you have to realize that
there are many different backup/restore/upgrade/migration scenarios,
and it's not easy to decide what's best to do.
In your specific case, if you are confident that you'll not need to be
able to abort in the middle and revert to the old engine+storage, you
could have removed it from the engine before taking the backup.
(cannot put on maintenance, but I was able to delete it from
Storage->Domains tab) After this, Saturn came up, got the "green arrow", so I
removed the lock file which the deploy give me, and the deploy continued...
After this, I selected the new SSD RAID1 on my Dell iSCSI box, and the deploy was finally
able to copy the HE to the iSCSI, so far so good :)
Finally, I've got my new 4.4.6 setup, with Self hosted HE on my new TrueNAS@Dell.
(and all my VMs running on Jupiter at this time, without any errors)
The next step was to live migrate off Jupiter to Saturn, DVD in, Jupiter remove from the
Cluster, and reinstall to 4.4.6, and readd Jupiter (this time, with HE deployed again)
After I've put back Jupiter, and made the required initial setup with the network
(VLANs pulled to the bonds, iSCSI IPs set, etc) the cluster was up and running.
The next step was to upgrade the hypervisors to the latest image with rolling update, it
was worked as before, so the time come to move the cluster and DC compatilbility level
from 4.3 -> 4.6... This forced me to reboot all my VMs, as it written when I made this
change, but this worked too. At last: I've restarted the HE with hosted-engine
--vm-shutdown / --vm-start, because it has some "!" at the BIOS version...
And this is my question actually: after the restart, the BIOS version of the HE machine
remained the old one, and still have the "!" which states: "The
chipset/firmware type doen not match the cluster chipset/firmware type"
Doen anyone know, how can the HE BIOS updated after compatibility leve increase?
Sorry, not sure. Perhaps you can try to edit the VM from the web admin
ui. Adding Arik.
Generally speaking:
1. The VM is created fresh, unrelated to anything in your backup, IMO.
2. So this is perhaps a result of starting from 4.4.6. To prevent
this, you could have upgraded the host to latest version *before*
starting the deploy/restore. But I might be wrong.
Sorry, if my mail should be goes to the users list, but because of "Catch 3" I
think is's better here.
Generally speaking it should have been on the users list - most bugs
are reported there, not here. But that's ok :-). This list is more for
developers. E.g. If you decide to write a patch for one of the above
issues, you might want to discuss it here if needed.
Also I want to write this down, maybe someone find usefull, and this procedure can be
used in the Docs too, if you want!
I think most of what you wrote is already covered by docs, no?
You might want to also check RHV docs (right now they tend to be more
up-to-date than oVirt), and also open docs bugs in bugzilla. If you
think there is something concrete missing, please file a docs bug (and
if it's missing in oVirt and exists in RHV, that's not a reason to not
file the bug - from oVirt's POV, the bug exists!).
Thanks and best regards,
--
Didi
3:09 a.m.
New subject: oVirt 4.3 -> 4.4 upgrade and Self hosted HE storage migration
2021. 09. 05. 9:24 keltezéssel, Yedidyah Bar David írta:
Hi,
On Sun, Sep 5, 2021 at 9:19 AM <erdosi.peter(a)kifu.gov.hu> wrote:
> Hy!
>
> I've just finished the process in the subject, and want to share a few things
with others, and also want to ask a question.
Thanks :-)
> Long story "short" (it's my IT infra@home, I've oVirt in bigger
setups at work): I had 3 machines before: two desktop, and one Dell R710 server. One of
the desktop running FreeNAS with an SSD RAID1, the other desktop and my Dell machine was
my two hypervisors. Luckily, i've just got two HP DL380 G6, which become my
hypervisors, and my Dell machine became the TrueNAS storage (redundant PSU, more core,
more RAM, more disk :) ).
>
> When I started the procedure, I used the latest oVirt 4.3, but it was the time to
upgrade the version, and also want to migrage all my Data (with the self hosted engine
too) to the TrueNAS Dell machine (but... I've only have the two SSD on my old FreeNAS,
so it has to be moved)
>
> After I replaced the hypervisors, and migrated all my VM data to the new storage to
new disks (iSCSI->iSCSI, live and cold storage migration) it was the time, to shut off
the FreeNAS, and start the HE redeploy.
>
> The main steps, what I take:
> - Undeployed the hosted-engine from the host with ID: 2 (the ID came from
hosted-engine --vm-status command, the machine name is: Jupiter)
> - Moved all my VMs to this host, only the HE remained on the machine with ID: 1
(name: Saturn)
> - Removed the remained stuff with: "hosted-engine --clean-metadata --host-id=2
--force-clean", so the Saturn was the only machine capable of running the HE
> - Set Global maintenance mode=true
> - Stop the ovirt-engine in the old HE machine
> - Create a backup with "engine-backup --mode=backup --file=engine.backup
--log=engine-backup.log" and copy it to an another machine (my desktop actually)
> - "hosted-engine --vm-shutdown"
> - Saturn shutdown, 4.4.6 installer in (it was written to DVD before, don't want
to create another) Saturn complete reinstall.
> - FreeNAS shutdown, SSD moved to TrueNAS, RAID 1/zVol create, iSCSI export...
> - After the base network was created, and the backup was moved back to the new
Saturn, I started the new host deploy with: "hosted-engine --deploy
--restore-from-file=engine.backup"
>
> Catch 1 and 2: If you do this, you must know the old DC and the Cluster name!
Are you sure you _must_? I think it will create new DC/cluster for you
if you type new names. Obviously, this might not be what you want.
So probably what you mean is something like:
The deploy process wrote it to me at
the beginning. I don't have
screenshots from it, but it was clear. Since I only have one DC and one
Cluster, and I don't want the new HE to put in an another, I think i
must do it, or my restore process will halt, or I may get Two clusters
with 1-1 machine, but not that what I want.
If I restore from backup, I think the tool should check the
DC/cluster
names in the backup, and let me select among them, perhaps even try to
intelligently guess the one that should be default (or just pick one
at random).
The tool not do this, since this question was at the beginning, but the
restore file only used (at least based the ansible tasks, what I saw
running) after the local VM created, and it just copied back to it, and
started engine-restore
I agree this makes sense, you are welcome to file an RFE bug.
Not sure it will be prioritized - patches are welcome!
Maybe :)
But what happen, if you have multiple DC and/or Cluster?!
> write it down somewhere! or you need to get out the PSQL dbs from the backup, and
extract from it... (like i had to)
Would also be useful if you detail your commands as a workaround, if
you file such a bug.
I think it's not a bug, it's my fault, but I did this
(after unpacked
the backup file, and get the two .db from it)
(installing PSQL11 to a dev machine)
pg_restore engine_backup.db > engine.sql
pg_restore dwh_backup.db > dwh.sql
createdb dwh
createdb engine
createuser engine
psql engine < engine.sql
psql dwh < dwh.sql
psql engine
select name from cluster;
and
psql dwh
select datacenter_name, create_date from datacenter_configuration order
by history_id desc limit 1;
Caution! there are a few more records in datacenter_configuration table,
historically the all what you named your DC... I need to get the last one.
> The process goes almost flawless, but at some point, I've got a deployment error
(code: 519) which tells me, the Saturn have missing networks, and I can connect to
https://saturn.mope.local:6900/ovirt-engine
>
> Catch 3: this URL not worked, since it was not the HE's URL, and I cannot login
because the FQDN checking...
The deploy process temporarily configures the engine to allow using
the host's FQDN to connect to the engine, and forwards port 6900 for
you.
There might have been other complications I do not understand that
prevented this from working well.
What happened when you tried?
If you think this is a real bug, and can reliably reproduce it, please
file one in bugzilla. Thanks!
I cant reproduce it, at least not now, but it's
looks like a bug. I have
two more oVirt 4.3, which has to be upgraded in 2021 - if nothing
blocking happen at work - so I may be able to reproduce it later.
Actually I got the frontend webpage after opening the link (I was in the
same subnet with the deploy host) and it wrote me the FQDN problem (this
is not a valid FQDN)
There was a link, to connect to the valid, but when I clicked on it, it
start to open the original HE domain (sun.mope.local) without port,
which is pointed to the old HE's IP address.
I've tried to edit my hosts file, and make the sun.mope.local be
resolved to the saturn's IP, but no luck. (the port 6900 was always
removed from the link if I remember right)
Is this may be impoved in a newer version of installer?
> This may need some further improvement ;) After some thinking, finally I've made
a socks proxy with SSH (1080 forwarded to Saturn) and I was able to login to the locally
running HE and make the network changes, which was required to continue the deploy
process... Also (since the old FreeNAS box was on my desk, the HE and the two hypervisor
was unable to connect to my old SSD iSCSI, so I have to remove it...
I understand this was somewhat annoying, but you have to realize that
there are many different backup/restore/upgrade/migration scenarios,
and it's not easy to decide what's best to do.
I understand completly ;)
there is no two same setup...
In your specific case, if you are confident that you'll not need
to be
able to abort in the middle and revert to the old engine+storage, you
could have removed it from the engine before taking the backup.
How can I remove
from the engine the domain, which the engine running
from? (and only the engine... all other VMs copied to other place)
> (cannot put on maintenance, but I was able to delete it from Storage->Domains tab)
After this, Saturn came up, got the "green arrow", so I removed the lock file
which the deploy give me, and the deploy continued...
>
> After this, I selected the new SSD RAID1 on my Dell iSCSI box, and the deploy was
finally able to copy the HE to the iSCSI, so far so good :)
>
> Finally, I've got my new 4.4.6 setup, with Self hosted HE on my new TrueNAS@Dell.
(and all my VMs running on Jupiter at this time, without any errors)
>
> The next step was to live migrate off Jupiter to Saturn, DVD in, Jupiter remove from
the Cluster, and reinstall to 4.4.6, and readd Jupiter (this time, with HE deployed
again)
>
> After I've put back Jupiter, and made the required initial setup with the network
(VLANs pulled to the bonds, iSCSI IPs set, etc) the cluster was up and running.
>
> The next step was to upgrade the hypervisors to the latest image with rolling update,
it was worked as before, so the time come to move the cluster and DC compatilbility level
from 4.3 -> 4.6... This forced me to reboot all my VMs, as it written when I made this
change, but this worked too. At last: I've restarted the HE with hosted-engine
--vm-shutdown / --vm-start, because it has some "!" at the BIOS version...
>
> And this is my question actually: after the restart, the BIOS version of the HE
machine remained the old one, and still have the "!" which states: "The
chipset/firmware type doen not match the cluster chipset/firmware type"
>
> Doen anyone know, how can the HE BIOS updated after compatibility leve increase?
Sorry, not sure. Perhaps you can try to edit the VM from the web admin
ui. Adding Arik.
Thanks.
The GUI wrote me: "There was an attempt to change Hosted Engine VM
values that are locked."
Generally speaking:
1. The VM is created fresh, unrelated to anything in your backup, IMO.
2. So this is perhaps a result of starting from 4.4.6. To prevent
this, you could have upgraded the host to latest version *before*
starting the deploy/restore. But I might be wrong.
I think you wrong. This not came
because of I used 4.4.6 installer...
This issue started, when I moved up the cluster compatibility level.
If I use the latest iso, the compatibility level have to rised up from
the same value (4.3) to the same value (4.6) either... (after the new
HE deployed)
> Sorry, if my mail should be goes to the users list, but because of "Catch
3" I think is's better here.
Generally speaking it should have been on the users list - most bugs
are reported there, not here. But that's ok :-). This list is more for
developers. E.g. If you decide to write a patch for one of the above
issues, you might want to discuss it here if needed.
Okay! Should I send it there
too, and move the conversation to the user
list?
> Also I want to write this down, maybe someone find usefull, and
this procedure can be used in the Docs too, if you want!
I think most of what you wrote is already covered by docs, no?
I had to figure out
a few things by myself :)
You might want to also check RHV docs (right now they tend to be
more
up-to-date than oVirt), and also open docs bugs in bugzilla. If you
think there is something concrete missing, please file a docs bug (and
if it's missing in oVirt and exists in RHV, that's not a reason to not
file the bug - from oVirt's POV, the bug exists!).
Okay, i'll if I have
some time, read them again.
BTW, I started from this solution (at least the preparation steps had
been done by this) https://access.redhat.com/solutions/2998291
The oVirt doc may need some extensions, based on my workflow, but I'll
try another cluster with the latest iso:
https://www.ovirt.org/documentation/upgrade_guide/#SHE_Upgrading_from_4-3
(the main steps are there)
Thanks and best regards,
Regards:
Peter
--
*Erdősi Péter
* /Informatikus, IKT Fejlesztési Főosztály /
*Kormányzati Informatikai Fejlesztési Ügynökség
* cím: 1134 Budapest, Váci út 35.
tel: +36 1 450 3080 e-mail: erdosi.peter(a)kifu.gov.hu
<mailto:erdosi.peter@kifu.gov.hu>
KIFÜ - www.kifu.gov.hu <http://kifu.gov.hu/kifu/>
963
days inactive
963
days old
2 comments
3 participants
participants (3)
-
erdosi.peter＠kifu.gov.hu -
Erdősi Péter
-
Yedidyah Bar David