[Users] Migrate cluster 3.3 -> 3.4 hosted on existing hosts
--------------080102010006090400050504 Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit Current setup: * 3 identical hosts running on HP GL180 g5 servers o gluster running 5 volumes in replica 3 * engine running on VMWare Server on another computer (that computer is NOT available to convert to a host) Where I want to end up: * 3 identical hosted-engine hosts running on HP GL180 g5 servers o gluster running 6 volumes in replica 3 + new volume will be nfs storage for engine VM * hosted engine in oVirt VM * as few changes to current setup as possible The two pages I found on the wiki are: Hosted Engine Howto <http://www.ovirt.org/Hosted_Engine_Howto> and Migrate to Hosted Engine <http://www.ovirt.org/Migrate_to_Hosted_Engine>. Both were written during the testing process, and have not been updated to reflect production status. I don't know if anything in the process has changed since they were written. Process outlined in above two pages (as I understand it): have nfs file store ready to hold VM Do minimal install (not clear if ovirt node, Centos, or Fedora was used--I am Centos-based) # yum install ovirt-hosted-engine-setup # hosted-engine --deploy Install OS on VM return to host console at "Please install the engine in the VM" prompt on host on VM console # yum install ovirt-engine on old engine: service ovirt-engine stop chkconfig ovirt-engine off set up dns for new engine # engine-backup --mode=backup --file=backup1 --log=backup1.log scp backup file to new engine VM on new VM: # engine-backup --mode=restore --file=backup1 --log=backup1-restore.log --change-db-credentials --db-host=didi-lap --db-user=engine --db-password --db-name=engine # engine-setup on host: run script until: "The system will wait until the VM is down." on new VM: # reboot on Host: finish script My questions: 1. Is the above still the recommended way to do a hosted-engine install? 2. Will it blow up at me if I use my existing host (with glusterfs all set up, etc) as the starting point, instead of a clean install? Thank you for letting me benefit from your experience, Ted Miller Elkhart, IN, USA --------------080102010006090400050504 Content-Type: text/html; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> </head> <body text="#000000" bgcolor="#FFFFFF"> Current setup:<br> <ul> <li>3 identical hosts running on HP GL180 g5 servers</li> <ul> <li>gluster running 5 volumes in replica 3<br> </li> </ul> <li>engine running on VMWare Server on another computer (that computer is NOT available to convert to a host)<br> </li> </ul> Where I want to end up:<br> <ul> <li>3 identical hosted-engine hosts running on HP GL180 g5 servers</li> <ul> <li>gluster running 6 volumes in replica 3</li> <ul> <li>new volume will be nfs storage for engine VM</li> </ul> </ul> <li>hosted engine in oVirt VM</li> <li>as few changes to current setup as possible</li> </ul> <p>The two pages I found on the wiki are: <a href="http://www.ovirt.org/Hosted_Engine_Howto">Hosted Engine Howto</a> and <a href="http://www.ovirt.org/Migrate_to_Hosted_Engine">Migrate to Hosted Engine</a>. Both were written during the testing process, and have not been updated to reflect production status. I don't know if anything in the process has changed since they were written.<br> </p> <p>Process outlined in above two pages (as I understand it):<br> </p> <blockquote> <p>have nfs file store ready to hold VM</p> <p>Do minimal install (not clear if ovirt node, Centos, or Fedora was used--I am Centos-based)</p> # yum install ovirt-hosted-engine-setup<br> # hosted-engine --deploy<br> <p>Install OS on VM<br> </p> <p>return to host console<br> </p> <p>at "Please install the engine in the VM" prompt on host<br> </p> <p>on VM console<br> # yum install ovirt-engine<br> </p> <p>on old engine: <br> service ovirt-engine stop<br> chkconfig ovirt-engine off</p> <p>set up dns for new engine<br> </p> <p># engine-backup --mode=backup --file=backup1 --log=backup1.log<br> scp backup file to new engine VM<br> </p> <p>on new VM:<br> # engine-backup --mode=restore --file=backup1 --log=backup1-restore.log --change-db-credentials --db-host=didi-lap --db-user=engine --db-password --db-name=engine<br> # engine-setup<br> </p> <p>on host:<br> run script until: "The system will wait until the VM is down."<br> </p> <p>on new VM:<br> # reboot<br> </p> <p>on Host: finish script<br> </p> </blockquote> My questions:<br> <br> 1. Is the above still the recommended way to do a hosted-engine install?<br> <br> 2. Will it blow up at me if I use my existing host (with glusterfs all set up, etc) as the starting point, instead of a clean install?<br> <br> Thank you for letting me benefit from your experience,<br> Ted Miller<br> Elkhart, IN, USA<br> <br> </body> </html> --------------080102010006090400050504--
----- Original Message -----
From: "Ted Miller" <tmiller@hcjb.org> To: "users" <users@ovirt.org> Sent: Tuesday, April 1, 2014 10:40:38 PM Subject: [Users] Migrate cluster 3.3 -> 3.4 hosted on existing hosts
Current setup:
* 3 identical hosts running on HP GL180 g5 servers
* gluster running 5 volumes in replica 3 * engine running on VMWare Server on another computer (that computer is NOT available to convert to a host)
Where I want to end up:
* 3 identical hosted-engine hosts running on HP GL180 g5 servers
* gluster running 6 volumes in replica 3
* new volume will be nfs storage for engine VM * hosted engine in oVirt VM * as few changes to current setup as possible
The two pages I found on the wiki are: Hosted Engine Howto and Migrate to Hosted Engine . Both were written during the testing process, and have not been updated to reflect production status. I don't know if anything in the process has changed since they were written.
Basically things remained the same, with some details changing perhaps.
Process outlined in above two pages (as I understand it):
have nfs file store ready to hold VM
Do minimal install (not clear if ovirt node, Centos, or Fedora was used--I am Centos-based)
Fedora/Centos/RHEL are supposed to work. ovirt node is currently not supported - iirc it's planned to be supported soon, not sure.
# yum install ovirt-hosted-engine-setup # hosted-engine --deploy
Install OS on VM
return to host console
at "Please install the engine in the VM" prompt on host
on VM console # yum install ovirt-engine
on old engine: service ovirt-engine stop chkconfig ovirt-engine off
set up dns for new engine
# engine-backup --mode=backup --file=backup1 --log=backup1.log scp backup file to new engine VM
on new VM:
Please see [1]. Specifically, if you had a local db, you'll first have to create it yourself. [1] http://www.ovirt.org/Ovirt-engine-backup#Howto
# engine-backup --mode=restore --file=backup1 --log=backup1-restore.log --change-db-credentials --db-host=didi-lap --db-user=engine --db-password --db-name=engine
The above assumes a db was already created and ready to use (access etc) using the supplied credentials. You'll naturally have to provide your own.
# engine-setup
on host: run script until: "The system will wait until the VM is down."
on new VM: # reboot
on Host: finish script My questions:
1. Is the above still the recommended way to do a hosted-engine install?
Yes.
2. Will it blow up at me if I use my existing host (with glusterfs all set up, etc) as the starting point, instead of a clean install?
a. Probably yes, for now. I did not hear much about testing such a migration using an existing host - ovirt or gluster or both. I did not test that myself either. If at all possible, you should use a new clean host. Do plan well and test. Also see discussions on the mailing lists, e.g. this one: http://lists.ovirt.org/pipermail/users/2014-March/thread.html#22441 Good luck, and please report back! -- Didi
On 4/2/2014 1:58 AM, Yedidyah Bar David wrote:
----- Original Message -----
From: "Ted Miller" <tmiller@hcjb.org> To: "users" <users@ovirt.org> Sent: Tuesday, April 1, 2014 10:40:38 PM Subject: [Users] Migrate cluster 3.3 -> 3.4 hosted on existing hosts
Current setup: * 3 identical hosts running on HP GL180 g5 servers * gluster running 5 volumes in replica 3 * engine running on VMWare Server on another computer (that computer is NOT available to convert to a host)
Where I want to end up: * 3 identical hosted-engine hosts running on HP GL180 g5 servers * gluster running 6 volumes in replica 3 * new volume will be nfs storage for engine VM * hosted engine in oVirt VM * as few changes to current setup as possible
The two pages I found on the wiki are: Hosted Engine Howto and Migrate to Hosted Engine . Both were written during the testing process, and have not been updated to reflect production status. I don't know if anything in the process has changed since they were written. Basically things remained the same, with some details changing perhaps.
Process outlined in above two pages (as I understand it):
have nfs file store ready to hold VM
Do minimal install (not clear if ovirt node, Centos, or Fedora was used--I am Centos-based) Fedora/Centos/RHEL are supposed to work. ovirt node is currently not supported - iirc it's planned to be supported soon, not sure.
# yum install ovirt-hosted-engine-setup # hosted-engine --deploy
Install OS on VM
return to host console
at "Please install the engine in the VM" prompt on host
on VM console # yum install ovirt-engine
on old engine: service ovirt-engine stop chkconfig ovirt-engine off
set up dns for new engine
# engine-backup --mode=backup --file=backup1 --log=backup1.log scp backup file to new engine VM
on new VM: Please see [1]. Specifically, if you had a local db, you'll first have to create it yourself.
[1] http://www.ovirt.org/Ovirt-engine-backup#Howto
# engine-backup --mode=restore --file=backup1 --log=backup1-restore.log --change-db-credentials --db-host=didi-lap --db-user=engine --db-password --db-name=engine The above assumes a db was already created and ready to use (access etc) using the supplied credentials. You'll naturally have to provide your own.
# engine-setup
on host: run script until: "The system will wait until the VM is down."
on new VM: # reboot
on Host: finish script My questions:
1. Is the above still the recommended way to do a hosted-engine install? Yes.
2. Will it blow up at me if I use my existing host (with glusterfs all set up, etc) as the starting point, instead of a clean install? a. Probably yes, for now. I did not hear much about testing such a migration using an existing host - ovirt or gluster or both. I did not test that myself either.
If at all possible, you should use a new clean host. Do plan well and test.
Also see discussions on the mailing lists, e.g. this one:
http://lists.ovirt.org/pipermail/users/2014-March/thread.html#22441
Good luck, and please report back! I have good news and bad news.
I migrated the 3 host cluster from 3.4 to 3.4 hosted. The process went fairly smoothly. Engine ran, I was able to add the three hosts to the engine's domain, etc. That was all working about Thursday. (I did not get fencing set up). Friday, at the end of the day, I shut down the entire system (it is not yet in production) because I was leaving for a week's vacation/holiday. I am fairly certain that I put the system into global maintenance mode before shutting down. I know I shut down the engine before shutting down the hosts. Monday (10 days later) I came back from vacation and powered up the three machines. The hosts came up fine, but the engine will not start. (I found some gluster split-brain errors, and chased that for a couple of days, until I realized that the split-brain was not the fundamental problem.) During bootup /var/log/messages shows: May 21 19:22:00 s2 ovirt-ha-broker mgmt_bridge.MgmtBridge ERROR Failed to getVdsCapabilities: VDSM initialization timeout May 21 19:22:00 s2 ovirt-ha-broker mem_free.MemFree ERROR Failed to getVdsStats: VDSM initialization timeout May 21 19:22:00 s2 ovirt-ha-broker cpu_load_no_engine.EngineHealth ERROR Failed to getVmStats: VDSM initialization timeout May 21 19:22:00 s2 ovirt-ha-broker engine_health.CpuLoadNoEngine ERROR Failed to getVmStats: VDSM initialization timeout May 21 19:22:03 s2 vdsm vds WARNING Unable to load the json rpc server module. Please make sure it is installed. and then /var/log/ovirt-hosted-engine-ha/agent.log shows: MainThread::ERROR::2014-05-21 19:22:04,198::hosted_engine::414::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Failed trying to connect storage: MainThread::CRITICAL::2014-05-21 19:22:04,199::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Could not start ha-agent Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 97, in run self._run_agent() File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 154, in _run_agent hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring() File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 299, in start_monitoring self._initialize_vdsm() File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 415, in _initialize_vdsm raise Exception("Failed trying to connect storage") Exception: Failed trying to connect storage I can manually mount and start storage, but then my sanlock/gluster/split-brain bites me, and there are no instructions out there on how to recover from this situation. I am starting over the "sanitary" way. The only thing getting reused is the gluster file system. I will migrate one host at a time, and will get the gluster system running under the hosts as I go, adding the host back into the system as I go along. You were right, :( Ted Miller Elkhart, IN, USA P.S. I hope that the install scripts can eventually be modified to migrate existing hosts, but I understand that first you must have it stable from a clean install.
participants (2)
-
Ted Miller -
Yedidyah Bar David