On Thu, Feb 9, 2017 at 11:00 AM, Doug Ingham <dougti@gmail.com> wrote:


On 9 February 2017 at 12:03, Dan Yasny <dyasny@gmail.com> wrote:

On Thu, Feb 9, 2017 at 9:55 AM, Doug Ingham <dougti@gmail.com> wrote:
Hi Dan,

On 8 February 2017 at 18:26, Dan Yasny <dyasny@gmail.com> wrote:

But seriously, above all, I'd recommend you backup the engine (it comes with a utility) often and well. I do it via cron every hour in production, keeping a rotation of hourly and daily backups, just in case. It doesn't take much space or resources, but it's more than just best practice - that database is the summary of the entire setup.


If you don't mind, may I ask what process you use for backing up your engine? If you use HE, do you keep one server dedicated to just that VM?
I've not had that particular issue in the restore process yet, however I read that it's recommended the HE host is free of virtual load before the backup takes place. And as they need to be done frequently, I'm reading that as a dedicated host...


If you use a dedicated host, you might as well abandon self hosted. HE is nice for small setups with the HA built in for extra fun, but once you scale, it might not be able to cope and you'll need real hardware. You're running a heavy-ish java engine plus two databases after all. 

So as I said, all I do is add the engine-backup command to cron on the engine, and then my backup server comes in and pulls out the files via scp, also through cron. Nothing fancy really, but it lets me sleep at night

This particular project has 10 new maxed out servers to back it, and I don't see it outgrowing that for at least a year or so. It's hardly a full DC.
I presume the DB will become the heaviest part of the load, and I'm already planning a separate high I/O environment for dedicated HA DB hosts.

See the top section of this page:
http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment

It seems that I'll always have to keep at least one host free to be able to avoid restore problems. If not, and I were to keep hourly backups, then migrating VMs off the host every hour would just be a pain.

Yeah, that's another downside to using HE I suppose. Though maybe you don't need to be as paranoid as I am and do backups out of hours once per day, which will be much less disruptive. 

This is pretty easy to script with the SDK - just connect to the API with a python script and deactivate a host. Once it's in maintenance, run engine-backup and pull the backup out, then activate the host again. 
 


--
Doug