how to prepare for an engine re-install

--Sig_/9FfbJVEsL_QD6_U8ceRqc4+ Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable I'm running ovirt (engine 3.4.0-1.el6) on a dedicated engine machine with a degraded raid array. This has caused the root filesystem to go read-only, and the engine crashed. I can restart the engine, but after doing so, the web page just loads a blank page. All nodes and VMs seem to be fine, I just can't manage them via oVirt. Because I'm offsite, my recovery option is likely limited to a reinstall of the OS and ovirt. I'm hoping for some pointers on what I can do in advance to minimize the pain of this transition.=20 - Am I going to be able to recover without having to shut down VMs, exporting and importing? - what directories should I back up? - when I reinstall, should I upgrade and install 3.5.1, or first install 3.4, get it working, and then think about upgrading? - Is it possible/recommended/crazy to migrate to a hosted engine instead of a dedicated machine? So I want to prepare for the worst before I try a reboot. If I'm lucky, the reboot will at least come up with a writable filesystem, and I won't need to re-install. If it doesn't come up, I want to be as ready as possible for the reinstall. Any hints and/or tips appreciated.. (I'll also be hanging out in the irc channel if anyone wants to respond in that venue..) Robert --=20 Senior Software Engineer @ Parsons --Sig_/9FfbJVEsL_QD6_U8ceRqc4+ Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJU5KvuAAoJEMHFVuy5l8Y48rwQAKbV/omCGFyztV/QDBF+T4qX ZtI7+j263nisA38eN7XHbfdfvEux00rqYMy9n7e+IPmsvbMIdPqps9BO9iH7fFjx 7ZtkMqyhKEju6vCiIwdWi3/KWPAhfufQvPXGOqMdgro2tim6yC1cwWQS/bAVexKh RnvntPo3WAdDdE986OdQ4JofJQqUwKAde7oPEWMjhbzx4Mxb/NSq7ExhyfWj0+wh 2iWqEEQLs2KaSOSzaqbXjIoJhxVmQPxToKdUxs1hmAhB4X/lFdIRkSY/x7SMhrKV eDB+4vd55saNz5dPfNSokQ1tTFR7z4iRkOEh5knKlRJwNdrabm+oIwrtrUbR4BX+ L0sHdB12x08aM2aio1OzY5xMLhpJSD/YiAfXs8oCPtHRbYrO2HyvOjilktY4azeO bZkCard2zxUO+UVWOEllcrXVpEgYRMlaCwhs1vsEI0mArpe0Ip9zEGYVN3IgHR6w sKrWi9u2JueqTiJOmuvGwE4EEsYKW4KRId6r1sS16neWAcbuBLjLMVzZkZ7pET5j hoERFPonAURGz1wXUuW8xK/hpZJ4xW71QpqKQfs86rwP1rIMnw3KaFqtf0AFUEPs LTKT44ibC6DNMNj3g3Kdj0iUW6/IM5xRL9QZ4exKipEyJxPf2Er6SZoYC2HffKfo gqTOxAdaa3bCVIn19Zro =ife/ -----END PGP SIGNATURE----- --Sig_/9FfbJVEsL_QD6_U8ceRqc4+--

----- Original Message -----
From: "Robert Story" <rstory@tislabs.com> To: Users@ovirt.org Sent: Wednesday, February 18, 2015 5:12:40 PM Subject: [ovirt-users] how to prepare for an engine re-install
I'm running ovirt (engine 3.4.0-1.el6) on a dedicated engine machine with a degraded raid array. This has caused the root filesystem to go read-only, and the engine crashed. I can restart the engine, but after doing so, the web page just loads a blank page. All nodes and VMs seem to be fine, I just can't manage them via oVirt.
Because I'm offsite, my recovery option is likely limited to a reinstall of the OS and ovirt. I'm hoping for some pointers on what I can do in advance to minimize the pain of this transition.
- Am I going to be able to recover without having to shut down VMs, exporting and importing?
Perhaps, depends on your plan
- what directories should I back up?
If at all possible, you should try engine-backup. I guess it will fail if the fs is read-only, because postgres will not start and engine-backup uses it. You might get by with copying the stuff engine-backup does (search for BACKUP_PATHS in its start), and also a cold copy of the database - all of /var/lib/pgsql/data . Try copying the db to another machine and 'service postgresql start'. If it works, and you can connect to the db, full recovery will probably work.
- when I reinstall, should I upgrade and install 3.5.1, or first install 3.4, get it working, and then think about upgrading?
I'd separate them. Reinstall with exact same version if possible - not sure you can find 3.4.0 on the yum repo, perhaps you can find somewhere in your cache the version you use, or something like that - then upgrade when all is quiet.
- Is it possible/recommended/crazy to migrate to a hosted engine instead of a dedicated machine?
In principle should be possible.
So I want to prepare for the worst before I try a reboot.
If it's currently read-only, why not simply copy all of it somewhere? Then if a reboot kills it, you still have the copy to try and recover from.
If I'm lucky, the reboot will at least come up with a writable filesystem, and I won't need to re-install. If it doesn't come up, I want to be as ready as possible for the reinstall.
Any hints and/or tips appreciated..
If at all possible, practice on a test system before trying live :-) See [1] for an overview of a migration to hosted-engine. Most of it is relevant to any migration, if engine-backup works for you. Otherwise it's even simpler - just copy files and hope that postgresql will manage to work with the db. [1] http://www.ovirt.org/Migrate_to_Hosted_Engine Good luck and best regards, -- Didi

--Sig_/qE=58Mje8MK/=jqgsf6vEin Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Wed, 18 Feb 2015 10:46:06 -0500 (EST) Yedidyah wrote: YBD> Good luck and best regards, Thanks... after a backup and tracking down a spare disk for the array, I went ahead with a reboot. Filesystem came up rw, engine started, and GUI is working. Issues still remain, however. While I can ssh to my nodes and VMs, VMs show as ?/unknown, and nodes show as non-responsive. I'll be looking into that tomorrow. Robert --=20 Senior Software Engineer @ Parsons --Sig_/qE=58Mje8MK/=jqgsf6vEin Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJU5PQdAAoJEMHFVuy5l8Y4E7QQAJYx5wWQtKBrjhlwZv9h6yGc wVGh9RulJYEOmW1eYvqd0+Ny26u5IlEObIec6GKlhTAFeiKDqPCNA9DlZCaGizI0 VXmoVYzzoG9ET4S26EhsdLgDfiQVwpQ4ZavOYNBbO+m0VKLekTNpbma6N4Vz/kzp fmxn5kiEe7xYdV9RJE0kgMgYupDqtcAwDW0cWkMTmHNbhrsAxXTPX2q/NA+xcHjN F+zXj8uagTMY1flE8KzEj2+EHS1/JrGIHdeAzqQQn4tehCn3HKNCe2qUdkmcOayy yaM/a/NNI9hRKPCR024hHTMkBCICXNLnwO23X3K9ZPp94cgra/XHBSMrxvGLmUTc o3a8XjHnZwvaoEwOHBr2siGcfGJQ96nqZN8hL1g7NFTO5x7ZVb3+Y/tf+jYOF5yH q9/Kx2h2tMT8pQjqGIoaa4AASmDIElxB84uFff9NjzyrMIQyFYKF/5QoQEC7JHkx +AIFqyEDvwh9h6SQrPjzSCn/hS1aXtYTVpZVTrUjAnpg5RykLm0uTIoQeaHTpdtC keH20K5gzzP1ieSx9LDieDjsAwswHwUXo1tQe1A1OWFvIN1yX15s5Ou/1eWmC5Ac NadyMjmRkyc7MPJgV7hZX8t4grKuz3+t9IbSM0by9TCKJyekuYEeiCnFVdXxugv0 8RFxV9INIzwtf+qQb7VH =zpXw -----END PGP SIGNATURE----- --Sig_/qE=58Mje8MK/=jqgsf6vEin--
participants (2)
-
Robert Story
-
Yedidyah Bar David