Hi, Long time read-only user here :) Unfortunately I need some info which I can't seem to find in the archives and on Google. In the past 5 months we’ve had two very rare occasions of power failures in my hometown. First one was a ms during spike or drop, just enough to cause a reboot, the other was a 30min failure. Problem After the power failure my ovirt nodes boot into CentOS 6.6 and are running fine. In both cases my ovirt-engine-VM wouldn't "start" due to service Postgresql not starting. I tried some things before reverting back to an older DB snapshot that was on the VM disk, but still not working. Postgresql is not starting due to "Duplicate UUID" My solution Calm down girlfriend and don't sleep Reinstall 1 node, create new ovirt-engine, recreate VM's and copy VM disk data from old store to new store. Add second node Other solutions I can think of 1. Start cluster without reinstall ;) 2. Buy UPS (€150 one time, extra €60 yearly power usage = €450 in 5 years) At this moment I’m 1 click away from buying the UPS, but I prefer a more elegant solution. After the last power failure I did NOT reinstall my second ovirt node and the old engine is also on storage available. Is there someone that wants some data to troubleshoot/analyse? Or is this just one of those things? Buy the UPS and get on with your life? Short HW overview 2x ovirt 3.5.x (i3/32GB ram,1Gb eth VM network, dual x520 NFS eth) 1x NFS (i3/4GB ram, 5TB SAS and 6TB SATA on Dell PERC,1Gb eth mgmt, dual x520 10Gb) 1Gb is used for VM networking, the 10Gb is connected via DAC cables and runs NFS-storage. This works great! Whole cluster runs below 120w and VM disk performance is around 700MB/s :) I bought all the HW with power usage in mind, PicoPSU's and 35w CPU's in all nodes. So adding 30w "extra" just for the UPS feels a bit like killing an elephant with tissues... kind regards Raymond
On 09/02/15 10:50, Raymond wrote:
Other solutions I can think of 1. Start cluster without reinstall ;) 2. Buy UPS (€150 one time, extra €60 yearly power usage = €450 in 5 years)
At this moment I’m 1 click away from buying the UPS, but I prefer a more elegant solution.
Well imho the elegant solution _is_ to get a UPS :) welcome to the list btw -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH & Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
----- Original Message -----
From: "Sven Kieske" <s.kieske@mittwald.de> To: users@ovirt.org Sent: Friday, February 13, 2015 9:23:42 AM Subject: Re: [ovirt-users] Ovirt 3.5.x Power failure recovery
On 09/02/15 10:50, Raymond wrote:
Other solutions I can think of 1. Start cluster without reinstall ;) 2. Buy UPS (€150 one time, extra €60 yearly power usage = €450 in 5 years)
At this moment I’m 1 click away from buying the UPS, but I prefer a more elegant solution.
Well imho the elegant solution _is_ to get a UPS :)
welcome to the list btw
Indeed. You really do need a UPS. Especially for any databases -- power loss during database operations is usually very bad for the database. Pre-UPS, I've experienced similar issues with MySQL that backed my mail server. Now every machine in our house (including game consoles -- PS4 gets *really* mad when he loses power) has a UPS. If your power supply is new-ish, you may need a true sine-wave UPS. I like Cyberpower ones, if for no other reason than to prevent an APC monopoly :) Best wishes, Greg
-- Mit freundlichen Grüßen / Regards
Sven Kieske
Systemadministrator Mittwald CM Service GmbH & Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Tx for the reply's! I'm thinking about a setup with NUT : http://www.networkupstools.org/ In that case APC is not a real option, but Eaton is. Glanced at the Ellipse PRO : http://tweakers.net/pricewatch/415636/eaton-ellipse-pro-650-din.html Nice shinny display on the front and fully supported by NUT in the back. Just wanted to check if I could get around the power issue without too much investment ;) And as the power in NL is quite good is one of the last things you will see here in a home cluster. I'm a home programmer and usually use Mysql, most of the time it survives stupid mistakes. kind regards Raymond ----- Original Message ----- From: "Greg Sheremeta" <gshereme@redhat.com> To: "users" <users@ovirt.org> Sent: Friday, February 13, 2015 3:28:51 PM Subject: Re: [ovirt-users] Ovirt 3.5.x Power failure recovery ----- Original Message -----
From: "Sven Kieske" <s.kieske@mittwald.de> To: users@ovirt.org Sent: Friday, February 13, 2015 9:23:42 AM Subject: Re: [ovirt-users] Ovirt 3.5.x Power failure recovery
On 09/02/15 10:50, Raymond wrote:
Other solutions I can think of 1. Start cluster without reinstall ;) 2. Buy UPS (€150 one time, extra €60 yearly power usage = €450 in 5 years)
At this moment I’m 1 click away from buying the UPS, but I prefer a more elegant solution.
Well imho the elegant solution _is_ to get a UPS :)
welcome to the list btw
Indeed. You really do need a UPS. Especially for any databases -- power loss during database operations is usually very bad for the database. Pre-UPS, I've experienced similar issues with MySQL that backed my mail server. Now every machine in our house (including game consoles -- PS4 gets *really* mad when he loses power) has a UPS. If your power supply is new-ish, you may need a true sine-wave UPS. I like Cyberpower ones, if for no other reason than to prevent an APC monopoly :) Best wishes, Greg
-- Mit freundlichen Grüßen / Regards
Sven Kieske
Systemadministrator Mittwald CM Service GmbH & Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On 02/09/2015 04:50 AM, Raymond wrote:
Hi, Long time read-only user here :) Unfortunately I need some info which I can't seem to find in the archives and on Google.
In the past 5 months we’ve had two very rare occasions of power failures in my hometown. First one was a ms during spike or drop, just enough to cause a reboot, the other was a 30min failure.
Problem After the power failure my ovirt nodes boot into CentOS 6.6 and are running fine. In both cases my ovirt-engine-VM wouldn't "start" due to service Postgresql not starting. I tried some things before reverting back to an older DB snapshot that was on the VM disk, but still not working. Postgresql is not starting due to "Duplicate UUID"
My solution Calm down girlfriend and don't sleep Reinstall 1 node, create new ovirt-engine, recreate VM's and copy VM disk data from old store to new store. Add second node
Other solutions I can think of 1. Start cluster without reinstall ;) 2. Buy UPS (€150 one time, extra €60 yearly power usage = €450 in 5 years)
At this moment I’m 1 click away from buying the UPS, but I prefer a more elegant solution.
After the last power failure I did NOT reinstall my second ovirt node and the old engine is also on storage available. Is there someone that wants some data to troubleshoot/analyse?
Or is this just one of those things? Buy the UPS and get on with your life?
Short HW overview 2x ovirt 3.5.x (i3/32GB ram,1Gb eth VM network, dual x520 NFS eth) 1x NFS (i3/4GB ram, 5TB SAS and 6TB SATA on Dell PERC,1Gb eth mgmt, dual x520 10Gb)
1Gb is used for VM networking, the 10Gb is connected via DAC cables and runs NFS-storage. This works great! Whole cluster runs below 120w and VM disk performance is around 700MB/s :) I bought all the HW with power usage in mind, PicoPSU's and 35w CPU's in all nodes. So adding 30w "extra" just for the UPS feels a bit like killing an elephant with tissues...
kind regards Raymond _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
if your DC is 3.5, you don't need to copy the vm disks, just install new engine and import the data domain http://www.ovirt.org/Features/ImportStorageDomain (i assume there is a way to fix the postgres issue itself as well)
participants (4)
-
Greg Sheremeta -
Itamar Heim -
Raymond -
Sven Kieske