3:07 p.m.
Hi Guys,
My Hosted-Engine has failed & it looks like the easiest solution will be
to install a new one. Now before I try to re-add the old hosts (still
running the guest VMs) & import the storage domain into the new engine, in
case things don't go to plan, I want to make sure I'm able to bring up the
guests on the hosts manually.
The problem is vdsClient is giving me an "Unexpected exception", without
much more info as to why it's failing.
Any idea?
[root@v0 ~]# vdsClient -s 0 list table | grep georep
9d1c3fef-498e-4c20-b124-01364d4d45a8 30455 georep-proxy Down
[root@v0 ~]# vdsClient -s 0 continue 9d1c3fef-498e-4c20-b124-01364d4d45a8
Unexpected exception
/var/log/vdsm/vdsm.log
periodic/1063::WARNING::2017-02-08
17:57:52,532::periodic::276::virt.periodic.VmDispatcher::(__call__) could
not run <class 'vdsm.virt.periodic.DriveWatermarkMonitor'> on
['65c9807c-7216-40b3-927c-5fd93bbd42ba',
u'9d1c3fef-498e-4c20-b124-01364d4d45a8']
Cheers,
--
Doug
3:10 p.m.
On Wed, Feb 8, 2017 at 4:07 PM, Doug Ingham <dougti(a)gmail.com> wrote:
continue meane un-pause, not "start from a stopped state"
now having said that, if you expect the VMs not to be able to start after
you rebuild the engine and the VMs exist on the hosts, I'd collect a virsh
-r dumpxml VMNAME for each - that way you have the disks in use, and all
the VM configuration in a file, and with some minor LVM manipulation you'll
be able to start the VM via virsh
3:18 p.m.
Hi Dan,
On 8 February 2017 at 18:10, Dan Yasny <dyasny(a)gmail.com> wrote:
I search the manual for start/init/resume syntax, and "continue" was the
closest thing I found.
My main concern is that I might have to halt the VMs or VDSM services for
some reason when trying to migrate to the new engine. I just want to make
sure that no matter what happens, I can still get the VMs back online.
I'm still getting myself acquainted with virsh/vdsClient. Could you provide
any insight into what I'd have to do to restart the guests manually?
Thanks,
--
Doug
3:26 p.m.
On Wed, Feb 8, 2017 at 4:18 PM, Doug Ingham <dougti(a)gmail.com> wrote:
I don't have a 4.x vdsm handy, but on 3.6 the verb is "create". With a LOT
of params of course.
The way oVirt works is very much tied into the engine DB. When you click
"start" on a VM, the engine will query the DB, pull out the VM details (CPU
config, disks, RAM etc), pick a suitable host, enable the hosts' access to
the disks on the storage domain, generate the libvirt domxml for the VM
(the file you'd get from virsh dumpxml) and start the VM according to the
generated XML. With vdsClient and without the engine DB you'll be missing
all those details the database provides, while my way, with the XML, they
are all already in place, populated by the engine when it was still alive.
see above. But seriously, above all, I'd recommend you backup the engine
(it comes with a utility) often and well. I do it via cron every hour in
production, keeping a rotation of hourly and daily backups, just in case.
It doesn't take much space or resources, but it's more than just best
practice - that database is the summary of the entire setup.
8:55 a.m.
Hi Dan,
On 8 February 2017 at 18:26, Dan Yasny <dyasny(a)gmail.com> wrote:
If you don't mind, may I ask what process you use for backing up your
engine? If you use HE, do you keep one server dedicated to just that VM?
I've not had that particular issue in the restore process yet, however I
read that it's recommended the HE host is free of virtual load before the
backup takes place. And as they need to be done frequently, I'm reading
that as a dedicated host...
--
Doug
9:03 a.m.
On Thu, Feb 9, 2017 at 9:55 AM, Doug Ingham <dougti(a)gmail.com> wrote:
If you use a dedicated host, you might as well abandon self hosted. HE is
nice for small setups with the HA built in for extra fun, but once you
scale, it might not be able to cope and you'll need real hardware. You're
running a heavy-ish java engine plus two databases after all.
So as I said, all I do is add the engine-backup command to cron on the
engine, and then my backup server comes in and pulls out the files via scp,
also through cron. Nothing fancy really, but it lets me sleep at night
--
Doug
10 a.m.
On 9 February 2017 at 12:03, Dan Yasny <dyasny(a)gmail.com> wrote:
This particular project has 10 new maxed out servers to back it, and I
don't see it outgrowing that for at least a year or so. It's hardly a full
DC.
I presume the DB will become the heaviest part of the load, and I'm already
planning a separate high I/O environment for dedicated HA DB hosts.
See the top section of this page:
http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restor...
It seems that I'll always have to keep at least one host free to be able to
avoid restore problems. If not, and I were to keep hourly backups, then
migrating VMs off the host every hour would just be a pain.
--
Doug
12:05 p.m.
On Thu, Feb 9, 2017 at 11:00 AM, Doug Ingham <dougti(a)gmail.com> wrote:
Yeah, that's another downside to using HE I suppose. Though maybe you don't
need to be as paranoid as I am and do backups out of hours once per day,
which will be much less disruptive.
This is pretty easy to script with the SDK - just connect to the API with a
python script and deactivate a host. Once it's in maintenance, run
engine-backup and pull the backup out, then activate the host again.
1:21 p.m.
On 9 February 2017 at 15:48, Yaniv Kaul <ykaul(a)redhat.com> wrote:
Um, that wasn't my suggestion. To be honest, a fortnight's worth of daily
engine-backups & snapshots of the engine volume should suffice for us. It's
a "Hyperconverged" setup using gluster storage on the compute nodes.
--
Doug
7:20 a.m.
This is a multi-part message in MIME format.
--------------D69F31E133DB61D4D95110A9
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: quoted-printable
Le 09/02/2017 =E0 19:48, Yaniv Kaul a =E9crit :
I'd be interested to know what type of scale needs a real hardware for=20
engine, rather 100 vms or 1000 vms? it may be about the hosts number?
Restoring_an_EL-Based_Self-Hosted_Environment
_Restoring_an_EL-Based_Self-Hosted_Environment
--=20
Nathana=EBl Blanchet
Supervision r=E9seau
P=F4le Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5 =09
T=E9l. 33 (0)4 67 54 84 55
Fax 33 (0)4 67 54 84 14
blanchet(a)abes.fr
--------------D69F31E133DB61D4D95110A9
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
<html <head <meta content=3D"text/html;
charset=3Dwindows-1252"
http-equiv=3D"Content-Type"
</head <body
bgcolor=3D"#FFFFFF" text=3D"#000000"
<p><br </p <br
<div class=3D"moz-cite-prefix">Le 09/02/2017 =E0 19:48, Yaniv Kaul a
=E9crit=A0:<br </div <blockquote
cite=3D"mid:CAJgorsZxmr=3Dargj2kVkONNE4RFKP0z35r2AEANOKf-+40ZfM8w@mail.gm=
ail.com"
type=3D"cite" <div
dir=3D"ltr"><br <div
class=3D"gmail_extra"><br
<div class=3D"gmail_quote">On Thu, Feb 9, 2017 at 6:00 PM, Doug
Ingham <span dir=3D"ltr"><<a
moz-do-not-send=3D"true"
href=3D"mailto:dougti@gmail.com"
target=3D"_blank">dougti=
@gmail.com</a <blockquote
class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex" <div dir=3D"ltr"><br <div
class=3D"gmail_extra"><br
<div class=3D"gmail_quote"><span class=3D"">On 9
Februa=
ry
2017 at 12:03, Dan Yasny <span
dir=3D"ltr"><<a
moz-do-not-send=3D"true"
href=3D"mailto:dyasny@gmail.com" target=3D"_bla=
nk">dyasny(a)gmail.com</a <blockquote
class=3D"gmail_quote" style=3D"margin:0=
px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex" <div dir=3D"ltr" <div
class=3D"gmail_extra"><br
<div class=3D"gmail_quote"><span
class=3D"m_5385279511923747685gmail-">On
Thu, Feb 9, 2017 at 9:55 AM, Doug Ingham
<span dir=3D"ltr"><<a
moz-do-not-send=3D"true"
href=3D"mailto:dougti@gmail.com"
target=3D"_blank">dougti(a)gmail.com</a=
<blockquote class=3D"gmail_quote"
style=3D"margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex" <div
dir=3D"ltr">Hi Dan,<br
<div class=3D"gmail_extra"><br <div
class=3D"gmail_quote"><span>On
8 February 2017 at 18:26, Dan
Yasny <span
dir=3D"ltr"><<a
moz-do-not-send=3D"true"
href=3D"mailto:dyasny@gmail=
.com"
target=3D"_blank">dyasny@gm=
ail.com</a
<div dir=3D"ltr"><span></span=
<div
class=3D"gmail_extra"
<div class=3D"gmail_quote=
"><span
<div><br
</div
</span
<div>But seriously,
above all, I'd
recommend you backup
the engine (it comes
with a utility)
often and well. I do
it via cron every
hour in production,
keeping a rotation
of hourly and daily
backups, just in
case. It doesn't
take much space or
resources, but it's
more than just best
practice - that
database is the
summary of the
entire setup.</div <br
clear=3D"all"
</div
</div
</div
</blockquote
<div><br
</div
</span
<div>If you don't mind, may I
ask what process you use for
backing up your engine? If you
use HE, do you keep one server
dedicated to just that VM?<br </div <div>I've not
had that
particular issue in the
restore process yet, however I
read that it's recommended the
HE host is free of virtual
load before the backup takes
place. And as they need to be
done frequently, I'm reading
that as a dedicated host...<spa=
n
class=3D"m_5385279511923747685gmail-m_-816755705011915139HOEnZb"><font
color=3D"#888888"><br
</font></span></div
</div
<span
class=3D"m_5385279511923747685gma=
il-m_-816755705011915139HOEnZb"><font
color=3D"#888888"><br
</font></span></div
</div
</blockquote
<div><br
</div
</span
<div>If you use a dedicated host, you
might as well abandon self hosted. HE is
nice for small setups with the HA built
in for extra fun, but once you scale, it
might not be able to cope and you'll
need real hardware. You're running a
heavy-ish java engine plus two databases
after all. <br
</div
</div
</div </div </blockquote </span></div </div
</div </blockquote </div
</div </div </blockquote
I'd be interested to know what type of scale needs a real hardware
for engine, rather 100 vms or 1000 vms? it may be about the hosts
number?<br <blockquote
cite=3D"mid:CAJgorsZxmr=3Dargj2kVkONNE4RFKP0z35r2AEANOKf-+40ZfM8w@mail.gm=
ail.com"
type=3D"cite" <div
dir=3D"ltr" <div
class=3D"gmail_extra" <div
class=3D"gmail_quote" <blockquote
class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex" <div dir=3D"ltr" <div class=3D"gmail_extra" <div
class=3D"gmail_quote"><span class=3D"" <blockquote
class=3D"gmail_quote" style=3D"margin:0=
px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex" <div dir=3D"ltr" <div
class=3D"gmail_extra" <div
class=3D"gmail_quote"
<div><br
</div
<div>So as I said, all I do is add the
engine-backup command to cron on the
engine, and then my backup server comes
in and pulls out the files via scp, also
through cron. Nothing fancy really, but
it lets me sleep at night</div </div </div
</div
</blockquote
<div><br </div </span
<div>This particular project has 10 new maxed out
servers to back it, and I don't see it outgrowing
that for at least a year or so. It's hardly a full
DC.<br I presume the
DB will become the heaviest part of
the load, and I'm already planning a separate high
I/O environment for dedicated HA DB hosts.<br <br
</div <div>See
the top section of this page:</div
</div <a
moz-do-not-send=3D"true"
href=3D"http://www.ovirt.org/documentation/self-hosted/chap-Backing_...
d_Restoring_an_EL-Based_Self-Hosted_Environment"
target=3D"_blank">http://www.ovirt.org/<wbr>documenta=
tion/self-hosted/<wbr>chap-Backing_up_and_Restoring_<wbr>an_EL-Based_Self=
-Hosted_<wbr>Environment</a><br
<br </div <div class=3D"gmail_extra">It
seems that I'll always have
to keep at least one host free to be able to avoid
restore problems. If not, and I were to keep hourly
backups, then migrating VMs off the host every hour
would just be a pain.</div
</div </blockquote <div><br
</div <div>I don't
see the point in an hourly backup. Of what? The
DB? The VM? What storage will it be based on?</div <div>I suggest revising the
strategy.</div
<div>=A0</div <blockquote
class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex" <div dir=3D"ltr" <div
class=3D"gmail_extra"><span class=3D"HOEnZb"><font
color=3D"#888888"><br
</font></span></div
<span class=3D"HOEnZb"><font color=3D"#888888" <div
class=3D"gmail_extra"><br
-- <br <div
class=3D"m_5385279511923747685gmail_signature"=
>Doug</div </div </font></span></div <br
______________________________<wbr>_________________<br Users mailing list<br <a moz-do-not-send=3D"true"
href=3D"mailto:Users@ovirt.org"=
>Users(a)ovirt.org</a><br
<a moz-do-not-send=3D"true"
href=3D"http://lists.ovirt.org/mailman/listinfo/users"
rel=3D"noreferrer"
target=3D"_blank">http://lists.ovirt.o=
rg/<wbr>mailman/listinfo/users</a><br
<br </blockquote </div
<br </div
</div <br <fieldset
class=3D"mimeAttachmentHeader"></fieldset
<br <pre
wrap=3D"">_______________________________________________
Users mailing list
<a class=3D"moz-txt-link-abbreviated"
href=3D"mailto:Users@ovirt.org">Use=
rs(a)ovirt.org</a <a
class=3D"moz-txt-link-freetext" href=3D"http://lists.ovirt.org/mailman=
/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a </pre
</blockquote <br <pre class=3D"moz-signature"
cols=3D"72">--=20
Nathana=EBl Blanchet
Supervision r=E9seau
P=F4le Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5 =09
T=E9l. 33 (0)4 67 54 84 55
Fax 33 (0)4 67 54 84 14
<a class=3D"moz-txt-link-abbreviated"
href=3D"mailto:blanchet@abes.fr">bl=
anchet(a)abes.fr</a> </pre </body </html
--------------D69F31E133DB61D4D95110A9--
></span>
wrote:<br></span>
wrote:<br></span>
wrote:<br></span>
wrote:
<blockquote
class=3D"gmail_quote"
style=3D"margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex"
3026
days inactive
3028
days old
12 comments
4 participants
participants (4)
-
Dan Yasny -
Doug Ingham -
Nathanaël Blanchet -
Yaniv Kaul