[ovirt-users] engine tries to allocate 11GB and aborts (was: hosted engine does not start correctly (CentOS 7.1, overt-engine 3.5.2))

[H. Haven Liu]

Didi,

Both source and destination machines (assuming you are referring to the
physical host source and virtual host destination of the migration) are
running ovirt-engine 3.5.2.1-1. However, the "source" is on CentOS 6.6,
whereas the "destination" is on CentOS 7.1. The source is running
postgresql(/-server) 8.4.20, Java 1.7.0_79, httpd 2.2.15. Destination is
running postgresql(/-server) 9.2.10, Java 1.7.0.79, jboss-httpd 2.4.6. I'm
not sure which over package version is relevant.

The failure occurs after engine-setup, and the web UI doesn't load, and the
host reports the engine is not up (I'm not in front of the host, so I don't
have the exact language).

Our current setup has 2 physical hosts, ~15 active VMs. Each host has 64GB
RAM. Even with all the VM on one host, the utilization is less than 50%. We
have another physical storage server, that provides the NFS storage backend
to the hosts and VMs. Prior to attempting the hosted-engine migration, I
migrated all the VMs to host #1, where the engine was originally installed.
I then removed host #2 from the host list via the UI, and performed
hosted-engine --deploy on host #2. A few other quirks I encountered:

1. I originally wanted to do "hosted-engine --deploy" on host #1, but that
it was unsuccessful due to it complaining the storage connection was busy
(or something).
2. I then tried to do "hosted-engine --deploy" on host #2, using the same
storage connection path as what I used for host #1 originally. The setup
process asked if it was an "additional host", I said no, but later it
cannot activate the storage connection.
3. Lastly I did "hosted-engine --deploy" on host #2, but pointing to a
different storage connection path, still on the same storage server. This
was successful until the I encountered the failure in question.

Thanks,

Haven



On Tue, May 26, 2015 at 2:20 AM, Yedidyah Bar David <didi at redhat.com> wrote:

> ----- Original Message -----
> > From: "Haoyang Haven Liu" <haven.liu at ucla.edu>
> > To: users at ovirt.org
> > Sent: Tuesday, May 26, 2015 8:34:00 AM
> > Subject: [ovirt-users] hosted engine does not start correctly (CentOS
> 7.1,    overt-engine 3.5.2)
> >
> > Hello,
> >
> > I tried to follow < http://www.ovirt.org/Migrate_to_Hosted_Engine > to
> > migrate an existing ovirt environment. I managed to complete the
> > “engine-backup —mode=restore…” step, and the following “engine-setup”
> step
> > appear to have completed. However, the ovirt-engine service was not
> started
> > successfully has errors:
> >
> > [root at ovirt ~]# systemctl status ovirt-engine
> > ovirt-engine.service - oVirt Engine
> > Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled)
> > Active: failed (Result: exit-code) since Mon 2015-05-25 22:26:12 PDT;
> 43s ago
> > Main PID: 15931 (code=exited, status=1/FAILURE)
> >
> > May 25 22:26:11 ovirt.polysep.ucla.edu systemd[1]: Started oVirt Engine.
> > May 25 22:26:12 ovirt.polysep.ucla.edu ovirt-engine.py[15931]:
> 2015-05-25
> > 22:26:12,269 ovirt-engine: ERROR run:532 Error: process termina...code 1
> > May 25 22:26:12 ovirt.polysep.ucla.edu systemd[1]:
> ovirt-engine.service: main
> > process exited, code=exited, status=1/FAILURE
> > May 25 22:26:12 ovirt.polysep.ucla.edu systemd[1]: Unit
> ovirt-engine.service
> > entered failed state.
> > Hint: Some lines were ellipsized, use -l to show in full.
> >
> > Additionally, /var/log/ovirt-engine/console.log complains “cannot
> allocate
> > memory”. The VM engine has 4GB of memory, but ovirt-engine engine seems
> to
> > want ~11GB of memory, is that right?
> >
> > [root at ovirt ~]# cat /var/log/ovirt-engine/console.log
> > OpenJDK 64-Bit Server VM warning: INFO:
> os::commit_memory(0x0000000412180000,
> > 11250696192, 0) failed; error='Cannot allocate memory' (errno=12)
> > #
> > # There is insufficient memory for the Java Runtime Environment to
> continue.
> > # Native memory allocation (malloc) failed to allocate 11250696192 bytes
> for
> > committing reserved memory.
> > # An error report file with more information is saved as:
> > # /tmp/jvm-15974/hs_error.log
> > [root at ovirt ~]# free -m
> > total used free shared buff/cache available
> > Mem: 3791 123 3175 28 492 3428
> > Swap: 2559 0 2559
> > [root at ovirt ~]#
> >
> > The hs_error.log is attached.
> >
> > Any help is appreciated.
>
> Can you please compare versions of all installed packages between the
> source
> and destination machines?
>
> I assume the failure above is right after engine-setup, right? Before
> continuing
> the interaction on hosted-engine side?
>
> Did you try to restart the engine (systemctl restart ovirt-engine)? Did it
> fail
> the same way?
>
> How much memory do you have on the source machine? And how much does the
> engine
> actually use there?
>
> How large is your system (hosts, VMs etc)?
>
> Best,
> --
> Didi
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20150526/a07c1372/attachment-0001.html>