[ovirt-users] engine tries to allocate 11GB and aborts (was: hosted engine does not start correctly (CentOS 7.1, overt-engine 3.5.2))

[Yedidyah Bar David]

----- Original Message -----
> From: "H. Haven Liu" <haven.liu at ucla.edu>
> To: "Yedidyah Bar David" <didi at redhat.com>
> Cc: "oVirt Mailing List" <users at ovirt.org>
> Sent: Tuesday, May 26, 2015 10:26:07 PM
> Subject: Re: engine tries to allocate 11GB and aborts (was: hosted engine does not start correctly (CentOS 7.1,
> overt-engine 3.5.2))
> 
> Didi,
> 
> Both source and destination machines (assuming you are referring to the
> physical host source and virtual host destination of the migration) are
> running ovirt-engine 3.5.2.1-1. However, the "source" is on CentOS 6.6,
> whereas the "destination" is on CentOS 7.1. The source is running
> postgresql(/-server) 8.4.20, Java 1.7.0_79, httpd 2.2.15. Destination is
> running postgresql(/-server) 9.2.10, Java 1.7.0.79, jboss-httpd 2.4.6. I'm
> not sure which over package version is relevant.

Sorry for not replying earlier.

Now opened [1] to track such upgrades. Didn't try that myself.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1234257

> 
> The failure occurs after engine-setup, and the web UI doesn't load, and the
> host reports the engine is not up (I'm not in front of the host, so I don't
> have the exact language).
> 
> Our current setup has 2 physical hosts, ~15 active VMs. Each host has 64GB
> RAM. Even with all the VM on one host, the utilization is less than 50%. We
> have another physical storage server, that provides the NFS storage backend
> to the hosts and VMs. Prior to attempting the hosted-engine migration, I
> migrated all the VMs to host #1, where the engine was originally installed.
> I then removed host #2 from the host list via the UI, and performed
> hosted-engine --deploy on host #2. A few other quirks I encountered:
> 
> 1. I originally wanted to do "hosted-engine --deploy" on host #1, but that
> it was unsuccessful due to it complaining the storage connection was busy
> (or something).
> 2. I then tried to do "hosted-engine --deploy" on host #2, using the same
> storage connection path as what I used for host #1 originally. The setup
> process asked if it was an "additional host", I said no, but later it
> cannot activate the storage connection.
> 3. Lastly I did "hosted-engine --deploy" on host #2, but pointing to a
> different storage connection path, still on the same storage server. This
> was successful until the I encountered the failure in question.

Not sure about the exact causes for these issues.

If still not resolved, please retry on a clean OS reinstalled.

Best,

> 
> Thanks,
> 
> Haven
> 
> 
> 
> On Tue, May 26, 2015 at 2:20 AM, Yedidyah Bar David <didi at redhat.com> wrote:
> 
> > ----- Original Message -----
> > > From: "Haoyang Haven Liu" <haven.liu at ucla.edu>
> > > To: users at ovirt.org
> > > Sent: Tuesday, May 26, 2015 8:34:00 AM
> > > Subject: [ovirt-users] hosted engine does not start correctly (CentOS
> > 7.1,    overt-engine 3.5.2)
> > >
> > > Hello,
> > >
> > > I tried to follow < http://www.ovirt.org/Migrate_to_Hosted_Engine > to
> > > migrate an existing ovirt environment. I managed to complete the
> > > “engine-backup —mode=restore…” step, and the following “engine-setup”
> > step
> > > appear to have completed. However, the ovirt-engine service was not
> > started
> > > successfully has errors:
> > >
> > > [root at ovirt ~]# systemctl status ovirt-engine
> > > ovirt-engine.service - oVirt Engine
> > > Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled)
> > > Active: failed (Result: exit-code) since Mon 2015-05-25 22:26:12 PDT;
> > 43s ago
> > > Main PID: 15931 (code=exited, status=1/FAILURE)
> > >
> > > May 25 22:26:11 ovirt.polysep.ucla.edu systemd[1]: Started oVirt Engine.
> > > May 25 22:26:12 ovirt.polysep.ucla.edu ovirt-engine.py[15931]:
> > 2015-05-25
> > > 22:26:12,269 ovirt-engine: ERROR run:532 Error: process termina...code 1
> > > May 25 22:26:12 ovirt.polysep.ucla.edu systemd[1]:
> > ovirt-engine.service: main
> > > process exited, code=exited, status=1/FAILURE
> > > May 25 22:26:12 ovirt.polysep.ucla.edu systemd[1]: Unit
> > ovirt-engine.service
> > > entered failed state.
> > > Hint: Some lines were ellipsized, use -l to show in full.
> > >
> > > Additionally, /var/log/ovirt-engine/console.log complains “cannot
> > allocate
> > > memory”. The VM engine has 4GB of memory, but ovirt-engine engine seems
> > to
> > > want ~11GB of memory, is that right?
> > >
> > > [root at ovirt ~]# cat /var/log/ovirt-engine/console.log
> > > OpenJDK 64-Bit Server VM warning: INFO:
> > os::commit_memory(0x0000000412180000,
> > > 11250696192, 0) failed; error='Cannot allocate memory' (errno=12)
> > > #
> > > # There is insufficient memory for the Java Runtime Environment to
> > continue.
> > > # Native memory allocation (malloc) failed to allocate 11250696192 bytes
> > for
> > > committing reserved memory.
> > > # An error report file with more information is saved as:
> > > # /tmp/jvm-15974/hs_error.log
> > > [root at ovirt ~]# free -m
> > > total used free shared buff/cache available
> > > Mem: 3791 123 3175 28 492 3428
> > > Swap: 2559 0 2559
> > > [root at ovirt ~]#
> > >
> > > The hs_error.log is attached.
> > >
> > > Any help is appreciated.
> >
> > Can you please compare versions of all installed packages between the
> > source
> > and destination machines?
> >
> > I assume the failure above is right after engine-setup, right? Before
> > continuing
> > the interaction on hosted-engine side?
> >
> > Did you try to restart the engine (systemctl restart ovirt-engine)? Did it
> > fail
> > the same way?
> >
> > How much memory do you have on the source machine? And how much does the
> > engine
> > actually use there?
> >
> > How large is your system (hosts, VMs etc)?
> >
> > Best,
> > --
> > Didi
> >
> 

-- 
Didi