Re: [ovirt-users] engine tries to allocate 11GB and aborts (was: hosted engine does not start correctly (CentOS 7.1, overt-engine 3.5.2))

From: "H. Haven Liu" <haven.liu(a)ucla.edu&gt; To: "Yedidyah Bar David" <didi(a)redhat.com&gt; Cc: "oVirt Mailing List" <users(a)ovirt.org&gt; Sent: Tuesday, May 26, 2015 10:26:07 PM Subject: Re: engine tries to allocate 11GB and aborts (was: hosted engine does not start correctly (CentOS 7.1, overt-engine 3.5.2)) Didi, Both source and destination machines (assuming you are referring to the physical host source and virtual host destination of the migration) are running ovirt-engine 3.5.2.1-1. However, the "source" is on CentOS 6.6, whereas the "destination" is on CentOS 7.1. The source is running postgresql(/-server) 8.4.20, Java 1.7.0_79, httpd 2.2.15. Destination is running postgresql(/-server) 9.2.10, Java 1.7.0.79, jboss-httpd 2.4.6. I'm not sure which over package version is relevant.

The failure occurs after engine-setup, and the web UI doesn't load, and the host reports the engine is not up (I'm not in front of the host, so I don't have the exact language). Our current setup has 2 physical hosts, ~15 active VMs. Each host has 64GB RAM. Even with all the VM on one host, the utilization is less than 50%. We have another physical storage server, that provides the NFS storage backend to the hosts and VMs. Prior to attempting the hosted-engine migration, I migrated all the VMs to host #1, where the engine was originally installed. I then removed host #2 from the host list via the UI, and performed hosted-engine --deploy on host #2. A few other quirks I encountered: 1. I originally wanted to do "hosted-engine --deploy" on host #1, but that it was unsuccessful due to it complaining the storage connection was busy (or something). 2. I then tried to do "hosted-engine --deploy" on host #2, using the same storage connection path as what I used for host #1 originally. The setup process asked if it was an "additional host", I said no, but later it cannot activate the storage connection. 3. Lastly I did "hosted-engine --deploy" on host #2, but pointing to a different storage connection path, still on the same storage server. This was successful until the I encountered the failure in question.

Thanks, Haven On Tue, May 26, 2015 at 2:20 AM, Yedidyah Bar David <didi(a)redhat.com&gt; wrote: > ----- Original Message ----- > > From: "Haoyang Haven Liu" <haven.liu(a)ucla.edu&gt; > > To: users(a)ovirt.org > > Sent: Tuesday, May 26, 2015 8:34:00 AM > > Subject: [ovirt-users] hosted engine does not start correctly (CentOS > 7.1, overt-engine 3.5.2) > > > > Hello, > > > > I tried to follow < http://www.ovirt.org/Migrate_to_Hosted_Engine > to > > migrate an existing ovirt environment. I managed to complete the > > “engine-backup —mode=restore…” step, and the following “engine-setup” > step > > appear to have completed. However, the ovirt-engine service was not > started > > successfully has errors: > > > > [root@ovirt ~]# systemctl status ovirt-engine > > ovirt-engine.service - oVirt Engine > > Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled) > > Active: failed (Result: exit-code) since Mon 2015-05-25 22:26:12 PDT; > 43s ago > > Main PID: 15931 (code=exited, status=1/FAILURE) > > > > May 25 22:26:11 ovirt.polysep.ucla.edu systemd[1]: Started oVirt Engine. > > May 25 22:26:12 ovirt.polysep.ucla.edu ovirt-engine.py[15931]: > 2015-05-25 > > 22:26:12,269 ovirt-engine: ERROR run:532 Error: process termina...code 1 > > May 25 22:26:12 ovirt.polysep.ucla.edu systemd[1]: > ovirt-engine.service: main > > process exited, code=exited, status=1/FAILURE > > May 25 22:26:12 ovirt.polysep.ucla.edu systemd[1]: Unit > ovirt-engine.service > > entered failed state. > > Hint: Some lines were ellipsized, use -l to show in full. > > > > Additionally, /var/log/ovirt-engine/console.log complains “cannot > allocate > > memory”. The VM engine has 4GB of memory, but ovirt-engine engine seems > to > > want ~11GB of memory, is that right? > > > > [root@ovirt ~]# cat /var/log/ovirt-engine/console.log > > OpenJDK 64-Bit Server VM warning: INFO: > os::commit_memory(0x0000000412180000, > > 11250696192, 0) failed; error='Cannot allocate memory' (errno=12) > > # > > # There is insufficient memory for the Java Runtime Environment to > continue. > > # Native memory allocation (malloc) failed to allocate 11250696192 bytes > for > > committing reserved memory. > > # An error report file with more information is saved as: > > # /tmp/jvm-15974/hs_error.log > > [root@ovirt ~]# free -m > > total used free shared buff/cache available > > Mem: 3791 123 3175 28 492 3428 > > Swap: 2559 0 2559 > > [root@ovirt ~]# > > > > The hs_error.log is attached. > > > > Any help is appreciated. > > Can you please compare versions of all installed packages between the > source > and destination machines? > > I assume the failure above is right after engine-setup, right? Before > continuing > the interaction on hosted-engine side? > > Did you try to restart the engine (systemctl restart ovirt-engine)? Did it > fail > the same way? > > How much memory do you have on the source machine? And how much does the > engine > actually use there? > > How large is your system (hosts, VMs etc)? > > Best, > -- > Didi >