hosted engine does not start correctly (CentOS 7.1, overt-engine 3.5.2)
--Apple-Mail=_30E8FDCC-2248-490E-B934-CF7DF2FA1D68 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hello, I tried to follow <http://www.ovirt.org/Migrate_to_Hosted_Engine = <http://www.ovirt.org/Migrate_to_Hosted_Engine>> to migrate an existing = ovirt environment. I managed to complete the =E2=80=9Cengine-backup = =E2=80=94mode=3Drestore=E2=80=A6=E2=80=9D step, and the following = =E2=80=9Cengine-setup=E2=80=9D step appear to have completed. However, = the ovirt-engine service was not started successfully has errors: [root@ovirt ~]# systemctl status ovirt-engine ovirt-engine.service - oVirt Engine Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; = enabled) Active: failed (Result: exit-code) since Mon 2015-05-25 22:26:12 PDT; = 43s ago Main PID: 15931 (code=3Dexited, status=3D1/FAILURE) May 25 22:26:11 ovirt.polysep.ucla.edu systemd[1]: Started oVirt Engine. May 25 22:26:12 ovirt.polysep.ucla.edu ovirt-engine.py[15931]: = 2015-05-25 22:26:12,269 ovirt-engine: ERROR run:532 Error: process = termina...code 1 May 25 22:26:12 ovirt.polysep.ucla.edu systemd[1]: ovirt-engine.service: = main process exited, code=3Dexited, status=3D1/FAILURE May 25 22:26:12 ovirt.polysep.ucla.edu systemd[1]: Unit = ovirt-engine.service entered failed state. Hint: Some lines were ellipsized, use -l to show in full. Additionally, /var/log/ovirt-engine/console.log complains =E2=80=9Ccannot = allocate memory=E2=80=9D. The VM engine has 4GB of memory, but = ovirt-engine engine seems to want ~11GB of memory, is that right? [root@ovirt ~]# cat /var/log/ovirt-engine/console.log=20 OpenJDK 64-Bit Server VM warning: INFO: = os::commit_memory(0x0000000412180000, 11250696192, 0) failed; = error=3D'Cannot allocate memory' (errno=3D12) # # There is insufficient memory for the Java Runtime Environment to = continue. # Native memory allocation (malloc) failed to allocate 11250696192 bytes = for committing reserved memory. # An error report file with more information is saved as: # /tmp/jvm-15974/hs_error.log [root@ovirt ~]# free -m total used free shared buff/cache = available Mem: 3791 123 3175 28 492 = 3428 Swap: 2559 0 2559 [root@ovirt ~]#=20 The hs_error.log is attached. Any help is appreciated. Haven --Apple-Mail=_30E8FDCC-2248-490E-B934-CF7DF2FA1D68 Content-Type: multipart/mixed; boundary="Apple-Mail=_9C9BD97B-7859-43CB-BDAB-F17C0E9F66D0" --Apple-Mail=_9C9BD97B-7859-43CB-BDAB-F17C0E9F66D0 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D"">Hello,<div class=3D""><br class=3D""></div><div class=3D"">I = tried to follow <<a = href=3D"http://www.ovirt.org/Migrate_to_Hosted_Engine" = class=3D"">http://www.ovirt.org/Migrate_to_Hosted_Engine</a>>; to = migrate an existing ovirt environment. I managed to complete the = =E2=80=9Cengine-backup =E2=80=94mode=3Drestore=E2=80=A6=E2=80=9D step, = and the following =E2=80=9Cengine-setup=E2=80=9D step appear to have = completed. However, the ovirt-engine service was not started = successfully has errors:</div><div class=3D""><br class=3D""></div><div = class=3D""><div class=3D"">[root@ovirt ~]# systemctl status = ovirt-engine</div><div class=3D"">ovirt-engine.service - oVirt = Engine</div><div class=3D""> Loaded: loaded = (/usr/lib/systemd/system/ovirt-engine.service; enabled)</div><div = class=3D""> Active: failed (Result: exit-code) since Mon = 2015-05-25 22:26:12 PDT; 43s ago</div><div class=3D""> Main PID: = 15931 (code=3Dexited, status=3D1/FAILURE)</div><div class=3D""><br = class=3D""></div><div class=3D"">May 25 22:26:11 <a = href=3D"http://ovirt.polysep.ucla.edu" = class=3D"">ovirt.polysep.ucla.edu</a> systemd[1]: Started oVirt = Engine.</div><div class=3D"">May 25 22:26:12 <a = href=3D"http://ovirt.polysep.ucla.edu" = class=3D"">ovirt.polysep.ucla.edu</a> ovirt-engine.py[15931]: 2015-05-25 = 22:26:12,269 ovirt-engine: ERROR run:532 Error: process termina...code = 1</div><div class=3D"">May 25 22:26:12 <a = href=3D"http://ovirt.polysep.ucla.edu" = class=3D"">ovirt.polysep.ucla.edu</a> systemd[1]: ovirt-engine.service: = main process exited, code=3Dexited, status=3D1/FAILURE</div><div = class=3D"">May 25 22:26:12 <a href=3D"http://ovirt.polysep.ucla.edu" = class=3D"">ovirt.polysep.ucla.edu</a> systemd[1]: Unit = ovirt-engine.service entered failed state.</div><div class=3D"">Hint: = Some lines were ellipsized, use -l to show in full.</div></div><div = class=3D""><br class=3D""></div><div class=3D"">Additionally, = /var/log/ovirt-engine/console.log complains =E2=80=9Ccannot allocate = memory=E2=80=9D. The VM engine has 4GB of memory, but ovirt-engine = engine seems to want ~11GB of memory, is that right?</div><div = class=3D""><br class=3D""></div><div class=3D""><div = class=3D"">[root@ovirt ~]# cat = /var/log/ovirt-engine/console.log </div><div class=3D"">OpenJDK = 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000412180000, = 11250696192, 0) failed; error=3D'Cannot allocate memory' = (errno=3D12)</div><div class=3D"">#</div><div class=3D""># There is = insufficient memory for the Java Runtime Environment to = continue.</div><div class=3D""># Native memory allocation (malloc) = failed to allocate 11250696192 bytes for committing reserved = memory.</div><div class=3D""># An error report file with more = information is saved as:</div><div class=3D""># = /tmp/jvm-15974/hs_error.log</div><div class=3D""><div = class=3D"">[root@ovirt ~]# free -m</div><div class=3D""> = total used = free shared = buff/cache available</div><div class=3D"">Mem: = 3791 123 = 3175 28 = 492 3428</div><div = class=3D"">Swap: 2559 = 0 2559</div><div = class=3D"">[root@ovirt ~]# </div></div></div><div class=3D""><br = class=3D""></div><div class=3D"">The hs_error.log is attached.</div><div = class=3D""><br class=3D""></div><div class=3D"">Any help is = appreciated.</div><div class=3D""><br class=3D""></div><div = class=3D"">Haven</div><div class=3D""></div></body></html>= --Apple-Mail=_9C9BD97B-7859-43CB-BDAB-F17C0E9F66D0 Content-Disposition: attachment; filename=hs_error.log Content-Type: application/octet-stream; name="hs_error.log" Content-Transfer-Encoding: quoted-printable #=0A#=20There=20is=20insufficient=20memory=20for=20the=20Java=20Runtime=20= Environment=20to=20continue.=0A#=20Native=20memory=20allocation=20= (malloc)=20failed=20to=20allocate=2011250696192=20bytes=20for=20= committing=20reserved=20memory.=0A#=20Possible=20reasons:=0A#=20=20=20= The=20system=20is=20out=20of=20physical=20RAM=20or=20swap=20space=0A#=20=20= =20In=2032=20bit=20mode,=20the=20process=20size=20limit=20was=20hit=0A#=20= Possible=20solutions:=0A#=20=20=20Reduce=20memory=20load=20on=20the=20= system=0A#=20=20=20Increase=20physical=20memory=20or=20swap=20space=0A#=20= =20=20Check=20if=20swap=20backing=20store=20is=20full=0A#=20=20=20Use=20= 64=20bit=20Java=20on=20a=2064=20bit=20OS=0A#=20=20=20Decrease=20Java=20= heap=20size=20(-Xmx/-Xms)=0A#=20=20=20Decrease=20number=20of=20Java=20= threads=0A#=20=20=20Decrease=20Java=20thread=20stack=20sizes=20(-Xss)=0A= #=20=20=20Set=20larger=20code=20cache=20with=20= -XX:ReservedCodeCacheSize=3D=0A#=20This=20output=20file=20may=20be=20= truncated=20or=20incomplete.=0A#=0A#=20=20Out=20of=20Memory=20Error=20= (os_linux.cpp:2813),=20pid=3D15974,=20tid=3D139640083666688=0A#=0A#=20= JRE=20version:=20=20(7.0_79-b14)=20(build=20)=0A#=20Java=20VM:=20OpenJDK=20= 64-Bit=20Server=20VM=20(24.79-b02=20mixed=20mode=20linux-amd64=20= compressed=20oops)=0A#=20Derivative:=20IcedTea=202.5.5=0A#=20= Distribution:=20Built=20on=20CentOS=20Linux=20release=207.1.1503=20= (Core)=20=20(Wed=20Apr=2015=2000:14:16=20UTC=202015)=0A#=20Failed=20to=20= write=20core=20dump.=20Core=20dumps=20have=20been=20disabled.=20To=20= enable=20core=20dumping,=20try=20"ulimit=20-c=20unlimited"=20before=20= starting=20Java=20again=0A#=0A=0A---------------=20=20T=20H=20R=20E=20A=20= D=20=20---------------=0A=0ACurrent=20thread=20(0x00007f007400a800):=20=20= JavaThread=20"Unknown=20thread"=20[_thread_in_vm,=20id=3D15975,=20= stack(0x00007f007d855000,0x00007f007d956000)]=0A=0AStack:=20= [0x00007f007d855000,0x00007f007d956000],=20=20sp=3D0x00007f007d9544f0,=20= =20free=20space=3D1021k=0ANative=20frames:=20(J=3Dcompiled=20Java=20= code,=20j=3Dinterpreted,=20Vv=3DVM=20code,=20C=3Dnative=20code)=0AV=20=20= [libjvm.so+0x9791c4]=0AV=20=20[libjvm.so+0x49f5a2]=0AV=20=20= [libjvm.so+0x7eed5b]=0AV=20=20[libjvm.so+0x7e942e]=0AV=20=20= [libjvm.so+0x85d46b]=0AV=20=20[libjvm.so+0x84d35e]=0AV=20=20= [libjvm.so+0x2abd4f]=0AV=20=20[libjvm.so+0x80fae9]=0AV=20=20= [libjvm.so+0x949397]=0AV=20=20[libjvm.so+0x9496d6]=0AV=20=20= [libjvm.so+0x5b5415]=0AV=20=20[libjvm.so+0x92baef]=0AV=20=20= [libjvm.so+0x625d57]=20=20JNI_CreateJavaVM+0x67=0AC=20=20= [libjli.so+0x2a68]=0A=0A=0A---------------=20=20P=20R=20O=20C=20E=20S=20= S=20=20---------------=0A=0AJava=20Threads:=20(=20=3D>=20current=20= thread=20)=0A=0AOther=20Threads:=0A=0A=3D>0x00007f007400a800=20(exited)=20= JavaThread=20"Unknown=20thread"=20[_thread_in_vm,=20id=3D15975,=20= stack(0x00007f007d855000,0x00007f007d956000)]=0A=0AVM=20state:not=20at=20= safepoint=20(not=20fully=20initialized)=0A=0AVM=20Mutex/Monitor=20= currently=20owned=20by=20a=20thread:=20None=0A=0AGC=20Heap=20History=20= (0=20events):=0ANo=20events=0A=0ADeoptimization=20events=20(0=20events):=0A= No=20events=0A=0AInternal=20exceptions=20(0=20events):=0ANo=20events=0A=0A= Events=20(0=20events):=0ANo=20events=0A=0A=0ADynamic=20libraries:=0A= 00400000-00401000=20r-xp=2000000000=20fd:03=20101965786=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre-abrt/bin= /java=0A00600000-00601000=20r--p=2000000000=20fd:03=20101965786=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre-abrt/bin= /java=0A00601000-00602000=20rw-p=2000001000=20fd:03=20101965786=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre-abrt/bin= /java=0A01125000-01146000=20rw-p=2000000000=2000:00=200=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20[heap]=0A402180000-412180000=20rw-p=2000000000=2000:00=200=20=0A= 6b0b00000-800000000=20rw-p=2000000000=2000:00=200=20=0A= 7f006e000000-7f006e270000=20rwxp=2000000000=2000:00=200=20=0A= 7f006e270000-7f0074023000=20rw-p=2000000000=2000:00=200=20=0A= 7f0074023000-7f0078000000=20---p=2000000000=2000:00=200=20=0A= 7f0078922000-7f0079e96000=20rw-p=2000000000=2000:00=200=20=0A= 7f0079e96000-7f007a91c000=20rw-p=2000000000=2000:00=200=20=0A= 7f007a91c000-7f007aa92000=20rw-p=2000000000=2000:00=200=20=0A= 7f007aa92000-7f007aa9a000=20r-xp=2000000000=20fd:03=2069783234=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/libzip.so=0A7f007aa9a000-7f007ac99000=20---p=2000008000=20fd:03=20= 69783234=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/libzip.so=0A7f007ac99000-7f007ac9a000=20r--p=2000007000=20fd:03=20= 69783234=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/libzip.so=0A7f007ac9a000-7f007ac9b000=20rw-p=2000008000=20fd:03=20= 69783234=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/libzip.so=0A7f007ac9b000-7f007aca6000=20r-xp=2000000000=20fd:03=20= 67155745=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libnss_files-2.17.so=0A7f007aca6000-7f007aea5000=20---p=20= 0000b000=20fd:03=2067155745=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20/usr/lib64/libnss_files-2.17.so=0A7f007aea5000-7f007aea6000=20= r--p=200000a000=20fd:03=2067155745=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20/usr/lib64/libnss_files-2.17.so=0A= 7f007aea6000-7f007aea7000=20rw-p=200000b000=20fd:03=2067155745=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libnss_files-2.17.so=0A7f007aea7000-7f007aed2000=20r-xp=20= 00000000=20fd:03=2069757264=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/libjava.so=0A7f007aed2000-7f007b0d1000=20---p=200002b000=20fd:03=20= 69757264=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/libjava.so=0A7f007b0d1000-7f007b0d2000=20r--p=200002a000=20fd:03=20= 69757264=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/libjava.so=0A7f007b0d2000-7f007b0d4000=20rw-p=200002b000=20fd:03=20= 69757264=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/libjava.so=0A7f007b0d4000-7f007b0e1000=20r-xp=2000000000=20fd:03=20= 69783233=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/libverify.so=0A7f007b0e1000-7f007b2e0000=20---p=200000d000=20fd:03=20= 69783233=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/libverify.so=0A7f007b2e0000-7f007b2e2000=20r--p=200000c000=20fd:03=20= 69783233=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/libverify.so=0A7f007b2e2000-7f007b2e3000=20rw-p=200000e000=20fd:03=20= 69783233=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/libverify.so=0A7f007b2e3000-7f007b2ea000=20r-xp=2000000000=20fd:03=20= 67155757=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/librt-2.17.so=0A7f007b2ea000-7f007b4e9000=20---p=2000007000=20= fd:03=2067155757=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/librt-2.17.so=0A7f007b4e9000-7f007b4ea000=20r--p=2000006000=20= fd:03=2067155757=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/librt-2.17.so=0A7f007b4ea000-7f007b4eb000=20rw-p=2000007000=20= fd:03=2067155757=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/librt-2.17.so=0A7f007b4eb000-7f007b500000=20r-xp=2000000000=20= fd:03=2067154088=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libgcc_s-4.8.3-20140911.so.1=0A7f007b500000-7f007b6ff000=20= ---p=2000015000=20fd:03=2067154088=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20/usr/lib64/libgcc_s-4.8.3-20140911.so.1=0A= 7f007b6ff000-7f007b700000=20r--p=2000014000=20fd:03=2067154088=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libgcc_s-4.8.3-20140911.so.1=0A7f007b700000-7f007b701000=20= rw-p=2000015000=20fd:03=2067154088=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20/usr/lib64/libgcc_s-4.8.3-20140911.so.1=0A= 7f007b701000-7f007b802000=20r-xp=2000000000=20fd:03=2067155735=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20/usr/lib64/libm-2.17.so=0A= 7f007b802000-7f007ba01000=20---p=2000101000=20fd:03=2067155735=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20/usr/lib64/libm-2.17.so=0A= 7f007ba01000-7f007ba02000=20r--p=2000100000=20fd:03=2067155735=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20/usr/lib64/libm-2.17.so=0A= 7f007ba02000-7f007ba03000=20rw-p=2000101000=20fd:03=2067155735=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20/usr/lib64/libm-2.17.so=0A= 7f007ba03000-7f007baec000=20r-xp=2000000000=20fd:03=2067155808=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libstdc++.so.6.0.19=0A7f007baec000-7f007bceb000=20---p=20= 000e9000=20fd:03=2067155808=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20/usr/lib64/libstdc++.so.6.0.19=0A7f007bceb000-7f007bcf3000=20= r--p=20000e8000=20fd:03=2067155808=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20/usr/lib64/libstdc++.so.6.0.19=0A= 7f007bcf3000-7f007bcf5000=20rw-p=20000f0000=20fd:03=2067155808=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libstdc++.so.6.0.19=0A7f007bcf5000-7f007bd0a000=20rw-p=20= 00000000=2000:00=200=20=0A7f007bd0a000-7f007c853000=20r-xp=2000000000=20= fd:03=2069783238=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/server/libjvm.so=0A7f007c853000-7f007ca52000=20---p=2000b49000=20fd:03=20= 69783238=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/server/libjvm.so=0A7f007ca52000-7f007caec000=20r--p=2000b48000=20fd:03=20= 69783238=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/server/libjvm.so=0A7f007caec000-7f007cb0f000=20rw-p=2000be2000=20fd:03=20= 69783238=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/server/libjvm.so=0A7f007cb0f000-7f007cb3d000=20rw-p=2000000000=2000:00=20= 0=20=0A7f007cb3d000-7f007cb52000=20r-xp=2000000000=20fd:03=2067155853=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libz.so.1.2.7=0A7f007cb52000-7f007cd51000=20---p=2000015000=20= fd:03=2067155853=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libz.so.1.2.7=0A7f007cd51000-7f007cd52000=20r--p=2000014000=20= fd:03=2067155853=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libz.so.1.2.7=0A7f007cd52000-7f007cd53000=20rw-p=2000015000=20= fd:03=2067155853=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libz.so.1.2.7=0A7f007cd53000-7f007cf09000=20r-xp=2000000000=20= fd:03=2067155727=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libc-2.17.so=0A7f007cf09000-7f007d109000=20---p=20001b6000=20= fd:03=2067155727=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libc-2.17.so=0A7f007d109000-7f007d10d000=20r--p=20001b6000=20= fd:03=2067155727=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libc-2.17.so=0A7f007d10d000-7f007d10f000=20rw-p=20001ba000=20= fd:03=2067155727=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libc-2.17.so=0A7f007d10f000-7f007d114000=20rw-p=2000000000=20= 00:00=200=20=0A7f007d114000-7f007d117000=20r-xp=2000000000=20fd:03=20= 67155733=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libdl-2.17.so=0A7f007d117000-7f007d316000=20---p=2000003000=20= fd:03=2067155733=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libdl-2.17.so=0A7f007d316000-7f007d317000=20r--p=2000002000=20= fd:03=2067155733=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libdl-2.17.so=0A7f007d317000-7f007d318000=20rw-p=2000003000=20= fd:03=2067155733=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libdl-2.17.so=0A7f007d318000-7f007d325000=20r-xp=2000000000=20= fd:03=2035357316=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/jli/libjli.so=0A7f007d325000-7f007d524000=20---p=200000d000=20fd:03=20= 35357316=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/jli/libjli.so=0A7f007d524000-7f007d525000=20r--p=200000c000=20fd:03=20= 35357316=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/jli/libjli.so=0A7f007d525000-7f007d526000=20rw-p=200000d000=20fd:03=20= 35357316=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.x86_64/jre/lib/amd6= 4/jli/libjli.so=0A7f007d526000-7f007d53c000=20r-xp=2000000000=20fd:03=20= 67155753=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libpthread-2.17.so=0A7f007d53c000-7f007d73c000=20---p=20= 00016000=20fd:03=2067155753=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20/usr/lib64/libpthread-2.17.so=0A7f007d73c000-7f007d73d000=20= r--p=2000016000=20fd:03=2067155753=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20/usr/lib64/libpthread-2.17.so=0A= 7f007d73d000-7f007d73e000=20rw-p=2000017000=20fd:03=2067155753=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/libpthread-2.17.so=0A7f007d73e000-7f007d742000=20rw-p=20= 00000000=2000:00=200=20=0A7f007d742000-7f007d763000=20r-xp=2000000000=20= fd:03=2067155720=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= /usr/lib64/ld-2.17.so=0A7f007d84d000-7f007d855000=20rw-s=2000000000=20= fd:03=20333874=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20/tmp/hsperfdata_ovirt/15974=0A7f007d855000-7f007d858000=20---p=20= 00000000=2000:00=200=20=0A7f007d858000-7f007d95b000=20rw-p=2000000000=20= 00:00=200=20=0A7f007d95e000-7f007d961000=20rw-p=2000000000=2000:00=200=20= =0A7f007d961000-7f007d962000=20r--p=2000000000=2000:00=200=20=0A= 7f007d962000-7f007d963000=20rw-p=2000000000=2000:00=200=20=0A= 7f007d963000-7f007d964000=20r--p=2000021000=20fd:03=2067155720=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20/usr/lib64/ld-2.17.so=0A= 7f007d964000-7f007d965000=20rw-p=2000022000=20fd:03=2067155720=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20/usr/lib64/ld-2.17.so=0A= 7f007d965000-7f007d966000=20rw-p=2000000000=2000:00=200=20=0A= 7fff4f316000-7fff4f337000=20rw-p=2000000000=2000:00=200=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20[stack]=0A= 7fff4f3fe000-7fff4f400000=20r-xp=2000000000=2000:00=200=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20[vdso]=0A= ffffffffff600000-ffffffffff601000=20r-xp=2000000000=2000:00=200=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20[vsyscall]=0A=0AVM=20= Arguments:=0Ajvm_args:=20-XX:+TieredCompilation=20-Xms16094M=20= -Xmx16094M=20-XX:PermSize=3D256m=20-XX:MaxPermSize=3D256m=20= -Djava.net.preferIPv4Stack=3Dtrue=20= -Dsun.rmi.dgc.client.gcInterval=3D3600000=20= -Dsun.rmi.dgc.server.gcInterval=3D3600000=20-Djava.awt.headless=3Dtrue=20= -Djsse.enableSNIExtension=3Dfalse=20= -Djava.security.krb5.conf=3D/etc/ovirt-engine/krb5.conf=20= -XX:+HeapDumpOnOutOfMemoryError=20= -XX:HeapDumpPath=3D/var/log/ovirt-engine/dump=20= -Djava.util.logging.manager=3Dorg.jboss.logmanager=20= -Dlogging.configuration=3Dfile:///var/lib/ovirt-engine/jboss_runtime/confi= g/ovirt-engine-logging.properties=20-Dorg.jboss.resolver.warning=3Dtrue=20= -Djboss.modules.system.pkgs=3Dorg.jboss.byteman=20= -Djboss.modules.write-indexes=3Dfalse=20= -Djboss.server.default.config=3Dovirt-engine=20= -Djboss.home.dir=3D/usr/share/ovirt-engine-jboss-as=20= -Djboss.server.base.dir=3D/usr/share/ovirt-engine=20= -Djboss.server.data.dir=3D/var/lib/ovirt-engine=20= -Djboss.server.log.dir=3D/var/log/ovirt-engine=20= -Djboss.server.config.dir=3D/var/lib/ovirt-engine/jboss_runtime/config=20= -Djboss.server.temp.dir=3D/var/lib/ovirt-engine/jboss_runtime/tmp=20= -Djboss.controller.temp.dir=3D/var/lib/ovirt-engine/jboss_runtime/tmp=20=0A= java_command:=20/usr/share/ovirt-engine-jboss-as/jboss-modules.jar=20-mp=20= /var/lib/ovirt-engine/jboss_runtime/modules/00-ovirt-engine-modules:/var/l= ib/ovirt-engine/jboss_runtime/modules/01-ovirt-engine-jboss-as-modules=20= -jaxpmodule=20javax.xml.jaxp-provider=20org.jboss.as.standalone=20-c=20= ovirt-engine.xml=0ALauncher=20Type:=20SUN_STANDARD=0A=0AEnvironment=20= Variables:=0A= PATH=3D/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin=0A=0A= Signal=20Handlers:=0ASIGSEGV:=20[libjvm.so+0x979d40],=20= sa_mask[0]=3D0x7ffbfeff,=20sa_flags=3D0x10000004=0ASIGBUS:=20= [libjvm.so+0x979d40],=20sa_mask[0]=3D0x7ffbfeff,=20sa_flags=3D0x10000004=0A= SIGFPE:=20[libjvm.so+0x7ea2c0],=20sa_mask[0]=3D0x7ffbfeff,=20= sa_flags=3D0x10000004=0ASIGPIPE:=20[libjvm.so+0x7ea2c0],=20= sa_mask[0]=3D0x7ffbfeff,=20sa_flags=3D0x10000004=0ASIGXFSZ:=20= [libjvm.so+0x7ea2c0],=20sa_mask[0]=3D0x7ffbfeff,=20sa_flags=3D0x10000004=0A= SIGILL:=20[libjvm.so+0x7ea2c0],=20sa_mask[0]=3D0x7ffbfeff,=20= sa_flags=3D0x10000004=0ASIGUSR1:=20SIG_DFL,=20sa_mask[0]=3D0x00000000,=20= sa_flags=3D0x00000000=0ASIGUSR2:=20[libjvm.so+0x7ea130],=20= sa_mask[0]=3D0x00000000,=20sa_flags=3D0x10000004=0ASIGHUP:=20SIG_IGN,=20= sa_mask[0]=3D0x00000000,=20sa_flags=3D0x00000000=0ASIGINT:=20SIG_DFL,=20= sa_mask[0]=3D0x00000000,=20sa_flags=3D0x00000000=0ASIGTERM:=20SIG_DFL,=20= sa_mask[0]=3D0x00000000,=20sa_flags=3D0x00000000=0ASIGQUIT:=20SIG_DFL,=20= sa_mask[0]=3D0x00000000,=20sa_flags=3D0x00000000=0ASIGTRAP:=20SIG_DFL,=20= sa_mask[0]=3D0x00000000,=20sa_flags=3D0x00000000=0A=0A=0A---------------=20= =20S=20Y=20S=20T=20E=20M=20=20---------------=0A=0AOS:CentOS=20Linux=20= release=207.1.1503=20(Core)=20=0A=0Auname:Linux=20= 3.10.0-229.4.2.el7.x86_64=20#1=20SMP=20Wed=20May=2013=2010:06:09=20UTC=20= 2015=20x86_64=0Alibc:glibc=202.17=20NPTL=202.17=20=0Arlimit:=20STACK=20= 8192k,=20CORE=200k,=20NPROC=2015100,=20NOFILE=2065535,=20AS=20infinity=0A= load=20average:0.59=200.19=200.10=0A=0A/proc/meminfo:=0AMemTotal:=20=20=20= =20=20=20=20=203882732=20kB=0AMemFree:=20=20=20=20=20=20=20=20=203194460=20= kB=0AMemAvailable:=20=20=20=203453684=20kB=0ABuffers:=20=20=20=20=20=20=20= =20=20=20=20=201328=20kB=0ACached:=20=20=20=20=20=20=20=20=20=20=20= 438472=20kB=0ASwapCached:=20=20=20=20=20=20=20=20=20=20=20=200=20kB=0A= Active:=20=20=20=20=20=20=20=20=20=20=20307816=20kB=0AInactive:=20=20=20=20= =20=20=20=20=20257888=20kB=0AActive(anon):=20=20=20=20=20140056=20kB=0A= Inactive(anon):=20=20=20=2014952=20kB=0AActive(file):=20=20=20=20=20= 167760=20kB=0AInactive(file):=20=20=20242936=20kB=0AUnevictable:=20=20=20= =20=20=20=20=20=20=20=200=20kB=0AMlocked:=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=200=20kB=0ASwapTotal:=20=20=20=20=20=20=202621436=20kB=0A= SwapFree:=20=20=20=20=20=20=20=202621436=20kB=0ADirty:=20=20=20=20=20=20=20= =20=20=20=20=20=2012548=20kB=0AWriteback:=20=20=20=20=20=20=20=20=20=20=20= =20=200=20kB=0AAnonPages:=20=20=20=20=20=20=20=20115844=20kB=0AMapped:=20= =20=20=20=20=20=20=20=20=20=20=2042464=20kB=0AShmem:=20=20=20=20=20=20=20= =20=20=20=20=20=2029104=20kB=0ASlab:=20=20=20=20=20=20=20=20=20=20=20=20=20= =2066528=20kB=0ASReclaimable:=20=20=20=20=20=2034928=20kB=0ASUnreclaim:=20= =20=20=20=20=20=20=2031600=20kB=0AKernelStack:=20=20=20=20=20=20=20=20= 1968=20kB=0APageTables:=20=20=20=20=20=20=20=20=206572=20kB=0A= NFS_Unstable:=20=20=20=20=20=20=20=20=20=200=20kB=0ABounce:=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=200=20kB=0AWritebackTmp:=20=20=20=20=20=20= =20=20=20=200=20kB=0ACommitLimit:=20=20=20=20=204562800=20kB=0A= Committed_AS:=20=20=20=205885860=20kB=0AVmallocTotal:=20=20=20= 34359738367=20kB=0AVmallocUsed:=20=20=20=20=20=20=2015772=20kB=0A= VmallocChunk:=20=20=2034359719676=20kB=0AHardwareCorrupted:=20=20=20=20=20= 0=20kB=0AAnonHugePages:=20=20=20=20=20=208192=20kB=0AHugePages_Total:=20=20= =20=20=20=20=200=0AHugePages_Free:=20=20=20=20=20=20=20=200=0A= HugePages_Rsvd:=20=20=20=20=20=20=20=200=0AHugePages_Surp:=20=20=20=20=20= =20=20=200=0AHugepagesize:=20=20=20=20=20=20=202048=20kB=0ADirectMap4k:=20= =20=20=20=20=20=2059380=20kB=0ADirectMap2M:=20=20=20=20=204134912=20kB=0A= =0A=0ACPU:total=202=20(1=20cores=20per=20cpu,=201=20threads=20per=20= core)=20family=206=20model=2042=20stepping=201,=20cmov,=20cx8,=20fxsr,=20= mmx,=20sse,=20sse2,=20sse3,=20ssse3,=20sse4.1,=20sse4.2,=20popcnt,=20= avx,=20aes,=20tsc=0A=0A/proc/cpuinfo:=0Aprocessor=09:=200=0Avendor_id=09= :=20GenuineIntel=0Acpu=20family=09:=206=0Amodel=09=09:=2042=0Amodel=20= name=09:=20Intel=20Xeon=20E312xx=20(Sandy=20Bridge)=0Astepping=09:=201=0A= microcode=09:=200x1=0Acpu=20MHz=09=09:=202399.998=0Acache=20size=09:=20= 4096=20KB=0Aphysical=20id=09:=200=0Asiblings=09:=201=0Acore=20id=09=09:=20= 0=0Acpu=20cores=09:=201=0Aapicid=09=09:=200=0Ainitial=20apicid=09:=200=0A= fpu=09=09:=20yes=0Afpu_exception=09:=20yes=0Acpuid=20level=09:=2013=0Awp=09= =09:=20yes=0Aflags=09=09:=20fpu=20de=20pse=20tsc=20msr=20pae=20mce=20cx8=20= apic=20sep=20mtrr=20pge=20mca=20cmov=20pat=20pse36=20clflush=20mmx=20= fxsr=20sse=20sse2=20syscall=20nx=20lm=20constant_tsc=20nopl=20eagerfpu=20= pni=20pclmulqdq=20ssse3=20cx16=20sse4_1=20sse4_2=20x2apic=20popcnt=20= tsc_deadline_timer=20aes=20xsave=20avx=20hypervisor=20lahf_lm=20xsaveopt=0A= bogomips=09:=204799.99=0Aclflush=20size=09:=2064=0Acache_alignment=09:=20= 64=0Aaddress=20sizes=09:=2046=20bits=20physical,=2048=20bits=20virtual=0A= power=20management:=0A=0Aprocessor=09:=201=0Avendor_id=09:=20= GenuineIntel=0Acpu=20family=09:=206=0Amodel=09=09:=2042=0Amodel=20name=09= :=20Intel=20Xeon=20E312xx=20(Sandy=20Bridge)=0Astepping=09:=201=0A= microcode=09:=200x1=0Acpu=20MHz=09=09:=202399.998=0Acache=20size=09:=20= 4096=20KB=0Aphysical=20id=09:=201=0Asiblings=09:=201=0Acore=20id=09=09:=20= 0=0Acpu=20cores=09:=201=0Aapicid=09=09:=201=0Ainitial=20apicid=09:=201=0A= fpu=09=09:=20yes=0Afpu_exception=09:=20yes=0Acpuid=20level=09:=2013=0Awp=09= =09:=20yes=0Aflags=09=09:=20fpu=20de=20pse=20tsc=20msr=20pae=20mce=20cx8=20= apic=20sep=20mtrr=20pge=20mca=20cmov=20pat=20pse36=20clflush=20mmx=20= fxsr=20sse=20sse2=20syscall=20nx=20lm=20constant_tsc=20nopl=20eagerfpu=20= pni=20pclmulqdq=20ssse3=20cx16=20sse4_1=20sse4_2=20x2apic=20popcnt=20= tsc_deadline_timer=20aes=20xsave=20avx=20hypervisor=20lahf_lm=20xsaveopt=0A= bogomips=09:=204799.99=0Aclflush=20size=09:=2064=0Acache_alignment=09:=20= 64=0Aaddress=20sizes=09:=2046=20bits=20physical,=2048=20bits=20virtual=0A= power=20management:=0A=0A=0A=0AMemory:=204k=20page,=20physical=20= 3882732k(3194460k=20free),=20swap=202621436k(2621436k=20free)=0A=0A= vm_info:=20OpenJDK=2064-Bit=20Server=20VM=20(24.79-b02)=20for=20= linux-amd64=20JRE=20(1.7.0_79-b14),=20built=20on=20Apr=2015=202015=20= 00:19:48=20by=20"mockbuild"=20with=20gcc=204.8.3=2020140911=20(Red=20Hat=20= 4.8.3-9)=0A=0Atime:=20Mon=20May=2025=2022:26:12=202015=0Aelapsed=20time:=20= 0=20seconds=0A=0A= --Apple-Mail=_9C9BD97B-7859-43CB-BDAB-F17C0E9F66D0 Content-Transfer-Encoding: 7bit Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""></div><div class=""><br class=""></div><div class=""><br class=""></div></body></html> --Apple-Mail=_9C9BD97B-7859-43CB-BDAB-F17C0E9F66D0-- --Apple-Mail=_30E8FDCC-2248-490E-B934-CF7DF2FA1D68--
----- Original Message -----
From: "Haoyang Haven Liu" <haven.liu@ucla.edu> To: users@ovirt.org Sent: Tuesday, May 26, 2015 8:34:00 AM Subject: [ovirt-users] hosted engine does not start correctly (CentOS 7.1, overt-engine 3.5.2)
Hello,
I tried to follow < http://www.ovirt.org/Migrate_to_Hosted_Engine > to migrate an existing ovirt environment. I managed to complete the “engine-backup —mode=restore…” step, and the following “engine-setup” step appear to have completed. However, the ovirt-engine service was not started successfully has errors:
[root@ovirt ~]# systemctl status ovirt-engine ovirt-engine.service - oVirt Engine Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled) Active: failed (Result: exit-code) since Mon 2015-05-25 22:26:12 PDT; 43s ago Main PID: 15931 (code=exited, status=1/FAILURE)
May 25 22:26:11 ovirt.polysep.ucla.edu systemd[1]: Started oVirt Engine. May 25 22:26:12 ovirt.polysep.ucla.edu ovirt-engine.py[15931]: 2015-05-25 22:26:12,269 ovirt-engine: ERROR run:532 Error: process termina...code 1 May 25 22:26:12 ovirt.polysep.ucla.edu systemd[1]: ovirt-engine.service: main process exited, code=exited, status=1/FAILURE May 25 22:26:12 ovirt.polysep.ucla.edu systemd[1]: Unit ovirt-engine.service entered failed state. Hint: Some lines were ellipsized, use -l to show in full.
Additionally, /var/log/ovirt-engine/console.log complains “cannot allocate memory”. The VM engine has 4GB of memory, but ovirt-engine engine seems to want ~11GB of memory, is that right?
[root@ovirt ~]# cat /var/log/ovirt-engine/console.log OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000412180000, 11250696192, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 11250696192 bytes for committing reserved memory. # An error report file with more information is saved as: # /tmp/jvm-15974/hs_error.log [root@ovirt ~]# free -m total used free shared buff/cache available Mem: 3791 123 3175 28 492 3428 Swap: 2559 0 2559 [root@ovirt ~]#
The hs_error.log is attached.
Any help is appreciated.
Can you please compare versions of all installed packages between the source and destination machines? I assume the failure above is right after engine-setup, right? Before continuing the interaction on hosted-engine side? Did you try to restart the engine (systemctl restart ovirt-engine)? Did it fail the same way? How much memory do you have on the source machine? And how much does the engine actually use there? How large is your system (hosts, VMs etc)? Best, -- Didi
Didi, Both source and destination machines (assuming you are referring to the physical host source and virtual host destination of the migration) are running ovirt-engine 3.5.2.1-1. However, the "source" is on CentOS 6.6, whereas the "destination" is on CentOS 7.1. The source is running postgresql(/-server) 8.4.20, Java 1.7.0_79, httpd 2.2.15. Destination is running postgresql(/-server) 9.2.10, Java 1.7.0.79, jboss-httpd 2.4.6. I'm not sure which over package version is relevant. The failure occurs after engine-setup, and the web UI doesn't load, and the host reports the engine is not up (I'm not in front of the host, so I don't have the exact language). Our current setup has 2 physical hosts, ~15 active VMs. Each host has 64GB RAM. Even with all the VM on one host, the utilization is less than 50%. We have another physical storage server, that provides the NFS storage backend to the hosts and VMs. Prior to attempting the hosted-engine migration, I migrated all the VMs to host #1, where the engine was originally installed. I then removed host #2 from the host list via the UI, and performed hosted-engine --deploy on host #2. A few other quirks I encountered: 1. I originally wanted to do "hosted-engine --deploy" on host #1, but that it was unsuccessful due to it complaining the storage connection was busy (or something). 2. I then tried to do "hosted-engine --deploy" on host #2, using the same storage connection path as what I used for host #1 originally. The setup process asked if it was an "additional host", I said no, but later it cannot activate the storage connection. 3. Lastly I did "hosted-engine --deploy" on host #2, but pointing to a different storage connection path, still on the same storage server. This was successful until the I encountered the failure in question. Thanks, Haven On Tue, May 26, 2015 at 2:20 AM, Yedidyah Bar David <didi@redhat.com> wrote:
----- Original Message -----
From: "Haoyang Haven Liu" <haven.liu@ucla.edu> To: users@ovirt.org Sent: Tuesday, May 26, 2015 8:34:00 AM Subject: [ovirt-users] hosted engine does not start correctly (CentOS 7.1, overt-engine 3.5.2)
Hello,
I tried to follow < http://www.ovirt.org/Migrate_to_Hosted_Engine > to migrate an existing ovirt environment. I managed to complete the “engine-backup —mode=restore…” step, and the following “engine-setup” step appear to have completed. However, the ovirt-engine service was not started successfully has errors:
[root@ovirt ~]# systemctl status ovirt-engine ovirt-engine.service - oVirt Engine Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled) Active: failed (Result: exit-code) since Mon 2015-05-25 22:26:12 PDT; 43s ago Main PID: 15931 (code=exited, status=1/FAILURE)
May 25 22:26:11 ovirt.polysep.ucla.edu systemd[1]: Started oVirt Engine. May 25 22:26:12 ovirt.polysep.ucla.edu ovirt-engine.py[15931]: 2015-05-25 22:26:12,269 ovirt-engine: ERROR run:532 Error: process termina...code 1 May 25 22:26:12 ovirt.polysep.ucla.edu systemd[1]: ovirt-engine.service: main process exited, code=exited, status=1/FAILURE May 25 22:26:12 ovirt.polysep.ucla.edu systemd[1]: Unit ovirt-engine.service entered failed state. Hint: Some lines were ellipsized, use -l to show in full.
Additionally, /var/log/ovirt-engine/console.log complains “cannot allocate memory”. The VM engine has 4GB of memory, but ovirt-engine engine seems to want ~11GB of memory, is that right?
[root@ovirt ~]# cat /var/log/ovirt-engine/console.log OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000412180000, 11250696192, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 11250696192 bytes for committing reserved memory. # An error report file with more information is saved as: # /tmp/jvm-15974/hs_error.log [root@ovirt ~]# free -m total used free shared buff/cache available Mem: 3791 123 3175 28 492 3428 Swap: 2559 0 2559 [root@ovirt ~]#
The hs_error.log is attached.
Any help is appreciated.
Can you please compare versions of all installed packages between the source and destination machines?
I assume the failure above is right after engine-setup, right? Before continuing the interaction on hosted-engine side?
Did you try to restart the engine (systemctl restart ovirt-engine)? Did it fail the same way?
How much memory do you have on the source machine? And how much does the engine actually use there?
How large is your system (hosts, VMs etc)?
Best, -- Didi
----- Original Message -----
From: "H. Haven Liu" <haven.liu@ucla.edu> To: "Yedidyah Bar David" <didi@redhat.com> Cc: "oVirt Mailing List" <users@ovirt.org> Sent: Tuesday, May 26, 2015 10:26:07 PM Subject: Re: engine tries to allocate 11GB and aborts (was: hosted engine does not start correctly (CentOS 7.1, overt-engine 3.5.2))
Didi,
Both source and destination machines (assuming you are referring to the physical host source and virtual host destination of the migration) are running ovirt-engine 3.5.2.1-1. However, the "source" is on CentOS 6.6, whereas the "destination" is on CentOS 7.1. The source is running postgresql(/-server) 8.4.20, Java 1.7.0_79, httpd 2.2.15. Destination is running postgresql(/-server) 9.2.10, Java 1.7.0.79, jboss-httpd 2.4.6. I'm not sure which over package version is relevant.
Sorry for not replying earlier. Now opened [1] to track such upgrades. Didn't try that myself. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1234257
The failure occurs after engine-setup, and the web UI doesn't load, and the host reports the engine is not up (I'm not in front of the host, so I don't have the exact language).
Our current setup has 2 physical hosts, ~15 active VMs. Each host has 64GB RAM. Even with all the VM on one host, the utilization is less than 50%. We have another physical storage server, that provides the NFS storage backend to the hosts and VMs. Prior to attempting the hosted-engine migration, I migrated all the VMs to host #1, where the engine was originally installed. I then removed host #2 from the host list via the UI, and performed hosted-engine --deploy on host #2. A few other quirks I encountered:
1. I originally wanted to do "hosted-engine --deploy" on host #1, but that it was unsuccessful due to it complaining the storage connection was busy (or something). 2. I then tried to do "hosted-engine --deploy" on host #2, using the same storage connection path as what I used for host #1 originally. The setup process asked if it was an "additional host", I said no, but later it cannot activate the storage connection. 3. Lastly I did "hosted-engine --deploy" on host #2, but pointing to a different storage connection path, still on the same storage server. This was successful until the I encountered the failure in question.
Not sure about the exact causes for these issues. If still not resolved, please retry on a clean OS reinstalled. Best,
Thanks,
Haven
On Tue, May 26, 2015 at 2:20 AM, Yedidyah Bar David <didi@redhat.com> wrote:
----- Original Message -----
From: "Haoyang Haven Liu" <haven.liu@ucla.edu> To: users@ovirt.org Sent: Tuesday, May 26, 2015 8:34:00 AM Subject: [ovirt-users] hosted engine does not start correctly (CentOS 7.1, overt-engine 3.5.2)
Hello,
I tried to follow < http://www.ovirt.org/Migrate_to_Hosted_Engine > to migrate an existing ovirt environment. I managed to complete the “engine-backup —mode=restore…” step, and the following “engine-setup” step appear to have completed. However, the ovirt-engine service was not started successfully has errors:
[root@ovirt ~]# systemctl status ovirt-engine ovirt-engine.service - oVirt Engine Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled) Active: failed (Result: exit-code) since Mon 2015-05-25 22:26:12 PDT; 43s ago Main PID: 15931 (code=exited, status=1/FAILURE)
May 25 22:26:11 ovirt.polysep.ucla.edu systemd[1]: Started oVirt Engine. May 25 22:26:12 ovirt.polysep.ucla.edu ovirt-engine.py[15931]: 2015-05-25 22:26:12,269 ovirt-engine: ERROR run:532 Error: process termina...code 1 May 25 22:26:12 ovirt.polysep.ucla.edu systemd[1]: ovirt-engine.service: main process exited, code=exited, status=1/FAILURE May 25 22:26:12 ovirt.polysep.ucla.edu systemd[1]: Unit ovirt-engine.service entered failed state. Hint: Some lines were ellipsized, use -l to show in full.
Additionally, /var/log/ovirt-engine/console.log complains “cannot allocate memory”. The VM engine has 4GB of memory, but ovirt-engine engine seems to want ~11GB of memory, is that right?
[root@ovirt ~]# cat /var/log/ovirt-engine/console.log OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000412180000, 11250696192, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 11250696192 bytes for committing reserved memory. # An error report file with more information is saved as: # /tmp/jvm-15974/hs_error.log [root@ovirt ~]# free -m total used free shared buff/cache available Mem: 3791 123 3175 28 492 3428 Swap: 2559 0 2559 [root@ovirt ~]#
The hs_error.log is attached.
Any help is appreciated.
Can you please compare versions of all installed packages between the source and destination machines?
I assume the failure above is right after engine-setup, right? Before continuing the interaction on hosted-engine side?
Did you try to restart the engine (systemctl restart ovirt-engine)? Did it fail the same way?
How much memory do you have on the source machine? And how much does the engine actually use there?
How large is your system (hosts, VMs etc)?
Best, -- Didi
-- Didi
participants (3)
-
H. Haven Liu -
Haoyang Haven Liu
-
Yedidyah Bar David