getting 404 after fresh install of oVirt 3.4 on CentOS 6.5 (+ solution)
Hello all, I was asked to post the following problem on this mailinglist. After installing a clean CentOS 6.5 with a clean oVirt 3.4, following the instructions from http://www.ovirt.org/Download#Red_Hat_Enterprise_Linux_6.2FCentOS_Installati... I went to http://localhost.localdomain:80/ovirt-engine as indicated and was redirected to a 404 page. The same happened when using https. During the installation, engine-setup will print out the following on my VM with 512MB RAM: [WARNING] Warning: Not enough memory is available on the host. Minimum requirement is 4096MB, and 16384MB is recommended. SSH fingerprint: 4B:DE:48:26:99:AA:C0:72:E3:C8:B5:64:5F:6E:6D:00 Internal CA FB:82:FE:14:35:3A:BE:1A:B1:E6:99:C2:DC:CD:6D:E0:44:64:0F:47 Web access is enabled at: http://localhost.localdomain:80/ovirt-engine https://localhost.localdomain:443/ovirt-engine Please use the user "admin" and password specified in order to login into oVirt Engine The consequence of not having enough RAM for oVirt is that it will silently fail to start up, without apparent errors or warnings. In my view, the warning above should be rephrased as "Not enough memory is available on the host, oVirt will refuse to start" and colored red. The solution to this problem is to have at least 4GB of RAM, after which oVirt seems to start up fine (with only 1.2GB of RAM in use). kr, -- Steven
----- Original Message -----
From: "Steven Van Acker" <Steven.VanAcker@cs.kuleuven.be> To: users@ovirt.org Sent: Tuesday, May 13, 2014 10:39:03 AM Subject: [ovirt-users] getting 404 after fresh install of oVirt 3.4 on CentOS 6.5 (+ solution)
Hello all,
I was asked to post the following problem on this mailinglist.
After installing a clean CentOS 6.5 with a clean oVirt 3.4, following the instructions from http://www.ovirt.org/Download#Red_Hat_Enterprise_Linux_6.2FCentOS_Installati... I went to http://localhost.localdomain:80/ovirt-engine as indicated and was redirected to a 404 page. The same happened when using https.
During the installation, engine-setup will print out the following on my VM with 512MB RAM:
[WARNING] Warning: Not enough memory is available on the host. Minimum requirement is 4096MB, and 16384MB is recommended. SSH fingerprint: 4B:DE:48:26:99:AA:C0:72:E3:C8:B5:64:5F:6E:6D:00 Internal CA FB:82:FE:14:35:3A:BE:1A:B1:E6:99:C2:DC:CD:6D:E0:44:64:0F:47 Web access is enabled at: http://localhost.localdomain:80/ovirt-engine https://localhost.localdomain:443/ovirt-engine Please use the user "admin" and password specified in order to login into oVirt Engine
The consequence of not having enough RAM for oVirt is that it will silently fail to start up, without apparent errors or warnings.
You mean, in the log?
In my view, the warning above should be rephrased as "Not enough memory is available on the host, oVirt will refuse to start" and colored red.
Did it 'refuse' to start, or simply failed to start? With what conditions exactly you want this error? What if the engine does manage to start (e.g. because you did some weird tweaking and managed to make it run in 256MB)? What if it later fails, after you add a few tens of hosts/VMs? I know I regularly use a 1GB VM for testing setup, never had problems. Generally, red stuff in setup is reserved to real errors - when something failed.
The solution to this problem is to have at least 4GB of RAM, after which oVirt seems to start up fine (with only 1.2GB of RAM in use).
How did you measure that? On my current test VM, I have: # ps aux | grep ovirt-engine ovirt 10840 0.0 0.2 207952 2168 ? Ss 10:08 0:00 /usr/bin/python /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output --systemd=notify start ovirt 10879 0.8 52.4 2247040 534892 ? Sl 10:08 0:46 ovirt-engine -server -XX:+TieredCompilation -Xms1g -Xmx1g -XX:PermSize=256m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true -Djsse.enableSNIExtension=false -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/ovirt-engine/dump -Djava.util.logging.manager=org.jboss.logmanager -Dlogging.configuration=file:///var/tmp/ovirt-engine/config/ovirt-engine-logging.properties -Dorg.jboss.resolver.warning=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.modules.write-indexes=false -Djboss.server.default.config=ovirt-engine -Djboss.home.dir=/usr/share/jboss-as -Djboss.server.base.dir=/usr/share/ovirt-engine -Djboss.server.data.dir=/var/lib/ovirt-engine -Djboss.server.log.dir=/var/log/ovirt-engine -Djboss.server.config.dir=/var/tmp/ovirt-engine/config -Djboss.server.temp.dir=/var/tmp/ovirt-engine/tmp -Djboss.controller.temp.dir=/var/tmp/ovirt-engine/tmp -jar /usr/share/jboss-as/jboss-modules.jar -mp /var/tmp/ovirt-engine/modules/00-ovirt-engin -modules:/var/tmp/ovirt-engine/modules/01-jboss-as-modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -c ovirt-engine.xml ovirt 10900 0.0 0.2 286084 2604 ? Ss 10:08 0:01 /usr/bin/python /usr/share/ovirt-engine/services/ovirt-websocket-proxy/ovirt-websocket-proxy.py --systemd=notify start root 12200 0.0 0.0 112636 964 pts/10 S+ 11:38 0:00 grep --color=auto ovirt-engine # free total used free shared buffers cached Mem: 1019660 936780 82880 0 5068 259496 -/+ buffers/cache: 672216 347444 Swap: 14749692 164328 14585364 So the engine uses 2.2GB of virtual memory, but only 530MB rss, and total system use is, without caches, 670MB. So it's likely that a 512MB VM will not be enough. Thanks for the report! -- Didi
Am 13.05.2014 10:45, schrieb Yedidyah Bar David:
So the engine uses 2.2GB of virtual memory, but only 530MB rss, and total system use is, without caches, 670MB. So it's likely that a 512MB VM will not be enough.
Doesn't this make you wonder where the minimum requirements come from? If it runs with less than 1 GB RAM, why do the docs say you need 4 GB and recommend even 16 GB ? Is it just a matter of scale(number of vms/hosts/DCs) ? What would make engine consume more RAM? Can you maybe lower the minimum requirements? I know RedHat does some heavy load/scale tests so maybe someone can share some information about ram/other resource usage in heavy load scenarios? -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH & Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
On Tue, 13 May 2014, Sven Kieske wrote:
Doesn't this make you wonder where the minimum requirements come from?
If it runs with less than 1 GB RAM, why do the docs say you need 4 GB and recommend even 16 GB ?
certainly a fair question ... There is also a statement in that setup script as to needed filesystem space which seems to have been simply 'pulled out of the air', rather than documented / explained
Is it just a matter of scale(number of vms/hosts/DCs) ? What would make engine consume more RAM?
Can you maybe lower the minimum requirements?
Or isolate the recommendations to a flat file which is commented, and sourced by the script, so a person can discern the difference between 'hard' requirements, and simple 'recommendations' for a stated use case -- Russ herrold
I've seen a similar problem (404 on webadmin) on my 4GB physical ovirt engine box, but it turned out to be the jboss timeout in deployment Relevant bz https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1078291 so it might have been that it wasn't a memory limit in the OP that caused failure, but the time it took jboss to deploy -John On Tue, May 13, 2014 at 12:31 PM, R P Herrold <herrold@owlriver.com> wrote:
On Tue, 13 May 2014, Sven Kieske wrote:
Doesn't this make you wonder where the minimum requirements come from?
If it runs with less than 1 GB RAM, why do the docs say you need 4 GB and recommend even 16 GB ?
certainly a fair question ... There is also a statement in that setup script as to needed filesystem space which seems to have been simply 'pulled out of the air', rather than documented / explained
Is it just a matter of scale(number of vms/hosts/DCs) ? What would make engine consume more RAM?
Can you maybe lower the minimum requirements?
Or isolate the recommendations to a flat file which is commented, and sourced by the script, so a person can discern the difference between 'hard' requirements, and simple 'recommendations' for a stated use case
-- Russ herrold _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I am the reporter of the below bz, and the author of the fix [1]. However, I don't know jboss at all - just searched around, found somewhere the option 'deployment-timeout' (not sure where, probably a quick search can find this) and verified that the fix works. It seems like neither any of the reviewers of the change know enough about jboss to do some more analysis. Can anyone else, that knows jboss, please look at this issue? Specifically: 1. Was anything related to the maximum time for deployments changed in relevant jboss versions? 2. What is affected? What isn't? - fedora 19/20(/21?), el6 - ovirt repos (3.3, 3.4, master) and minimum requirements John: If this is reproducible for you, you might want to try the fix [1] (linked to from the bug). If you do, please report - what exact versions (OS, jboss, ovirt, etc) you used, and (according to the logs) how long did the deployment take. Thanks! [1] http://gerrit.ovirt.org/25895 Best Regards, -- Didi ----- Original Message -----
From: "John Taylor" <jtt77777@gmail.com> To: "R P Herrold" <herrold@owlriver.com> Cc: users@ovirt.org Sent: Tuesday, May 13, 2014 11:36:57 PM Subject: Re: [ovirt-users] getting 404 after fresh install of oVirt 3.4 on CentOS 6.5 (+ solution)
I've seen a similar problem (404 on webadmin) on my 4GB physical ovirt engine box, but it turned out to be the jboss timeout in deployment Relevant bz https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1078291
so it might have been that it wasn't a memory limit in the OP that caused failure, but the time it took jboss to deploy
-John
On Tue, May 13, 2014 at 12:31 PM, R P Herrold <herrold@owlriver.com> wrote:
On Tue, 13 May 2014, Sven Kieske wrote:
Doesn't this make you wonder where the minimum requirements come from?
If it runs with less than 1 GB RAM, why do the docs say you need 4 GB and recommend even 16 GB ?
certainly a fair question ... There is also a statement in that setup script as to needed filesystem space which seems to have been simply 'pulled out of the air', rather than documented / explained
Is it just a matter of scale(number of vms/hosts/DCs) ? What would make engine consume more RAM?
Can you maybe lower the minimum requirements?
Or isolate the recommendations to a flat file which is commented, and sourced by the script, so a person can discern the difference between 'hard' requirements, and simple 'recommendations' for a stated use case
-- Russ herrold _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Didi, Some background is that I first ran into this when I added dwh+reports to my 3.4 setup, and I found your bz then and did the fix (with changes below) and it worked for me. The fixed ovirt-engine.xml.in got overwritten when I upgraded to 3.4.1 and I reapplied . But I did change <deployment-scanner scan-interval="5000" path="$tempdir/deployments"/ deployment-timeout="1200"> to <deployment-scanner scan-interval="5000" path="$tempdir/deployments" deployment-timeout="1200"/> i.e. the / looks to be misplaced and should be used to close the deployment-scanner tag.
If you do, please report - what exact versions (OS, jboss, ovirt, etc) you use fedora 19 w/ latest updates kernel-3.13.11-100.fc19.x86_64 ovirt-engine-3.4.1-1.fc19.noarch jboss-as-7.1.1-21.fc19.noarch ovirt-engine-reports-3.4.1-1.fc19.noarch
and (according to the logs) how long did the deployment take. Thanks! My last startup shows that was right at the 59 second mark. So you can see I'm right on the edge of it failing with default timeout of 60 seconds.
from server.log 2014-05-12 23:22:00,463 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) JBAS015876: Starting deployment of "engine.ear" 2014-05-12 23:22:00,464 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015876: Starting deployment of "ovirt-engine-reports.war" ... 2014-05-12 23:22:59,383 INFO [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS018559: Deployed "engine.ear" 2014-05-12 23:22:59,385 INFO [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS018559: Deployed "ovirt-engine-reports.war" Thanks, -John
----- Original Message -----
From: "R P Herrold" <herrold@owlriver.com> To: "Sven Kieske" <S.Kieske@mittwald.de> Cc: users@ovirt.org Sent: Tuesday, May 13, 2014 7:31:14 PM Subject: [ovirt-users] getting 404 after fresh install of oVirt 3.4 on CentOS 6.5 (+ solution)
On Tue, 13 May 2014, Sven Kieske wrote:
Doesn't this make you wonder where the minimum requirements come from?
If it runs with less than 1 GB RAM, why do the docs say you need 4 GB and recommend even 16 GB ?
certainly a fair question ... There is also a statement in that setup script as to needed filesystem space which seems to have been simply 'pulled out of the air', rather than documented / explained
I searched a bit and can't find a sizing guide etc. for the engine. You can find stuff for the hosts if you search a bit. My current guess is that 1. It's based on anecdotal real-world use 2. It's meant to prevent people from wasting time on not-enough-memory (and disk space) issues etc. 3. In practice, people that use ovirt for more than a minimal setup, will have to have some nice hardware for the hosts, and so dedicating part of that to the engine is not a big issue. Especially with hosted-engine where you do not need a dedicated physical machine. No-one prevents anyone from doing some research and publishing the results, you know - e.g. a table showing "An engine managing X VMs on Y hosts used such-and-such disk space over the first day/week/month/year of use, and had this-and-that average response time (or something more complex) when running with such-and-such RAM". If, based on that, you think we can/should provide more info regarding minimal/recommended RAM/disk for specific use cases, you are then welcome to update the wiki. Patches to setup are welcome too :-) Note that current limitations are never failing setup - they are always just warnings. You are welcome to ignore them (and feed that to your answer files if you run setup repeatedly).
Is it just a matter of scale(number of vms/hosts/DCs) ? What would make engine consume more RAM?
Can you maybe lower the minimum requirements?
Or isolate the recommendations to a flat file which is commented, and sourced by the script, so a person can discern the difference between 'hard' requirements, and simple 'recommendations' for a stated use case
Not sure I completely got you here, but if you meant to a file summarizing the results of such a hypothetical research, then I think that would be great. Best, -- Didi
On Wed, 14 May 2014, Yedidyah Bar David wrote:
herrold: certainly a fair question ... There is also a statement in that setup script as to needed filesystem space which seems to have been simply 'pulled out of the air', rather than documented / explained
I searched a bit and can't find a sizing guide etc. for the engine. You can find stuff for the hosts if you search a bit.
My current guess is that 1. It's based on anecdotal real-world use
The drive space requirement particularly is sized for a locally hosted (on the engine) filestore. Decoupling this seems like an obvious place for improvement
2. It's meant to prevent people from wasting time on not-enough-memory (and disk space) issues etc.
As to ram, the sheer size sought [16 G 'recommended' with a 4G 'minimum'] initially made me quite hesitant to set up, as it meant dedicating a significant host overon the testing bench to the project. It is seemingly running without problem in a 'minimum' of 2G here
3. In practice, people that use ovirt for more than a minimal setup, will have to have some nice hardware for the hosts, and so dedicating part of that to the engine is not a big issue. Especially with hosted-engine where you do not need a dedicated physical machine.
But this is certainly not the message being communicated by the warning. Getting more 'mass' of external testers is always a goal. Having 'drop in testers' for release candidates along the way -- who would set up a minimal candidate, and then tear it down at the end of the process -- is even more valuable,
No-one prevents anyone from doing some research and publishing the results, you know - e.g. a table showing "An engine managing X VMs on Y hosts used such-and-such disk space over the first day/week/month/year of use, and had this-and-that average response time (or something more complex) when running with such-and-such RAM". If, based on that, you think we can/should provide more info regarding minimal/recommended RAM/disk for specific use cases, you are then welcome to update the wiki. Patches to setup are welcome too :-)
Indeed, my personal hope was to get an automated puppet and buildbot 'end consumer of ovirt' setup working so I could 'chase bleeding edge', but I have not been able to attain this yet. It may be that other tools (Foreman ?) need to be explored by me
Note that current limitations are never failing setup - they are always just warnings. You are welcome to ignore them (and feed that to your answer files if you run setup repeatedly).
The manual nature of that script is also an issue, and so 'sourcing' a response file, and only asking unanswered questions, is also on my docket to write -- Russ herrold
participants (5)
-
John Taylor -
R P Herrold -
Steven Van Acker -
Sven Kieske -
Yedidyah Bar David