[Users] Problem in 3.3 with shmmax configuration
Hi, I'm setting up Engine for the 2nd time - the first time I answered a configuration question wrong. So I did: engine-setup engine-cleanup engine-setup Things worked, until I rebooted the system. I found that postgresql would not startup, and was failing with "could not create shared memory segment: Invalid Argument". I resolved this issue by creating a file /etc/sysctl.d/10-shmmax.conf, containing the line: kernel.shmmax = 1000000000 (I read somewhere that postgresql recommends setting shmmax to 1/4 of physical memory, and I have 4GB) 1. Is this a known bug? If not, should I file one? If so, how do I do that? :) 2. Is there a better fix than the one I settled on? Does the normal configuration wind up increasing shmmax, or reducing postgresql's limits? What are the default values for this in a normal engine configuration? Thanks, Bob
Il 02/11/2013 17:51, Bob Doolittle ha scritto:
Hi,
I'm setting up Engine for the 2nd time - the first time I answered a configuration question wrong. So I did:
engine-setup engine-cleanup engine-setup
Things worked, until I rebooted the system. I found that postgresql would not startup, and was failing with "could not create shared memory segment: Invalid Argument".
I resolved this issue by creating a file /etc/sysctl.d/10-shmmax.conf, containing the line: kernel.shmmax = 1000000000
(I read somewhere that postgresql recommends setting shmmax to 1/4 of physical memory, and I have 4GB)
1. Is this a known bug? If not, should I file one? If so, how do I do that? :)
Which version are you installing? Can you please attach all 3 logs from above sequence (setup, cleanup, setup)? I think something may have gone wrong on second setup execution while setting shmmax.
2. Is there a better fix than the one I settled on? Does the normal configuration wind up increasing shmmax, or reducing postgresql's limits? What are the default values for this in a normal engine configuration?
No better fix, engine-setup just do something like that, setting shmmax to 35554432.
Thanks, Bob
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
On 11/04/2013 05:43 AM, Sandro Bonazzola wrote:
Il 02/11/2013 17:51, Bob Doolittle ha scritto:
Hi,
I'm setting up Engine for the 2nd time - the first time I answered a configuration question wrong. So I did:
engine-setup engine-cleanup engine-setup
Things worked, until I rebooted the system. I found that postgresql would not startup, and was failing with "could not create shared memory segment: Invalid Argument".
I resolved this issue by creating a file /etc/sysctl.d/10-shmmax.conf, containing the line: kernel.shmmax = 1000000000
(I read somewhere that postgresql recommends setting shmmax to 1/4 of physical memory, and I have 4GB)
1. Is this a known bug? If not, should I file one? If so, how do I do that? :) Which version are you installing?
3.3, on Fedora 19.
Can you please attach all 3 logs from above sequence (setup, cleanup, setup)? I think something may have gone wrong on second setup execution while setting shmmax.
Unfortunately I no longer have those logs. *However* last night I powered down my node and engine. Last night engine would at least come up. Now when I boot up my engine I get the same error. So my guess is that the shmmax setting isn't being configured in a persistent fashion somehow, or is somehow reverting. My most recent setup log is here (I hate to send large logs to lists): https://dl.dropboxusercontent.com/u/35965416/ovirt-engine-setup-201311031416... -Bob
2. Is there a better fix than the one I settled on? Does the normal configuration wind up increasing shmmax, or reducing postgresql's limits? What are the default values for this in a normal engine configuration? No better fix, engine-setup just do something like that, setting shmmax to 35554432.
Thanks, Bob
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Il 04/11/2013 17:17, Bob Doolittle ha scritto:
On 11/04/2013 05:43 AM, Sandro Bonazzola wrote:
Il 02/11/2013 17:51, Bob Doolittle ha scritto:
Hi,
I'm setting up Engine for the 2nd time - the first time I answered a configuration question wrong. So I did:
engine-setup engine-cleanup engine-setup
Things worked, until I rebooted the system. I found that postgresql would not startup, and was failing with "could not create shared memory segment: Invalid Argument".
I resolved this issue by creating a file /etc/sysctl.d/10-shmmax.conf, containing the line: kernel.shmmax = 1000000000
(I read somewhere that postgresql recommends setting shmmax to 1/4 of physical memory, and I have 4GB)
1. Is this a known bug? If not, should I file one? If so, how do I do that? :) Which version are you installing?
3.3, on Fedora 19.
Can you please attach all 3 logs from above sequence (setup, cleanup, setup)? I think something may have gone wrong on second setup execution while setting shmmax.
Unfortunately I no longer have those logs.
*However* last night I powered down my node and engine. Last night engine would at least come up.
Now when I boot up my engine I get the same error. So my guess is that the shmmax setting isn't being configured in a persistent fashion somehow, or is somehow reverting.
My most recent setup log is here (I hate to send large logs to lists): https://dl.dropboxusercontent.com/u/35965416/ovirt-engine-setup-201311031416...
2013-11-03 14:18:18 DEBUG otopi.plugins.ovirt_engine_setup.system.sysctl plugin.execute:441 execute-output: ('/sbin/sysctl', '-n', 'kernel.shmmax') stdout: 35554432 setup detected that your shmmax is already configured with a good value so it didn't change the configuration. If you set above value before running setup, please retry running setup without setting it or setting it to a value less than 35554432. setup will detect it's too low and will create needed configuration files for fixing it.
-Bob
2. Is there a better fix than the one I settled on? Does the normal configuration wind up increasing shmmax, or reducing postgresql's limits? What are the default values for this in a normal engine configuration? No better fix, engine-setup just do something like that, setting shmmax to 35554432.
Thanks, Bob
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
On 11/05/2013 02:13 AM, Sandro Bonazzola wrote:
Il 04/11/2013 17:17, Bob Doolittle ha scritto:
On 11/04/2013 05:43 AM, Sandro Bonazzola wrote:
Il 02/11/2013 17:51, Bob Doolittle ha scritto:
Hi,
I'm setting up Engine for the 2nd time - the first time I answered a configuration question wrong. So I did:
engine-setup engine-cleanup engine-setup
Things worked, until I rebooted the system. I found that postgresql would not startup, and was failing with "could not create shared memory segment: Invalid Argument".
I resolved this issue by creating a file /etc/sysctl.d/10-shmmax.conf, containing the line: kernel.shmmax = 1000000000
(I read somewhere that postgresql recommends setting shmmax to 1/4 of physical memory, and I have 4GB)
1. Is this a known bug? If not, should I file one? If so, how do I do that? :) Which version are you installing? 3.3, on Fedora 19.
Can you please attach all 3 logs from above sequence (setup, cleanup, setup)? I think something may have gone wrong on second setup execution while setting shmmax. Unfortunately I no longer have those logs.
*However* last night I powered down my node and engine. Last night engine would at least come up.
Now when I boot up my engine I get the same error. So my guess is that the shmmax setting isn't being configured in a persistent fashion somehow, or is somehow reverting.
My most recent setup log is here (I hate to send large logs to lists): https://dl.dropboxusercontent.com/u/35965416/ovirt-engine-setup-201311031416... 2013-11-03 14:18:18 DEBUG otopi.plugins.ovirt_engine_setup.system.sysctl plugin.execute:441 execute-output: ('/sbin/sysctl', '-n', 'kernel.shmmax') stdout: 35554432
setup detected that your shmmax is already configured with a good value so it didn't change the configuration. If you set above value before running setup, please retry running setup without setting it or setting it to a value less than 35554432. setup will detect it's too low and will create needed configuration files for fixing it.
Except that when the system booted postgresql failed to come up because it failed to allocate the shm segment it wanted, and when I checked /proc/sys/kernel/shmmax the value was much lower than that. I have again set the value in syscfg and things are working now. So my guess is that something is reverting the value after engine-setup has set it, or else somehow it was not set in a persistent fashion by engine-setup. Where is the shmmax value stored by engine-setup? Should there be a file in /etc/syscfg.d? There was not. -Bob
Il 05/11/2013 15:13, Bob Doolittle ha scritto:
On 11/05/2013 02:13 AM, Sandro Bonazzola wrote:
Il 04/11/2013 17:17, Bob Doolittle ha scritto:
On 11/04/2013 05:43 AM, Sandro Bonazzola wrote:
Il 02/11/2013 17:51, Bob Doolittle ha scritto:
Hi,
I'm setting up Engine for the 2nd time - the first time I answered a configuration question wrong. So I did:
engine-setup engine-cleanup engine-setup
Things worked, until I rebooted the system. I found that postgresql would not startup, and was failing with "could not create shared memory segment: Invalid Argument".
I resolved this issue by creating a file /etc/sysctl.d/10-shmmax.conf, containing the line: kernel.shmmax = 1000000000
(I read somewhere that postgresql recommends setting shmmax to 1/4 of physical memory, and I have 4GB)
1. Is this a known bug? If not, should I file one? If so, how do I do that? :) Which version are you installing? 3.3, on Fedora 19.
Can you please attach all 3 logs from above sequence (setup, cleanup, setup)? I think something may have gone wrong on second setup execution while setting shmmax. Unfortunately I no longer have those logs.
*However* last night I powered down my node and engine. Last night engine would at least come up.
Now when I boot up my engine I get the same error. So my guess is that the shmmax setting isn't being configured in a persistent fashion somehow, or is somehow reverting.
My most recent setup log is here (I hate to send large logs to lists): https://dl.dropboxusercontent.com/u/35965416/ovirt-engine-setup-201311031416... 2013-11-03 14:18:18 DEBUG otopi.plugins.ovirt_engine_setup.system.sysctl plugin.execute:441 execute-output: ('/sbin/sysctl', '-n', 'kernel.shmmax') stdout: 35554432
setup detected that your shmmax is already configured with a good value so it didn't change the configuration. If you set above value before running setup, please retry running setup without setting it or setting it to a value less than 35554432. setup will detect it's too low and will create needed configuration files for fixing it.
Except that when the system booted postgresql failed to come up because it failed to allocate the shm segment it wanted, and when I checked /proc/sys/kernel/shmmax the value was much lower than that. I have again set the value in syscfg and things are working now.
So my guess is that something is reverting the value after engine-setup has set it, or else somehow it was not set in a persistent fashion by engine-setup. Where is the shmmax value stored by engine-setup? Should there be a file in /etc/syscfg.d? There was not.
the file created by engine-setup is /etc/sysctl.d/ovirt-engine.conf. Note that the file is created by engine-setup only if shmmax is < 35554432 when it's running. You can run # sysctl --system and see which files are loaded: # sysctl --system * Applying /usr/lib/sysctl.d/00-system.conf ... net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-arptables = 0 * Applying /usr/lib/sysctl.d/libvirtd.conf ... fs.aio-max-nr = 1048576 * Applying /usr/lib/sysctl.d/nepomuk-inotify.conf ... fs.inotify.max_user_watches = 524288 * Applying /etc/sysctl.d/ovirt-engine.conf ... kernel.shmmax = 35554432 * Applying /etc/sysctl.conf ...
-Bob
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
On 11/05/2013 10:17 AM, Sandro Bonazzola wrote:
the file created by engine-setup is /etc/sysctl.d/ovirt-engine.conf. Note that the file is created by engine-setup only if shmmax is < 35554432 when it's running.
Hmmm, I think this explains the initial problem I saw, after doing setup; cleanup; setup The first setup will bump up the shmmax. The cleanup will remove ovirt-engine.conf. However if you don't reboot, shmmax remains boosted. The second setup will detect that shmmax is fine, and not create ovirt-engine.conf. A subsequent reboot will revert shmmax to the default value, and postgresql will fail to come up. -Bob
This boils down to "Current runtime state" != "Current configuration", but you are basing configuration decisions on the runtime state. These situations are always thorny to deal with. -Bob On 11/05/2013 10:46 AM, Bob Doolittle wrote:
On 11/05/2013 10:17 AM, Sandro Bonazzola wrote:
the file created by engine-setup is /etc/sysctl.d/ovirt-engine.conf. Note that the file is created by engine-setup only if shmmax is < 35554432 when it's running.
Hmmm, I think this explains the initial problem I saw, after doing setup; cleanup; setup
The first setup will bump up the shmmax. The cleanup will remove ovirt-engine.conf. However if you don't reboot, shmmax remains boosted. The second setup will detect that shmmax is fine, and not create ovirt-engine.conf.
A subsequent reboot will revert shmmax to the default value, and postgresql will fail to come up.
-Bob
This boils down to "Current runtime state" !=3D "Current configuration",= =0A= but you are basing configuration decisions on the runtime state. These=0A= situations are always thorny to deal with.=0A= =0A= -Bob=0A= =0A= This should be worth a Redhat Bugzilla entry. From my experience=0A=
This is a multi-part message in MIME format. ------=_NextPartTM-000-2767a821-379f-4f27-8d4e-08c477cd899d Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable the developers of this "free" project take care about that. Nevertheless=0A= the first setup process still needs a lot of shaping. Additionally it=0A= is always a good idea to convince new users with smooth running=0A= installers. Especially when they expect a virtualization solution to=0A= be a stress-free layer. =0A= =0A= Best regards.=0A= =0A= Markus=0A= =0A= ------=_NextPartTM-000-2767a821-379f-4f27-8d4e-08c477cd899d Content-Type: text/plain; name="InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="InterScan_Disclaimer.txt" **************************************************************************** Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Über das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=_NextPartTM-000-2767a821-379f-4f27-8d4e-08c477cd899d--
Il 05/11/2013 16:46, Bob Doolittle ha scritto:
On 11/05/2013 10:17 AM, Sandro Bonazzola wrote:
the file created by engine-setup is /etc/sysctl.d/ovirt-engine.conf. Note that the file is created by engine-setup only if shmmax is < 35554432 when it's running.
Hmmm, I think this explains the initial problem I saw, after doing setup; cleanup; setup
The first setup will bump up the shmmax. The cleanup will remove ovirt-engine.conf. However if you don't reboot, shmmax remains boosted. The second setup will detect that shmmax is fine, and not create ovirt-engine.conf.
A subsequent reboot will revert shmmax to the default value, and postgresql will fail to come up.
can you open a bug about this?
-Bob
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
On 11/05/2013 11:10 AM, Sandro Bonazzola wrote:
Il 05/11/2013 16:46, Bob Doolittle ha scritto:
On 11/05/2013 10:17 AM, Sandro Bonazzola wrote:
the file created by engine-setup is /etc/sysctl.d/ovirt-engine.conf. Note that the file is created by engine-setup only if shmmax is < 35554432 when it's running. Hmmm, I think this explains the initial problem I saw, after doing setup; cleanup; setup
The first setup will bump up the shmmax. The cleanup will remove ovirt-engine.conf. However if you don't reboot, shmmax remains boosted. The second setup will detect that shmmax is fine, and not create ovirt-engine.conf.
A subsequent reboot will revert shmmax to the default value, and postgresql will fail to come up. can you open a bug about this?
Done: https://bugzilla.redhat.com/show_bug.cgi?id=1026926 Thanks, Bob
participants (4)
-
Bob Doolittle -
Bob Doolittle -
Markus Stockhausen -
Sandro Bonazzola