[Users] I am using the CentOS build from the 12th. I the engine is dieing with the following error.

Wed Jun 20 06:43:53 UTC 2012

On 06/20/2012 01:36 AM, Itamar Heim wrote:
> On 06/20/2012 04:09 AM, Robert Middleswarth wrote:
>> On 06/18/2012 04:47 AM, Juan Hernandez wrote:
>>> On 06/18/2012 04:25 AM, Robert Middleswarth wrote:
>>>> 2012-06-17 05:21:24,682 ERROR
>>>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>>> (ajp--0.0.0.0-8009-9) Failed to decryptjava.io.FileNotFoundException:
>>>> /etc/pki/ovirt-engine/.keystore (Too many open files)
>>>> 2012-06-17 05:21:26,121 ERROR
>>>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>>> (QuartzScheduler_Worker-88) Failed to
>>>> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
>>>> (Too many open files)
>>>> 2012-06-17 05:21:26,130 ERROR
>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
>>>> (QuartzScheduler_Worker-88) XML RPC error in command 
>>>> GetCapabilitiesVDS
>>>> ( Vds: ann100ovirt01 ), the error was:
>>>> java.util.concurrent.ExecutionException:
>>>> java.lang.reflect.InvocationTargetException, SocketException: Too many
>>>> open files
>>>> 2012-06-17 15:49:08,577 ERROR
>>>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>>> (QuartzScheduler_Worker-28) Failed to
>>>> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
>>>> (Too many open files)
>>>> 2012-06-17 20:21:13,943 ERROR
>>>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>>> (QuartzScheduler_Worker-64) Failed to
>>>> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
>>>> (Too many open files)
>>>> 2012-06-17 20:21:15,949 ERROR
>>>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>>> (QuartzScheduler_Worker-65) Failed to
>>>> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
>>>> (Too many open files)
>>>> 2012-06-17 21:00:15,133 ERROR
>>>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>>> (QuartzScheduler_Worker-98) Failed to
>>>> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
>>>> (Too many open files)
>>>> 2012-06-17 21:07:00,037 WARN [org.ovirt.engine.core.ServletUtils]
>>>> (ajp--0.0.0.0-8009-5) File "/usr/share/vdsm-bootstrap/deployUtil.py is
>>>> 47867 bytes long. Please reconsider using this servlet for files 
>>>> larger
>>>> than 8192 bytes.
>>>> 2012-06-17 21:07:00,115 WARN [org.ovirt.engine.core.ServletUtils]
>>>> (ajp--0.0.0.0-8009-6) File "/usr/share/vdsm-bootstrap/vds_bootstrap.py
>>>> is 33884 bytes long. Please reconsider using this servlet for files
>>>> larger than 8192 bytes.
>>>> 2012-06-17 21:32:10,686 WARN [org.ovirt.engine.core.ServletUtils]
>>>> (ajp--0.0.0.0-8009-6) File "/usr/share/vdsm-bootstrap/deployUtil.py is
>>>> 47867 bytes long. Please reconsider using this servlet for files 
>>>> larger
>>>> than 8192 bytes.
>>>> 2012-06-17 21:32:10,752 WARN [org.ovirt.engine.core.ServletUtils]
>>>> (ajp--0.0.0.0-8009-1) File "/usr/share/vdsm-bootstrap/vds_bootstrap.py
>>>> is 33884 bytes long. Please reconsider using this servlet for files
>>>> larger than 8192 bytes.
>>>> 2012-06-17 21:44:34,028 ERROR
>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
>>>> (QuartzScheduler_Worker-23) XML RPC error in command ListVDS ( Vds:
>>>> ann100ovirt01 ), the error was: 
>>>> java.util.concurrent.ExecutionException:
>>>> java.lang.reflect.InvocationTargetException, SocketException: Too many
>>>> open files
>>>> 2012-06-17 21:44:36,034 ERROR
>>>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>>> (QuartzScheduler_Worker-26) Failed to
>>>> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
>>>> (Too many open files)
>>>> 2012-06-17 21:44:36,045 ERROR
>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
>>>> (QuartzScheduler_Worker-26) XML RPC error in command 
>>>> GetCapabilitiesVDS
>>>> ( Vds: ann100ovirt01 ), the error was:
>>>> java.util.concurrent.ExecutionException:
>>>> java.lang.reflect.InvocationTargetException, SocketException: Too many
>>>> open files
>>>>
>>>>
>>>> I restarted ovirt-engine and ran lsof -p 3557
>>>>
>>>> I see hundreds of the same open file.
>>>>
>>>> java 3557 ovirt 839r REG 253,0 3230 7865608
>>>> /etc/pki/ovirt-engine/.keystore
>>>> java 3557 ovirt 840r REG 253,0 3230 7865608
>>>> /etc/pki/ovirt-engine/.keystore
>>>> java 3557 ovirt 841r REG 253,0 3230 7865608
>>>> /etc/pki/ovirt-engine/.keystore
>>>> java 3557 ovirt 842r REG 253,0 3230 7865608
>>>> /etc/pki/ovirt-engine/.keystore
>>>> java 3557 ovirt 843r REG 253,0 3230 7865608
>>>> /etc/pki/ovirt-engine/.keystore
>>>> java 3557 ovirt 844r REG 253,0 3230 7865608
>>>> /etc/pki/ovirt-engine/.keystore
>>>> java 3557 ovirt 845r REG 253,0 3230 7865608
>>>> /etc/pki/ovirt-engine/.keystore
>>>> java 3557 ovirt 846r REG 253,0 3230 7865608
>>>> /etc/pki/ovirt-engine/.keystore
>>>> java 3557 ovirt 847r REG 253,0 3230 7865608
>>>> /etc/pki/ovirt-engine/.keystore
>>>> java 3557 ovirt 848r REG 253,0 3230 7865608
>>>> /etc/pki/ovirt-engine/.keystore
>>>> java 3557 ovirt 849r REG 253,0 3230 7865608
>>>> /etc/pki/ovirt-engine/.keystore
>>>> java 3557 ovirt 850r REG 253,0 3230 7865608
>>>> /etc/pki/ovirt-engine/.keystore
>>>> java 3557 ovirt 851r REG 253,0 3230 7865608
>>>> /etc/pki/ovirt-engine/.keystore
>>>> java 3557 ovirt 852r REG 253,0 3230 7865608
>>>> /etc/pki/ovirt-engine/.keystore
>>>> java 3557 ovirt 853r REG 253,0 3230 7865608
>>>> /etc/pki/ovirt-engine/.keystore
>>>> java 3557 ovirt 854r REG 253,0 3230 7865608
>>>> /etc/pki/ovirt-engine/.keystore
>>>>
>>>> Is anyone else seeing this?
>>>
>>> There are two issues that might be affecting you here:
>>>
>>> 1. We may have a leak of file descriptors on the logic that opens the
>>> keystore. Can you take a look at the following change? Can you apply
>>> it and let us know if it fixes your problem?
>>>
>>> http://gerrit.ovirt.org/#/c/5423
>> I have been using rpm's but am attempting to compile the code.
>>>
>>> 2. You may be having a problem with the file descriptors limit due to
>>> other issues. During setup we should increase the file descriptor
>>> limit from 1024 (the default) to 65535, but we do that for the "jboss"
>>> user instead of for "ovirt". The fix for this issue is in review:
>>>
>>> http://gerrit.ovirt.org/#/c/5375
>>>
>>> The workaround for this is to edit the /etc/security/limits.conf file
>>> and replace "jboss" with "ovirt", then restart the engine.
>> I made that change but the web interface still dies after about 15 min
>> or so with the same number of open files and the same errror. I rebooted
>> the system so I am sure the change should have take place.
>>
>
> did you try with this patch applied?
> http://gerrit.ovirt.org/#/c/5423/
Not yet.  Haven't been able to get a build to complete yet.  On my 3rd try.

Thanks
Robert