[Users] I am using the CentOS build from the 12th. I the engine is dieing with the following error.

[Itamar Heim]

On 06/21/2012 09:47 PM, Juan Hernandez wrote:
> On 06/21/2012 08:25 PM, Itamar Heim wrote:
>> On 06/21/2012 08:20 PM, Robert Middleswarth wrote:
>>>> On 06/18/2012 04:25 AM, Robert Middleswarth wrote:
>>>>> 2012-06-17 05:21:24,682 ERROR
>>>>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>>>> (ajp--0.0.0.0-8009-9) Failed to decryptjava.io.FileNotFoundException:
>>>>> /etc/pki/ovirt-engine/.keystore (Too many open files)
>>>>> 2012-06-17 05:21:26,121 ERROR
>>>>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>>>> (QuartzScheduler_Worker-88) Failed to
>>>>> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
>>>>> (Too many open files)
>>>>> 2012-06-17 05:21:26,130 ERROR
>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
>>>>> (QuartzScheduler_Worker-88) XML RPC error in command
>>>>> GetCapabilitiesVDS
>>>>> ( Vds: ann100ovirt01 ), the error was:
>>>>> java.util.concurrent.ExecutionException:
>>>>> java.lang.reflect.InvocationTargetException, SocketException: Too many
>>>>> open files
>>>>> 2012-06-17 15:49:08,577 ERROR
>>>>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>>>> (QuartzScheduler_Worker-28) Failed to
>>>>> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
>>>>> (Too many open files)
>>>>> 2012-06-17 20:21:13,943 ERROR
>>>>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>>>> (QuartzScheduler_Worker-64) Failed to
>>>>> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
>>>>> (Too many open files)
>>>>> 2012-06-17 20:21:15,949 ERROR
>>>>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>>>> (QuartzScheduler_Worker-65) Failed to
>>>>> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
>>>>> (Too many open files)
>>>>> 2012-06-17 21:00:15,133 ERROR
>>>>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>>>> (QuartzScheduler_Worker-98) Failed to
>>>>> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
>>>>> (Too many open files)
>>>>> 2012-06-17 21:07:00,037 WARN [org.ovirt.engine.core.ServletUtils]
>>>>> (ajp--0.0.0.0-8009-5) File "/usr/share/vdsm-bootstrap/deployUtil.py is
>>>>> 47867 bytes long. Please reconsider using this servlet for files
>>>>> larger
>>>>> than 8192 bytes.
>>>>> 2012-06-17 21:07:00,115 WARN [org.ovirt.engine.core.ServletUtils]
>>>>> (ajp--0.0.0.0-8009-6) File "/usr/share/vdsm-bootstrap/vds_bootstrap.py
>>>>> is 33884 bytes long. Please reconsider using this servlet for files
>>>>> larger than 8192 bytes.
>>>>> 2012-06-17 21:32:10,686 WARN [org.ovirt.engine.core.ServletUtils]
>>>>> (ajp--0.0.0.0-8009-6) File "/usr/share/vdsm-bootstrap/deployUtil.py is
>>>>> 47867 bytes long. Please reconsider using this servlet for files
>>>>> larger
>>>>> than 8192 bytes.
>>>>> 2012-06-17 21:32:10,752 WARN [org.ovirt.engine.core.ServletUtils]
>>>>> (ajp--0.0.0.0-8009-1) File "/usr/share/vdsm-bootstrap/vds_bootstrap.py
>>>>> is 33884 bytes long. Please reconsider using this servlet for files
>>>>> larger than 8192 bytes.
>>>>> 2012-06-17 21:44:34,028 ERROR
>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
>>>>> (QuartzScheduler_Worker-23) XML RPC error in command ListVDS ( Vds:
>>>>> ann100ovirt01 ), the error was:
>>>>> java.util.concurrent.ExecutionException:
>>>>> java.lang.reflect.InvocationTargetException, SocketException: Too many
>>>>> open files
>>>>> 2012-06-17 21:44:36,034 ERROR
>>>>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>>>> (QuartzScheduler_Worker-26) Failed to
>>>>> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
>>>>> (Too many open files)
>>>>> 2012-06-17 21:44:36,045 ERROR
>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
>>>>> (QuartzScheduler_Worker-26) XML RPC error in command
>>>>> GetCapabilitiesVDS
>>>>> ( Vds: ann100ovirt01 ), the error was:
>>>>> java.util.concurrent.ExecutionException:
>>>>> java.lang.reflect.InvocationTargetException, SocketException: Too many
>>>>> open files
>>>>>
>>>>>
>>>>> I restarted ovirt-engine and ran lsof -p 3557
>>>>>
>>>>> I see hundreds of the same open file.
>>>>>
>>>>> java 3557 ovirt 839r REG 253,0 3230 7865608
>>>>> /etc/pki/ovirt-engine/.keystore
>>>>> java 3557 ovirt 840r REG 253,0 3230 7865608
>>>>> /etc/pki/ovirt-engine/.keystore
>>>>> java 3557 ovirt 841r REG 253,0 3230 7865608
>>>>> /etc/pki/ovirt-engine/.keystore
>>>>> java 3557 ovirt 842r REG 253,0 3230 7865608
>>>>> /etc/pki/ovirt-engine/.keystore
>>>>> java 3557 ovirt 843r REG 253,0 3230 7865608
>>>>> /etc/pki/ovirt-engine/.keystore
>>>>> java 3557 ovirt 844r REG 253,0 3230 7865608
>>>>> /etc/pki/ovirt-engine/.keystore
>>>>> java 3557 ovirt 845r REG 253,0 3230 7865608
>>>>> /etc/pki/ovirt-engine/.keystore
>>>>> java 3557 ovirt 846r REG 253,0 3230 7865608
>>>>> /etc/pki/ovirt-engine/.keystore
>>>>> java 3557 ovirt 847r REG 253,0 3230 7865608
>>>>> /etc/pki/ovirt-engine/.keystore
>>>>> java 3557 ovirt 848r REG 253,0 3230 7865608
>>>>> /etc/pki/ovirt-engine/.keystore
>>>>> java 3557 ovirt 849r REG 253,0 3230 7865608
>>>>> /etc/pki/ovirt-engine/.keystore
>>>>> java 3557 ovirt 850r REG 253,0 3230 7865608
>>>>> /etc/pki/ovirt-engine/.keystore
>>>>> java 3557 ovirt 851r REG 253,0 3230 7865608
>>>>> /etc/pki/ovirt-engine/.keystore
>>>>> java 3557 ovirt 852r REG 253,0 3230 7865608
>>>>> /etc/pki/ovirt-engine/.keystore
>>>>> java 3557 ovirt 853r REG 253,0 3230 7865608
>>>>> /etc/pki/ovirt-engine/.keystore
>>>>> java 3557 ovirt 854r REG 253,0 3230 7865608
>>>>> /etc/pki/ovirt-engine/.keystore
>>>>>
>>>>> Is anyone else seeing this?
>>>>
>>>> There are two issues that might be affecting you here:
>>>>
>>>> 1. We may have a leak of file descriptors on the logic that opens the
>>>> keystore. Can you take a look at the following change? Can you apply
>>>> it and let us know if it fixes your problem?
>>>>
>>>> http://gerrit.ovirt.org/#/c/5423
>>>>
>>> I am still tiring to get a working build environment setup. However
>>> dreyou at gmail.com added into his repo and my problem went away after the
>>> change was applied. I recommend that this fix gets cherry-picked into
>>> the 3.1 tree
>>
>> agree. is there an ovirt bug blocking the 3.1 blockers to track it?
>
> Here is the bug:
>
> https://bugzilla.redhat.com/834399
>
> Should it block "Tracker: oVirt 3.1 release"?

we can debate if it is a blocker or not, but i don't see a reason not to 
cherry pick it to the 3.1 branch for the build, hence to flag it for 
tracking under the blocker bug