[ovirt-users] hosted engine setup failed for 10 minutes delay.. engine seems alive
Simone Tiraboschi
stirabos at redhat.com
Mon May 2 09:14:59 UTC 2016
On Mon, May 2, 2016 at 11:06 AM, Yedidyah Bar David <didi at redhat.com> wrote:
> On Mon, May 2, 2016 at 11:48 AM, Gianluca Cecchi
> <gianluca.cecchi at gmail.com> wrote:
>> On Mon, May 2, 2016 at 9:58 AM, Simone Tiraboschi wrote:
>>>
>>>
>>>
>>> hosted-engine-setup creates a fresh VM and inject a cloud-init script
>>> to configure it and execute there engine-setup to configure the engine
>>> as needed.
>>> Since engine-setup is running on the engine VM triggered by
>>> cloud-init, hosted-engine-setup has no way to really control its
>>> process status so we simply gather its output with a timeout of 10
>>> minutes between each single output line.
>>> In nothing happens within 10 minutes (the value is easily
>>> customizable), hosted-engine-setup thinks that engine-setup is stuck.
>>
>>
>>
>> How can one customize the pre-set timeout?
To set 20 minutes you can pass this
OVEHOSTED_ENGINE/engineSetupTimeout=int:1200
>> Could it be better to ask the user at the end of timeout if he/she wants to
>> wait again, instead of directly fail?
>
> Perhaps, can you please open a bz?
+1
>>> So the issue we have to understood is why this simple command took
>>> more than 10 minutes in your env:
>>> 2016-04-30 17:56:57 DEBUG
>>> otopi.plugins.ovirt_engine_setup.ovirt_engine.config.aaajdbc
>>> plugin.executeRaw:828 execute: ('/usr/bin/ovirt-aaa-jdbc-tool',
>>> '--db-config=/etc/ovirt-engine/aaa/internal.properties', 'user',
>>> 'password-reset', 'admin', '--password=env:pass', '--force',
>>> '--password-valid-to=2216-03-13 17:56:57Z'), executable='None',
>>> cwd='None', env={'LANG': 'en_US.UTF-8', 'SHLVL': '1', 'PYTHONPATH':
>>> '/usr/share/ovirt-engine/setup/bin/..::', 'pass': '**FILTERED**',
>>> 'OVIRT_ENGINE_JAVA_HOME_FORCE': '1', 'PWD': '/',
>>> 'OVIRT_ENGINE_JAVA_HOME': u'/usr/lib/jvm/jre', 'PATH':
>>> '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'OTOPI_LOGFILE':
>>>
>>> '/var/log/ovirt-engine/setup/ovirt-engine-setup-20160430175551-dttt2p.log',
>>> 'OVIRT_JBOSS_HOME': '/usr/share/ovirt-engine-wildfly',
>>> 'OTOPI_EXECDIR': '/'}
>>
>>
>>
>>
>> It seemed quite strange to me too (see below further info on this)
>>
>>>
>>> Can you please check the entropy value on your host?
>>> cat /proc/sys/kernel/random/entropy_avail
>>>
>>
>> I have not at hand now the server. I'll check soon and report
>> Do you mean entropy of the physical server that will operate as hypervisor?
On the hypervisor
> That's a good question. Simone - do you know if we start the guest with
> virtio-rng?
AFAIK we are not.
> This is another case of [1], perhaps we should reopen it.
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1319827
>
>>
>>
>>>
>>> > As a last question how to clean up things in case I have to start from
>>> > scratch.
>>>
>>> I'd recommend to redeploy from scratch instead of trying fixing it
>>> but, before that, we need to understand the root issue.
>>>
>>
>> So, trying restart the setup with generated answer file I got:
>> 1) if VM still powered on, an error about this condition
>> 2) if VM powered down, an error abut storage domain already in place and
>> restart not supported in this condition.
>>
>> I was able to continue with these steps:
>>
>> a) remove what inside the partially setup self hosted engine storage domain
>> rm -rf /SHE_DOMAIN/*
>> cd SHE_DOMAIN
>> mklost+found
>>
>> b) reboot the hypervisor
>>
>> c) stop vdsmd
>>
>> d) start the setup again with the answer file
>> It seems all went well and this time strangely the step that took more than
>> 10 minutes before lasted less than 2 seconds....
>>
>> I was then able to deploy storage and iso domains without problems and self
>> hosted engine domain correctly detected and imported too.
>> Created two CentOS VMs without problems (6.7 and 7.2).
>>
>> See below the full output of deploy command
>>
>>
>> [root at ovirt01 ~]# hosted-engine --deploy
>> --config-append=/var/lib/ovirt-hosted-engine-setup/answers/answers-20160430200654.conf
>> [ INFO ] Stage: Initializing
>> [ INFO ] Generating a temporary VNC password.
>> [ INFO ] Stage: Environment setup
>> Configuration files:
>> ['/var/lib/ovirt-hosted-engine-setup/answers/answers-20160430200654.conf']
>> Log file:
>> /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20160501014326-8frbxk.log
>> Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos)
>> [ INFO ] Hardware supports virtualization
>> [ INFO ] Bridge ovirtmgmt already created
>> [ INFO ] Stage: Environment packages setup
>> [ INFO ] Stage: Programs detection
>> [ INFO ] Stage: Environment setup
>> [ INFO ] Stage: Environment customization
>>
>> --== STORAGE CONFIGURATION ==--
>>
>> During customization use CTRL-D to abort.
>> [ INFO ] Installing on first host
>>
>> --== SYSTEM CONFIGURATION ==--
>>
>>
>> --== NETWORK CONFIGURATION ==--
>>
>>
>> --== VM CONFIGURATION ==--
>>
>> [ INFO ] Checking OVF archive content (could take a few minutes depending
>> on archive size)
>> [ INFO ] Checking OVF XML content (could take a few minutes depending on
>> archive size)
>> [WARNING] OVF does not contain a valid image description, using default.
>> Enter root password that will be used for the engine appliance
>> (leave it empty to skip):
>> Confirm appliance root password:
>> [ INFO ] The engine VM will be configured to use 192.168.1.212/24
>> The following CPU types are supported by this host:
>> - model_Broadwell-noTSX: Intel Broadwell-noTSX Family
>> - model_Haswell-noTSX: Intel Haswell-noTSX Family
>> - model_SandyBridge: Intel SandyBridge Family
>> - model_Westmere: Intel Westmere Family
>> - model_Nehalem: Intel Nehalem Family
>> - model_Penryn: Intel Penryn Family
>> - model_Conroe: Intel Conroe Family
>> [WARNING] Minimum requirements for CPUs not met
>> [WARNING] Minimum requirements for disk size not met
>>
>> --== HOSTED ENGINE CONFIGURATION ==--
>>
>> Enter the name which will be used to identify this host inside the
>> Administrator Portal [hosted_engine_1]:
>> Enter 'admin at internal' user password that will be used for
>> accessing the Administrator Portal:
>> Confirm 'admin at internal' user password:
>> [ INFO ] Stage: Setup validation
>>
>> --== CONFIGURATION PREVIEW ==--
>>
>> Bridge interface : eth0
>> Engine FQDN : ovirt.lutwyn.org
>> Bridge name : ovirtmgmt
>> Host address : ovirt01.lutwyn.org
>> SSH daemon port : 22
>> Firewall manager : iptables
>> Gateway address : 192.168.1.1
>> Host name for web application : hosted_engine_1
>> Host ID : 1
>> Image size GB : 10
>> GlusterFS Share Name : hosted_engine_glusterfs
>> GlusterFS Brick Provisioning : False
>> Storage connection :
>> ovirt01.lutwyn.org:/SHE_DOMAIN
>> Console type : vnc
>> Memory size MB : 8192
>> MAC address : 00:16:3e:2e:93:15
>> Boot type : disk
>> Number of CPUs : 1
>> OVF archive (for disk boot) :
>> /usr/share/ovirt-engine-appliance/ovirt-engine-appliance-3.6-20160420.1.el7.centos.ova
>> Restart engine VM after engine-setup: True
>> CPU Type : model_Broadwell-noTSX
>> [ INFO ] Stage: Transaction setup
>> [ INFO ] Stage: Misc configuration
>> [ INFO ] Stage: Package installation
>> [ INFO ] Stage: Misc configuration
>> [ INFO ] Configuring libvirt
>> [ INFO ] Configuring VDSM
>> [ INFO ] Starting vdsmd
>> [ INFO ] Waiting for VDSM hardware info
>> [ INFO ] Creating Storage Domain
>> [ INFO ] Creating Storage Pool
>> [ INFO ] Connecting Storage Pool
>> [ INFO ] Verifying sanlock lockspace initialization
>> [ INFO ] Creating Image for 'hosted-engine.lockspace' ...
>> [ INFO ] Image for 'hosted-engine.lockspace' created successfully
>> [ INFO ] Creating Image for 'hosted-engine.metadata' ...
>> [ INFO ] Image for 'hosted-engine.metadata' created successfully
>> [ INFO ] Creating VM Image
>> [ INFO ] Extracting disk image from OVF archive (could take a few minutes
>> depending on archive size)
>> [ INFO ] Validating pre-allocated volume size
>> [ INFO ] Uploading volume to data domain (could take a few minutes
>> depending on archive size)
>> [ INFO ] Image successfully imported from OVF
>> [ INFO ] Destroying Storage Pool
>> [ INFO ] Start monitoring domain
>> [ INFO ] Configuring VM
>> [ INFO ] Updating hosted-engine configuration
>> [ INFO ] Stage: Transaction commit
>> [ INFO ] Stage: Closing up
>> [ INFO ] Creating VM
>> You can now connect to the VM with the following command:
>> /bin/remote-viewer vnc://localhost:5900
>> Use temporary password "4699ouyH" to connect to vnc console.
>> Please note that in order to use remote-viewer you need to be able
>> to run graphical applications.
>> This means that if you are using ssh you have to supply the -Y
>> flag (enables trusted X11 forwarding).
>> Otherwise you can run the command from a terminal in your
>> preferred desktop environment.
>> If you cannot run graphical applications you can connect to the
>> graphic console from another host or connect to the serial console using the
>> following command:
>> socat
>> UNIX-CONNECT:/var/run/ovirt-vmconsole-console/4dae6016-ff01-4a1f-87ea-09f049d0e58c.sock,user=ovirt-vmconsole
>> STDIO,raw,echo=0,escape=1
>> Please ensure that your Guest OS is properly configured to support
>> serial console according to your distro documentation.
>> Follow
>> http://www.ovirt.org/Serial_Console_Setup#I_need_to_access_the_console_the_old_way
>> for more info.
>> If you need to reboot the VM you will need to start it manually
>> using the command:
>> hosted-engine --vm-start
>> You can then set a temporary password using the command:
>> hosted-engine --add-console-password
>> [ INFO ] Running engine-setup on the appliance
>> |- [ INFO ] Stage: Initializing
>> |- [ INFO ] Stage: Environment setup
>> |- Configuration files:
>> ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf',
>> '/etc/ovirt-engine-setup.conf.d/10-packaging.conf',
>> '/root/ovirt-engine-answers', '/root/heanswers.conf']
>> |- Log file:
>> /var/log/ovirt-engine/setup/ovirt-engine-setup-20160430234733-h0rns4.log
>> |- Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos)
>> |- [ INFO ] Stage: Environment packages setup
>> |- [ INFO ] Stage: Programs detection
>> |- [ INFO ] Stage: Environment setup
>> |- [ INFO ] Stage: Environment customization
>> |-
>> |- --== PRODUCT OPTIONS ==--
>> |-
>> |-
>> |- --== PACKAGES ==--
>> |-
>> |-
>> |- --== ALL IN ONE CONFIGURATION ==--
>> |-
>> |-
>> |- --== NETWORK CONFIGURATION ==--
>> |-
>> |- [ INFO ] firewalld will be configured as firewall manager.
>> |-
>> |- --== DATABASE CONFIGURATION ==--
>> |-
>> |-
>> |- --== OVIRT ENGINE CONFIGURATION ==--
>> |-
>> |-
>> |- --== STORAGE CONFIGURATION ==--
>> |-
>> |-
>> |- --== PKI CONFIGURATION ==--
>> |-
>> |-
>> |- --== APACHE CONFIGURATION ==--
>> |-
>> |-
>> |- --== SYSTEM CONFIGURATION ==--
>> |-
>> |-
>> |- --== MISC CONFIGURATION ==--
>> |-
>> |-
>> |- --== END OF CONFIGURATION ==--
>> |-
>> |- [ INFO ] Stage: Setup validation
>> |- [WARNING] Cannot validate host name settings, reason: resolved
>> host does not match any of the local addresses
>> |- [WARNING] Less than 16384MB of memory is available
>> |-
>> |- --== CONFIGURATION PREVIEW ==--
>> |-
>> |- Application mode : virt
>> |- Default SAN wipe after delete : False
>> |- Firewall manager : firewalld
>> |- Update Firewall : True
>> |- Host FQDN :
>> ovirt.lutwyn.org
>> |- Engine database secured connection : False
>> |- Engine database host : localhost
>> |- Engine database user name : engine
>> |- Engine database name : engine
>> |- Engine database port : 5432
>> |- Engine database host name validation : False
>> |- Engine installation : True
>> |- PKI organization : lutwyn.org
>> |- Configure local Engine database : True
>> |- Set application as default page : True
>> |- Configure Apache SSL : True
>> |- Configure VMConsole Proxy : True
>> |- Engine Host FQDN :
>> ovirt.lutwyn.org
>> |- Configure WebSocket Proxy : True
>> |- [ INFO ] Stage: Transaction setup
>> |- [ INFO ] Stopping engine service
>> |- [ INFO ] Stopping ovirt-fence-kdump-listener service
>> |- [ INFO ] Stopping websocket-proxy service
>> |- [ INFO ] Stage: Misc configuration
>> |- [ INFO ] Stage: Package installation
>> |- [ INFO ] Stage: Misc configuration
>> |- [ INFO ] Initializing PostgreSQL
>> |- [ INFO ] Creating PostgreSQL 'engine' database
>> |- [ INFO ] Configuring PostgreSQL
>> |- [ INFO ] Creating/refreshing Engine database schema
>> |- [ INFO ] Creating/refreshing Engine 'internal' domain database
>> schema
>> |- [ INFO ] Upgrading CA
>> |- [ INFO ] Creating CA
>> |- [ INFO ] Setting up ovirt-vmconsole proxy helper PKI artifacts
>> |- [ INFO ] Setting up ovirt-vmconsole SSH PKI artifacts
>> |- [ INFO ] Configuring WebSocket Proxy
>> |- [ INFO ] Generating post install configuration file
>> '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
>> |- [ INFO ] Stage: Transaction commit
>> |- [ INFO ] Stage: Closing up
>> |-
>> |- --== SUMMARY ==--
>> |-
>> |- [WARNING] Less than 16384MB of memory is available
>> |- SSH fingerprint:
>> 13:dc:47:b9:38:66:15:eb:83:e8:ed:6f:71:cb:d4:ea
>> |- Internal CA
>> D8:4B:4C:74:45:EB:87:C4:8F:FE:DE:02:DF:99:5E:CB:AA:1B:D84
>> |- Note! If you want to gather statistical information
>> you can install Reports and/or DWH:
>> |- http://www.ovirt.org/Ovirt_DWH
>> |- http://www.ovirt.org/Ovirt_Reports
>> |- Web access is enabled at:
>> |- http://ovirt.lutwyn.org:80/ovirt-engine
>> |- https://ovirt.lutwyn.org:443/ovirt-engine
>> |- Please use the user 'admin at internal' and password
>> specified in order to login
>> |-
>> |- --== END OF SUMMARY ==--
>> |-
>> |- [ INFO ] Starting engine service
>> |- [ INFO ] Restarting httpd
>> |- [ INFO ] Restarting ovirt-vmconsole proxy service
>> |- [ INFO ] Stage: Clean up
>> |- Log file is located at
>> /var/log/ovirt-engine/setup/ovirt-engine-setup-20160430234733-h0rns4.log
>> |- [ INFO ] Generating answer file
>> '/var/lib/ovirt-engine/setup/answers/20160430234907-setup.conf'
>> |- [ INFO ] Stage: Pre-termination
>> |- [ INFO ] Stage: Termination
>> |- [ INFO ] Execution of setup completed successfully
>> |- HE_APPLIANCE_ENGINE_SETUP_SUCCESS
>> [ INFO ] Engine-setup successfully completed
>> [ INFO ] Engine is still unreachable
>> [ INFO ] Engine is still not reachable, waiting...
>> [ INFO ] Engine is still unreachable
>> [ INFO ] Engine is still not reachable, waiting...
>> [ INFO ] Engine replied: DB Up!Welcome to Health Status!
>> [ INFO ] Acquiring internal CA cert from the engine
>> [ INFO ] The following CA certificate is going to be used, please
>> immediately interrupt if not correct:
>> [ INFO ] Issuer: C=US, O=lutwyn.org, CN=ovirt.lutwyn.org.36587, Subject:
>> C=US, O=lutwyn.org, CN=ovirt.lutwyn.org.36587, Fingerprint (SHA-1):
>> D84B4C7445EB87C48FFEDE02DF995ECBAA1BD84
>> [ INFO ] Connecting to the Engine
>> [ INFO ] Waiting for the host to become operational in the engine. This may
>> take several minutes...
>> [ INFO ] The VDSM Host is now operational
>> [ INFO ] Saving hosted-engine configuration on the shared storage domain
>> [ INFO ] Shutting down the engine VM
>> [ INFO ] Enabling and starting HA services
>> [ INFO ] Stage: Clean up
>> [ INFO ] Generating answer file
>> '/var/lib/ovirt-hosted-engine-setup/answers/answers-20160501015145.conf'
>> [ INFO ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
>> [ INFO ] Stage: Pre-termination
>> [ INFO ] Stage: Termination
>> [ INFO ] Hosted Engine successfully set up
>> [root at ovirt01 ~]#
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
>
>
> --
> Didi
More information about the Users
mailing list