[ovirt-users] hosted engine setup failed for 10 minutes delay.. engine seems alive
Gianluca Cecchi
gianluca.cecchi at gmail.com
Mon May 2 08:48:33 UTC 2016
On Mon, May 2, 2016 at 9:58 AM, Simone Tiraboschi wrote:
>
>
> hosted-engine-setup creates a fresh VM and inject a cloud-init script
> to configure it and execute there engine-setup to configure the engine
> as needed.
> Since engine-setup is running on the engine VM triggered by
> cloud-init, hosted-engine-setup has no way to really control its
> process status so we simply gather its output with a timeout of 10
> minutes between each single output line.
> In nothing happens within 10 minutes (the value is easily
> customizable), hosted-engine-setup thinks that engine-setup is stuck.
>
How can one customize the pre-set timeout?
Could it be better to ask the user at the end of timeout if he/she wants to
wait again, instead of directly fail?
> So the issue we have to understood is why this simple command took
> more than 10 minutes in your env:
> 2016-04-30 17:56:57 DEBUG
> otopi.plugins.ovirt_engine_setup.ovirt_engine.config.aaajdbc
> plugin.executeRaw:828 execute: ('/usr/bin/ovirt-aaa-jdbc-tool',
> '--db-config=/etc/ovirt-engine/aaa/internal.properties', 'user',
> 'password-reset', 'admin', '--password=env:pass', '--force',
> '--password-valid-to=2216-03-13 17:56:57Z'), executable='None',
> cwd='None', env={'LANG': 'en_US.UTF-8', 'SHLVL': '1', 'PYTHONPATH':
> '/usr/share/ovirt-engine/setup/bin/..::', 'pass': '**FILTERED**',
> 'OVIRT_ENGINE_JAVA_HOME_FORCE': '1', 'PWD': '/',
> 'OVIRT_ENGINE_JAVA_HOME': u'/usr/lib/jvm/jre', 'PATH':
> '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'OTOPI_LOGFILE':
> '/var/log/ovirt-engine/setup/ovirt-engine-setup-20160430175551-dttt2p.log',
> 'OVIRT_JBOSS_HOME': '/usr/share/ovirt-engine-wildfly',
> 'OTOPI_EXECDIR': '/'}
>
It seemed quite strange to me too (see below further info on this)
> Can you please check the entropy value on your host?
> cat /proc/sys/kernel/random/entropy_avail
>
>
I have not at hand now the server. I'll check soon and report
Do you mean entropy of the physical server that will operate as hypervisor?
> > As a last question how to clean up things in case I have to start from
> > scratch.
>
> I'd recommend to redeploy from scratch instead of trying fixing it
> but, before that, we need to understand the root issue.
>
>
So, trying restart the setup with generated answer file I got:
1) if VM still powered on, an error about this condition
2) if VM powered down, an error abut storage domain already in place and
restart not supported in this condition.
I was able to continue with these steps:
a) remove what inside the partially setup self hosted engine storage domain
rm -rf /SHE_DOMAIN/*
cd SHE_DOMAIN
mklost+found
b) reboot the hypervisor
c) stop vdsmd
d) start the setup again with the answer file
It seems all went well and this time strangely the step that took more than
10 minutes before lasted less than 2 seconds....
I was then able to deploy storage and iso domains without problems and self
hosted engine domain correctly detected and imported too.
Created two CentOS VMs without problems (6.7 and 7.2).
See below the full output of deploy command
[root at ovirt01 ~]# hosted-engine --deploy
--config-append=/var/lib/ovirt-hosted-engine-setup/answers/answers-20160430200654.conf
[ INFO ] Stage: Initializing
[ INFO ] Generating a temporary VNC password.
[ INFO ] Stage: Environment setup
Configuration files:
['/var/lib/ovirt-hosted-engine-setup/answers/answers-20160430200654.conf']
Log file:
/var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20160501014326-8frbxk.log
Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos)
[ INFO ] Hardware supports virtualization
[ INFO ] Bridge ovirtmgmt already created
[ INFO ] Stage: Environment packages setup
[ INFO ] Stage: Programs detection
[ INFO ] Stage: Environment setup
[ INFO ] Stage: Environment customization
--== STORAGE CONFIGURATION ==--
During customization use CTRL-D to abort.
[ INFO ] Installing on first host
--== SYSTEM CONFIGURATION ==--
--== NETWORK CONFIGURATION ==--
--== VM CONFIGURATION ==--
[ INFO ] Checking OVF archive content (could take a few minutes depending
on archive size)
[ INFO ] Checking OVF XML content (could take a few minutes depending on
archive size)
[WARNING] OVF does not contain a valid image description, using default.
Enter root password that will be used for the engine appliance
(leave it empty to skip):
Confirm appliance root password:
[ INFO ] The engine VM will be configured to use 192.168.1.212/24
The following CPU types are supported by this host:
- model_Broadwell-noTSX: Intel Broadwell-noTSX Family
- model_Haswell-noTSX: Intel Haswell-noTSX Family
- model_SandyBridge: Intel SandyBridge Family
- model_Westmere: Intel Westmere Family
- model_Nehalem: Intel Nehalem Family
- model_Penryn: Intel Penryn Family
- model_Conroe: Intel Conroe Family
[WARNING] Minimum requirements for CPUs not met
[WARNING] Minimum requirements for disk size not met
--== HOSTED ENGINE CONFIGURATION ==--
Enter the name which will be used to identify this host inside
the Administrator Portal [hosted_engine_1]:
Enter 'admin at internal' user password that will be used for
accessing the Administrator Portal:
Confirm 'admin at internal' user password:
[ INFO ] Stage: Setup validation
--== CONFIGURATION PREVIEW ==--
Bridge interface : eth0
Engine FQDN : ovirt.lutwyn.org
Bridge name : ovirtmgmt
Host address : ovirt01.lutwyn.org
SSH daemon port : 22
Firewall manager : iptables
Gateway address : 192.168.1.1
Host name for web application : hosted_engine_1
Host ID : 1
Image size GB : 10
GlusterFS Share Name : hosted_engine_glusterfs
GlusterFS Brick Provisioning : False
Storage connection : ovirt01.lutwyn.org:
/SHE_DOMAIN
Console type : vnc
Memory size MB : 8192
MAC address : 00:16:3e:2e:93:15
Boot type : disk
Number of CPUs : 1
OVF archive (for disk boot) :
/usr/share/ovirt-engine-appliance/ovirt-engine-appliance-3.6-20160420.1.el7.centos.ova
Restart engine VM after engine-setup: True
CPU Type : model_Broadwell-noTSX
[ INFO ] Stage: Transaction setup
[ INFO ] Stage: Misc configuration
[ INFO ] Stage: Package installation
[ INFO ] Stage: Misc configuration
[ INFO ] Configuring libvirt
[ INFO ] Configuring VDSM
[ INFO ] Starting vdsmd
[ INFO ] Waiting for VDSM hardware info
[ INFO ] Creating Storage Domain
[ INFO ] Creating Storage Pool
[ INFO ] Connecting Storage Pool
[ INFO ] Verifying sanlock lockspace initialization
[ INFO ] Creating Image for 'hosted-engine.lockspace' ...
[ INFO ] Image for 'hosted-engine.lockspace' created successfully
[ INFO ] Creating Image for 'hosted-engine.metadata' ...
[ INFO ] Image for 'hosted-engine.metadata' created successfully
[ INFO ] Creating VM Image
[ INFO ] Extracting disk image from OVF archive (could take a few minutes
depending on archive size)
[ INFO ] Validating pre-allocated volume size
[ INFO ] Uploading volume to data domain (could take a few minutes
depending on archive size)
[ INFO ] Image successfully imported from OVF
[ INFO ] Destroying Storage Pool
[ INFO ] Start monitoring domain
[ INFO ] Configuring VM
[ INFO ] Updating hosted-engine configuration
[ INFO ] Stage: Transaction commit
[ INFO ] Stage: Closing up
[ INFO ] Creating VM
You can now connect to the VM with the following command:
/bin/remote-viewer vnc://localhost:5900
Use temporary password "4699ouyH" to connect to vnc console.
Please note that in order to use remote-viewer you need to be
able to run graphical applications.
This means that if you are using ssh you have to supply the -Y
flag (enables trusted X11 forwarding).
Otherwise you can run the command from a terminal in your
preferred desktop environment.
If you cannot run graphical applications you can connect to the
graphic console from another host or connect to the serial console using
the following command:
socat
UNIX-CONNECT:/var/run/ovirt-vmconsole-console/4dae6016-ff01-4a1f-87ea-09f049d0e58c.sock,user=ovirt-vmconsole
STDIO,raw,echo=0,escape=1
Please ensure that your Guest OS is properly configured to
support serial console according to your distro documentation.
Follow
http://www.ovirt.org/Serial_Console_Setup#I_need_to_access_the_console_the_old_way
for more info.
If you need to reboot the VM you will need to start it manually
using the command:
hosted-engine --vm-start
You can then set a temporary password using the command:
hosted-engine --add-console-password
[ INFO ] Running engine-setup on the appliance
|- [ INFO ] Stage: Initializing
|- [ INFO ] Stage: Environment setup
|- Configuration files:
['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf',
'/etc/ovirt-engine-setup.conf.d/10-packaging.conf',
'/root/ovirt-engine-answers', '/root/heanswers.conf']
|- Log file:
/var/log/ovirt-engine/setup/ovirt-engine-setup-20160430234733-h0rns4.log
|- Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos)
|- [ INFO ] Stage: Environment packages setup
|- [ INFO ] Stage: Programs detection
|- [ INFO ] Stage: Environment setup
|- [ INFO ] Stage: Environment customization
|-
|- --== PRODUCT OPTIONS ==--
|-
|-
|- --== PACKAGES ==--
|-
|-
|- --== ALL IN ONE CONFIGURATION ==--
|-
|-
|- --== NETWORK CONFIGURATION ==--
|-
|- [ INFO ] firewalld will be configured as firewall manager.
|-
|- --== DATABASE CONFIGURATION ==--
|-
|-
|- --== OVIRT ENGINE CONFIGURATION ==--
|-
|-
|- --== STORAGE CONFIGURATION ==--
|-
|-
|- --== PKI CONFIGURATION ==--
|-
|-
|- --== APACHE CONFIGURATION ==--
|-
|-
|- --== SYSTEM CONFIGURATION ==--
|-
|-
|- --== MISC CONFIGURATION ==--
|-
|-
|- --== END OF CONFIGURATION ==--
|-
|- [ INFO ] Stage: Setup validation
|- [WARNING] Cannot validate host name settings, reason: resolved
host does not match any of the local addresses
|- [WARNING] Less than 16384MB of memory is available
|-
|- --== CONFIGURATION PREVIEW ==--
|-
|- Application mode : virt
|- Default SAN wipe after delete : False
|- Firewall manager : firewalld
|- Update Firewall : True
|- Host FQDN :
ovirt.lutwyn.org
|- Engine database secured connection : False
|- Engine database host : localhost
|- Engine database user name : engine
|- Engine database name : engine
|- Engine database port : 5432
|- Engine database host name validation : False
|- Engine installation : True
|- PKI organization : lutwyn.org
|- Configure local Engine database : True
|- Set application as default page : True
|- Configure Apache SSL : True
|- Configure VMConsole Proxy : True
|- Engine Host FQDN :
ovirt.lutwyn.org
|- Configure WebSocket Proxy : True
|- [ INFO ] Stage: Transaction setup
|- [ INFO ] Stopping engine service
|- [ INFO ] Stopping ovirt-fence-kdump-listener service
|- [ INFO ] Stopping websocket-proxy service
|- [ INFO ] Stage: Misc configuration
|- [ INFO ] Stage: Package installation
|- [ INFO ] Stage: Misc configuration
|- [ INFO ] Initializing PostgreSQL
|- [ INFO ] Creating PostgreSQL 'engine' database
|- [ INFO ] Configuring PostgreSQL
|- [ INFO ] Creating/refreshing Engine database schema
|- [ INFO ] Creating/refreshing Engine 'internal' domain
database schema
|- [ INFO ] Upgrading CA
|- [ INFO ] Creating CA
|- [ INFO ] Setting up ovirt-vmconsole proxy helper PKI artifacts
|- [ INFO ] Setting up ovirt-vmconsole SSH PKI artifacts
|- [ INFO ] Configuring WebSocket Proxy
|- [ INFO ] Generating post install configuration file
'/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
|- [ INFO ] Stage: Transaction commit
|- [ INFO ] Stage: Closing up
|-
|- --== SUMMARY ==--
|-
|- [WARNING] Less than 16384MB of memory is available
|- SSH fingerprint:
13:dc:47:b9:38:66:15:eb:83:e8:ed:6f:71:cb:d4:ea
|- Internal CA
D8:4B:4C:74:45:EB:87:C4:8F:FE:DE:02:DF:99:5E:CB:AA:1B:D84
|- Note! If you want to gather statistical information
you can install Reports and/or DWH:
|- http://www.ovirt.org/Ovirt_DWH
|- http://www.ovirt.org/Ovirt_Reports
|- Web access is enabled at:
|- http://ovirt.lutwyn.org:80/ovirt-engine
|- https://ovirt.lutwyn.org:443/ovirt-engine
|- Please use the user 'admin at internal' and password
specified in order to login
|-
|- --== END OF SUMMARY ==--
|-
|- [ INFO ] Starting engine service
|- [ INFO ] Restarting httpd
|- [ INFO ] Restarting ovirt-vmconsole proxy service
|- [ INFO ] Stage: Clean up
|- Log file is located at
/var/log/ovirt-engine/setup/ovirt-engine-setup-20160430234733-h0rns4.log
|- [ INFO ] Generating answer file
'/var/lib/ovirt-engine/setup/answers/20160430234907-setup.conf'
|- [ INFO ] Stage: Pre-termination
|- [ INFO ] Stage: Termination
|- [ INFO ] Execution of setup completed successfully
|- HE_APPLIANCE_ENGINE_SETUP_SUCCESS
[ INFO ] Engine-setup successfully completed
[ INFO ] Engine is still unreachable
[ INFO ] Engine is still not reachable, waiting...
[ INFO ] Engine is still unreachable
[ INFO ] Engine is still not reachable, waiting...
[ INFO ] Engine replied: DB Up!Welcome to Health Status!
[ INFO ] Acquiring internal CA cert from the engine
[ INFO ] The following CA certificate is going to be used, please
immediately interrupt if not correct:
[ INFO ] Issuer: C=US, O=lutwyn.org, CN=ovirt.lutwyn.org.36587, Subject:
C=US, O=lutwyn.org, CN=ovirt.lutwyn.org.36587, Fingerprint (SHA-1):
D84B4C7445EB87C48FFEDE02DF995ECBAA1BD84
[ INFO ] Connecting to the Engine
[ INFO ] Waiting for the host to become operational in the engine. This
may take several minutes...
[ INFO ] The VDSM Host is now operational
[ INFO ] Saving hosted-engine configuration on the shared storage domain
[ INFO ] Shutting down the engine VM
[ INFO ] Enabling and starting HA services
[ INFO ] Stage: Clean up
[ INFO ] Generating answer file
'/var/lib/ovirt-hosted-engine-setup/answers/answers-20160501015145.conf'
[ INFO ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ INFO ] Hosted Engine successfully set up
[root at ovirt01 ~]#
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160502/1d298d51/attachment-0001.html>
More information about the Users
mailing list