On Mon, May 2, 2016 at 9:58 AM, Simone Tiraboschi wrote:


hosted-engine-setup creates a fresh VM and inject a cloud-init script
to configure it and execute there engine-setup to configure the engine
as needed.
Since engine-setup is running on the engine VM triggered by
cloud-init, hosted-engine-setup has no way to really control its
process status so we simply gather its output with a timeout of 10
minutes between each single output line.
In nothing happens within 10 minutes (the value is easily
customizable), hosted-engine-setup thinks that engine-setup is stuck.


How can one customize the pre-set timeout? 
Could it be better to ask the user at the end of timeout if he/she wants to wait again, instead of directly fail?



So the issue we have to understood is why this simple command took
more than 10 minutes in your env:
2016-04-30 17:56:57 DEBUG
otopi.plugins.ovirt_engine_setup.ovirt_engine.config.aaajdbc
plugin.executeRaw:828 execute: ('/usr/bin/ovirt-aaa-jdbc-tool',
'--db-config=/etc/ovirt-engine/aaa/internal.properties', 'user',
'password-reset', 'admin', '--password=env:pass', '--force',
'--password-valid-to=2216-03-13 17:56:57Z'), executable='None',
cwd='None', env={'LANG': 'en_US.UTF-8', 'SHLVL': '1', 'PYTHONPATH':
'/usr/share/ovirt-engine/setup/bin/..::', 'pass': '**FILTERED**',
'OVIRT_ENGINE_JAVA_HOME_FORCE': '1', 'PWD': '/',
'OVIRT_ENGINE_JAVA_HOME': u'/usr/lib/jvm/jre', 'PATH':
'/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'OTOPI_LOGFILE':
'/var/log/ovirt-engine/setup/ovirt-engine-setup-20160430175551-dttt2p.log',
'OVIRT_JBOSS_HOME': '/usr/share/ovirt-engine-wildfly',
'OTOPI_EXECDIR': '/'}



It seemed quite strange to me too (see below further info on this) 


Can you please check the entropy value on your host?
 cat /proc/sys/kernel/random/entropy_avail


I have not at hand now the server. I'll check soon and report
Do you mean entropy of the physical server that will operate as hypervisor?

 
> As a last question how to clean up things in case I have to start from
> scratch.

I'd recommend to redeploy from scratch instead of trying fixing it
but, before that, we need to understand the root issue.


So, trying restart the setup with generated answer file I got:
1) if VM still powered on, an error about this condition
2) if VM powered down, an error abut storage domain already in place and restart not supported in this condition.

I was able to continue with these steps:

a) remove what inside the partially setup self hosted engine storage domain
rm -rf /SHE_DOMAIN/* 
cd SHE_DOMAIN
mklost+found

b) reboot the hypervisor

c) stop vdsmd

d) start the setup again with the answer file 
It seems all went well and this time strangely the step that took more than 10 minutes before lasted less than 2 seconds....

I was then able to deploy storage and iso domains without problems and self hosted engine domain correctly detected and imported too.
Created two CentOS VMs without problems (6.7 and 7.2).

See below the full output of deploy command


[root@ovirt01 ~]# hosted-engine --deploy --config-append=/var/lib/ovirt-hosted-engine-setup/answers/answers-20160430200654.conf 
[ INFO  ] Stage: Initializing
[ INFO  ] Generating a temporary VNC password.
[ INFO  ] Stage: Environment setup
          Configuration files: ['/var/lib/ovirt-hosted-engine-setup/answers/answers-20160430200654.conf']
          Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20160501014326-8frbxk.log
          Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos)
[ INFO  ] Hardware supports virtualization
[ INFO  ] Bridge ovirtmgmt already created
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup
[ INFO  ] Stage: Environment customization
         
          --== STORAGE CONFIGURATION ==--
         
          During customization use CTRL-D to abort.
[ INFO  ] Installing on first host
         
          --== SYSTEM CONFIGURATION ==--
         
         
          --== NETWORK CONFIGURATION ==--
         
         
          --== VM CONFIGURATION ==--
         
[ INFO  ] Checking OVF archive content (could take a few minutes depending on archive size)
[ INFO  ] Checking OVF XML content (could take a few minutes depending on archive size)
[WARNING] OVF does not contain a valid image description, using default.
          Enter root password that will be used for the engine appliance (leave it empty to skip): 
          Confirm appliance root password: 
[ INFO  ] The engine VM will be configured to use 192.168.1.212/24
          The following CPU types are supported by this host:
          - model_Broadwell-noTSX: Intel Broadwell-noTSX Family
          - model_Haswell-noTSX: Intel Haswell-noTSX Family
          - model_SandyBridge: Intel SandyBridge Family
          - model_Westmere: Intel Westmere Family
          - model_Nehalem: Intel Nehalem Family
          - model_Penryn: Intel Penryn Family
          - model_Conroe: Intel Conroe Family
[WARNING] Minimum requirements for CPUs not met
[WARNING] Minimum requirements for disk size not met
         
          --== HOSTED ENGINE CONFIGURATION ==--
         
          Enter the name which will be used to identify this host inside the Administrator Portal [hosted_engine_1]: 
          Enter 'admin@internal' user password that will be used for accessing the Administrator Portal: 
          Confirm 'admin@internal' user password: 
[ INFO  ] Stage: Setup validation
         
          --== CONFIGURATION PREVIEW ==--
         
          Bridge interface                   : eth0
          Engine FQDN                        : ovirt.lutwyn.org
          Bridge name                        : ovirtmgmt
          Host address                       : ovirt01.lutwyn.org
          SSH daemon port                    : 22
          Firewall manager                   : iptables
          Gateway address                    : 192.168.1.1
          Host name for web application      : hosted_engine_1
          Host ID                            : 1
          Image size GB                      : 10
          GlusterFS Share Name               : hosted_engine_glusterfs
          GlusterFS Brick Provisioning       : False
          Storage connection                 : ovirt01.lutwyn.org:/SHE_DOMAIN
          Console type                       : vnc
          Memory size MB                     : 8192
          MAC address                        : 00:16:3e:2e:93:15
          Boot type                          : disk
          Number of CPUs                     : 1
          OVF archive (for disk boot)        : /usr/share/ovirt-engine-appliance/ovirt-engine-appliance-3.6-20160420.1.el7.centos.ova
          Restart engine VM after engine-setup: True
          CPU Type                           : model_Broadwell-noTSX
[ INFO  ] Stage: Transaction setup
[ INFO  ] Stage: Misc configuration
[ INFO  ] Stage: Package installation
[ INFO  ] Stage: Misc configuration
[ INFO  ] Configuring libvirt
[ INFO  ] Configuring VDSM
[ INFO  ] Starting vdsmd
[ INFO  ] Waiting for VDSM hardware info
[ INFO  ] Creating Storage Domain
[ INFO  ] Creating Storage Pool
[ INFO  ] Connecting Storage Pool
[ INFO  ] Verifying sanlock lockspace initialization
[ INFO  ] Creating Image for 'hosted-engine.lockspace' ...
[ INFO  ] Image for 'hosted-engine.lockspace' created successfully
[ INFO  ] Creating Image for 'hosted-engine.metadata' ...
[ INFO  ] Image for 'hosted-engine.metadata' created successfully
[ INFO  ] Creating VM Image
[ INFO  ] Extracting disk image from OVF archive (could take a few minutes depending on archive size)
[ INFO  ] Validating pre-allocated volume size
[ INFO  ] Uploading volume to data domain (could take a few minutes depending on archive size)
[ INFO  ] Image successfully imported from OVF
[ INFO  ] Destroying Storage Pool
[ INFO  ] Start monitoring domain
[ INFO  ] Configuring VM
[ INFO  ] Updating hosted-engine configuration
[ INFO  ] Stage: Transaction commit
[ INFO  ] Stage: Closing up
[ INFO  ] Creating VM
          You can now connect to the VM with the following command:
          /bin/remote-viewer vnc://localhost:5900
          Use temporary password "4699ouyH" to connect to vnc console.
          Please note that in order to use remote-viewer you need to be able to run graphical applications.
          This means that if you are using ssh you have to supply the -Y flag (enables trusted X11 forwarding).
          Otherwise you can run the command from a terminal in your preferred desktop environment.
          If you cannot run graphical applications you can connect to the graphic console from another host or connect to the serial console using the following command:
          socat UNIX-CONNECT:/var/run/ovirt-vmconsole-console/4dae6016-ff01-4a1f-87ea-09f049d0e58c.sock,user=ovirt-vmconsole STDIO,raw,echo=0,escape=1
          Please ensure that your Guest OS is properly configured to support serial console according to your distro documentation.
          Follow http://www.ovirt.org/Serial_Console_Setup#I_need_to_access_the_console_the_old_way for more info.
          If you need to reboot the VM you will need to start it manually using the command:
          hosted-engine --vm-start
          You can then set a temporary password using the command:
          hosted-engine --add-console-password
[ INFO  ] Running engine-setup on the appliance
          |- [ INFO  ] Stage: Initializing
          |- [ INFO  ] Stage: Environment setup
          |-           Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/root/ovirt-engine-answers', '/root/heanswers.conf']
          |-           Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20160430234733-h0rns4.log
          |-           Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos)
          |- [ INFO  ] Stage: Environment packages setup
          |- [ INFO  ] Stage: Programs detection
          |- [ INFO  ] Stage: Environment setup
          |- [ INFO  ] Stage: Environment customization
          |-          
          |-           --== PRODUCT OPTIONS ==--
          |-          
          |-          
          |-           --== PACKAGES ==--
          |-          
          |-          
          |-           --== ALL IN ONE CONFIGURATION ==--
          |-          
          |-          
          |-           --== NETWORK CONFIGURATION ==--
          |-          
          |- [ INFO  ] firewalld will be configured as firewall manager.
          |-          
          |-           --== DATABASE CONFIGURATION ==--
          |-          
          |-          
          |-           --== OVIRT ENGINE CONFIGURATION ==--
          |-          
          |-          
          |-           --== STORAGE CONFIGURATION ==--
          |-          
          |-          
          |-           --== PKI CONFIGURATION ==--
          |-          
          |-          
          |-           --== APACHE CONFIGURATION ==--
          |-          
          |-          
          |-           --== SYSTEM CONFIGURATION ==--
          |-          
          |-          
          |-           --== MISC CONFIGURATION ==--
          |-          
          |-          
          |-           --== END OF CONFIGURATION ==--
          |-          
          |- [ INFO  ] Stage: Setup validation
          |- [WARNING] Cannot validate host name settings, reason: resolved host does not match any of the local addresses
          |- [WARNING] Less than 16384MB of memory is available
          |-          
          |-           --== CONFIGURATION PREVIEW ==--
          |-          
          |-           Application mode                        : virt
          |-           Default SAN wipe after delete           : False
          |-           Firewall manager                        : firewalld
          |-           Update Firewall                         : True
          |-           Host FQDN                               : ovirt.lutwyn.org
          |-           Engine database secured connection      : False
          |-           Engine database host                    : localhost
          |-           Engine database user name               : engine
          |-           Engine database name                    : engine
          |-           Engine database port                    : 5432
          |-           Engine database host name validation    : False
          |-           Engine installation                     : True
          |-           PKI organization                        : lutwyn.org
          |-           Configure local Engine database         : True
          |-           Set application as default page         : True
          |-           Configure Apache SSL                    : True
          |-           Configure VMConsole Proxy               : True
          |-           Engine Host FQDN                        : ovirt.lutwyn.org
          |-           Configure WebSocket Proxy               : True
          |- [ INFO  ] Stage: Transaction setup
          |- [ INFO  ] Stopping engine service
          |- [ INFO  ] Stopping ovirt-fence-kdump-listener service
          |- [ INFO  ] Stopping websocket-proxy service
          |- [ INFO  ] Stage: Misc configuration
          |- [ INFO  ] Stage: Package installation
          |- [ INFO  ] Stage: Misc configuration
          |- [ INFO  ] Initializing PostgreSQL
          |- [ INFO  ] Creating PostgreSQL 'engine' database
          |- [ INFO  ] Configuring PostgreSQL
          |- [ INFO  ] Creating/refreshing Engine database schema
          |- [ INFO  ] Creating/refreshing Engine 'internal' domain database schema
          |- [ INFO  ] Upgrading CA
          |- [ INFO  ] Creating CA
          |- [ INFO  ] Setting up ovirt-vmconsole proxy helper PKI artifacts
          |- [ INFO  ] Setting up ovirt-vmconsole SSH PKI artifacts
          |- [ INFO  ] Configuring WebSocket Proxy
          |- [ INFO  ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
          |- [ INFO  ] Stage: Transaction commit
          |- [ INFO  ] Stage: Closing up
          |-          
          |-           --== SUMMARY ==--
          |-          
          |- [WARNING] Less than 16384MB of memory is available
          |-           SSH fingerprint: 13:dc:47:b9:38:66:15:eb:83:e8:ed:6f:71:cb:d4:ea
          |-           Internal CA D8:4B:4C:74:45:EB:87:C4:8F:FE:DE:02:DF:99:5E:CB:AA:1B:D84
          |-           Note! If you want to gather statistical information you can install Reports and/or DWH:
          |-               http://www.ovirt.org/Ovirt_DWH
          |-               http://www.ovirt.org/Ovirt_Reports
          |-           Web access is enabled at:
          |-               http://ovirt.lutwyn.org:80/ovirt-engine
          |-               https://ovirt.lutwyn.org:443/ovirt-engine
          |-           Please use the user 'admin@internal' and password specified in order to login
          |-          
          |-           --== END OF SUMMARY ==--
          |-          
          |- [ INFO  ] Starting engine service
          |- [ INFO  ] Restarting httpd
          |- [ INFO  ] Restarting ovirt-vmconsole proxy service
          |- [ INFO  ] Stage: Clean up
          |-           Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20160430234733-h0rns4.log
          |- [ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20160430234907-setup.conf'
          |- [ INFO  ] Stage: Pre-termination
          |- [ INFO  ] Stage: Termination
          |- [ INFO  ] Execution of setup completed successfully
          |- HE_APPLIANCE_ENGINE_SETUP_SUCCESS
[ INFO  ] Engine-setup successfully completed 
[ INFO  ] Engine is still unreachable
[ INFO  ] Engine is still not reachable, waiting...
[ INFO  ] Engine is still unreachable
[ INFO  ] Engine is still not reachable, waiting...
[ INFO  ] Engine replied: DB Up!Welcome to Health Status!
[ INFO  ] Acquiring internal CA cert from the engine
[ INFO  ] The following CA certificate is going to be used, please immediately interrupt if not correct:
[ INFO  ] Issuer: C=US, O=lutwyn.org, CN=ovirt.lutwyn.org.36587, Subject: C=US, O=lutwyn.org, CN=ovirt.lutwyn.org.36587, Fingerprint (SHA-1): D84B4C7445EB87C48FFEDE02DF995ECBAA1BD84
[ INFO  ] Connecting to the Engine
[ INFO  ] Waiting for the host to become operational in the engine. This may take several minutes...
[ INFO  ] The VDSM Host is now operational
[ INFO  ] Saving hosted-engine configuration on the shared storage domain
[ INFO  ] Shutting down the engine VM
[ INFO  ] Enabling and starting HA services
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20160501015145.conf'
[ INFO  ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ INFO  ] Hosted Engine successfully set up
[root@ovirt01 ~]#