Re: [ovirt-users] hosted engine setup failed for 10 minutes delay.. engine seems alive

On Mon, May 2, 2016 at 9:58 AM, Simone Tiraboschi wrote: > > > > hosted-engine-setup creates a fresh VM and inject a cloud-init script > to configure it and execute there engine-setup to configure the engine > as needed. > Since engine-setup is running on the engine VM triggered by > cloud-init, hosted-engine-setup has no way to really control its > process status so we simply gather its output with a timeout of 10 > minutes between each single output line. > In nothing happens within 10 minutes (the value is easily > customizable), hosted-engine-setup thinks that engine-setup is stuck. How can one customize the pre-set timeout? Could it be better to ask the user at the end of timeout if he/she wants to wait again, instead of directly fail?

> > So the issue we have to understood is why this simple command took > more than 10 minutes in your env: > 2016-04-30 17:56:57 DEBUG > otopi.plugins.ovirt_engine_setup.ovirt_engine.config.aaajdbc > plugin.executeRaw:828 execute: ('/usr/bin/ovirt-aaa-jdbc-tool', > '--db-config=/etc/ovirt-engine/aaa/internal.properties', 'user', > 'password-reset', 'admin', '--password=env:pass', '--force', > '--password-valid-to=2216-03-13 17:56:57Z'), executable='None', > cwd='None', env={'LANG': 'en_US.UTF-8', 'SHLVL': '1', 'PYTHONPATH': > '/usr/share/ovirt-engine/setup/bin/..::', 'pass': '**FILTERED**', > 'OVIRT_ENGINE_JAVA_HOME_FORCE': '1', 'PWD': '/', > 'OVIRT_ENGINE_JAVA_HOME': u'/usr/lib/jvm/jre', 'PATH': > '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'OTOPI_LOGFILE': > > '/var/log/ovirt-engine/setup/ovirt-engine-setup-20160430175551-dttt2p.log', > 'OVIRT_JBOSS_HOME': '/usr/share/ovirt-engine-wildfly', > 'OTOPI_EXECDIR': '/'} It seemed quite strange to me too (see below further info on this) > > Can you please check the entropy value on your host? > cat /proc/sys/kernel/random/entropy_avail > I have not at hand now the server. I'll check soon and report Do you mean entropy of the physical server that will operate as hypervisor?

> > > As a last question how to clean up things in case I have to start from > > scratch. > > I'd recommend to redeploy from scratch instead of trying fixing it > but, before that, we need to understand the root issue. > So, trying restart the setup with generated answer file I got: 1) if VM still powered on, an error about this condition 2) if VM powered down, an error abut storage domain already in place and restart not supported in this condition. I was able to continue with these steps: a) remove what inside the partially setup self hosted engine storage domain rm -rf /SHE_DOMAIN/* cd SHE_DOMAIN mklost+found b) reboot the hypervisor c) stop vdsmd d) start the setup again with the answer file It seems all went well and this time strangely the step that took more than 10 minutes before lasted less than 2 seconds.... I was then able to deploy storage and iso domains without problems and self hosted engine domain correctly detected and imported too. Created two CentOS VMs without problems (6.7 and 7.2). See below the full output of deploy command [root@ovirt01 ~]# hosted-engine --deploy --config-append=/var/lib/ovirt-hosted-engine-setup/answers/answers-20160430200654.conf [ INFO ] Stage: Initializing [ INFO ] Generating a temporary VNC password. [ INFO ] Stage: Environment setup Configuration files: ['/var/lib/ovirt-hosted-engine-setup/answers/answers-20160430200654.conf'] Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20160501014326-8frbxk.log Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos) [ INFO ] Hardware supports virtualization [ INFO ] Bridge ovirtmgmt already created [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ INFO ] Stage: Environment customization --== STORAGE CONFIGURATION ==-- During customization use CTRL-D to abort. [ INFO ] Installing on first host --== SYSTEM CONFIGURATION ==-- --== NETWORK CONFIGURATION ==-- --== VM CONFIGURATION ==-- [ INFO ] Checking OVF archive content (could take a few minutes depending on archive size) [ INFO ] Checking OVF XML content (could take a few minutes depending on archive size) [WARNING] OVF does not contain a valid image description, using default. Enter root password that will be used for the engine appliance (leave it empty to skip): Confirm appliance root password: [ INFO ] The engine VM will be configured to use 192.168.1.212/24 The following CPU types are supported by this host: - model_Broadwell-noTSX: Intel Broadwell-noTSX Family - model_Haswell-noTSX: Intel Haswell-noTSX Family - model_SandyBridge: Intel SandyBridge Family - model_Westmere: Intel Westmere Family - model_Nehalem: Intel Nehalem Family - model_Penryn: Intel Penryn Family - model_Conroe: Intel Conroe Family [WARNING] Minimum requirements for CPUs not met [WARNING] Minimum requirements for disk size not met --== HOSTED ENGINE CONFIGURATION ==-- Enter the name which will be used to identify this host inside the Administrator Portal [hosted_engine_1]: Enter 'admin@internal' user password that will be used for accessing the Administrator Portal: Confirm 'admin@internal' user password: [ INFO ] Stage: Setup validation --== CONFIGURATION PREVIEW ==-- Bridge interface : eth0 Engine FQDN : ovirt.lutwyn.org Bridge name : ovirtmgmt Host address : ovirt01.lutwyn.org SSH daemon port : 22 Firewall manager : iptables Gateway address : 192.168.1.1 Host name for web application : hosted_engine_1 Host ID : 1 Image size GB : 10 GlusterFS Share Name : hosted_engine_glusterfs GlusterFS Brick Provisioning : False Storage connection : ovirt01.lutwyn.org:/SHE_DOMAIN Console type : vnc Memory size MB : 8192 MAC address : 00:16:3e:2e:93:15 Boot type : disk Number of CPUs : 1 OVF archive (for disk boot) : /usr/share/ovirt-engine-appliance/ovirt-engine-appliance-3.6-20160420.1.el7.centos.ova Restart engine VM after engine-setup: True CPU Type : model_Broadwell-noTSX [ INFO ] Stage: Transaction setup [ INFO ] Stage: Misc configuration [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ INFO ] Configuring libvirt [ INFO ] Configuring VDSM [ INFO ] Starting vdsmd [ INFO ] Waiting for VDSM hardware info [ INFO ] Creating Storage Domain [ INFO ] Creating Storage Pool [ INFO ] Connecting Storage Pool [ INFO ] Verifying sanlock lockspace initialization [ INFO ] Creating Image for 'hosted-engine.lockspace' ... [ INFO ] Image for 'hosted-engine.lockspace' created successfully [ INFO ] Creating Image for 'hosted-engine.metadata' ... [ INFO ] Image for 'hosted-engine.metadata' created successfully [ INFO ] Creating VM Image [ INFO ] Extracting disk image from OVF archive (could take a few minutes depending on archive size) [ INFO ] Validating pre-allocated volume size [ INFO ] Uploading volume to data domain (could take a few minutes depending on archive size) [ INFO ] Image successfully imported from OVF [ INFO ] Destroying Storage Pool [ INFO ] Start monitoring domain [ INFO ] Configuring VM [ INFO ] Updating hosted-engine configuration [ INFO ] Stage: Transaction commit [ INFO ] Stage: Closing up [ INFO ] Creating VM You can now connect to the VM with the following command: /bin/remote-viewer vnc://localhost:5900 Use temporary password "4699ouyH" to connect to vnc console. Please note that in order to use remote-viewer you need to be able to run graphical applications. This means that if you are using ssh you have to supply the -Y flag (enables trusted X11 forwarding). Otherwise you can run the command from a terminal in your preferred desktop environment. If you cannot run graphical applications you can connect to the graphic console from another host or connect to the serial console using the following command: socat UNIX-CONNECT:/var/run/ovirt-vmconsole-console/4dae6016-ff01-4a1f-87ea-09f049d0e58c.sock,user=ovirt-vmconsole STDIO,raw,echo=0,escape=1 Please ensure that your Guest OS is properly configured to support serial console according to your distro documentation. Follow http://www.ovirt.org/Serial_Console_Setup#I_need_to_access_the_console_th... for more info. If you need to reboot the VM you will need to start it manually using the command: hosted-engine --vm-start You can then set a temporary password using the command: hosted-engine --add-console-password [ INFO ] Running engine-setup on the appliance |- [ INFO ] Stage: Initializing |- [ INFO ] Stage: Environment setup |- Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/root/ovirt-engine-answers', '/root/heanswers.conf'] |- Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20160430234733-h0rns4.log |- Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos) |- [ INFO ] Stage: Environment packages setup |- [ INFO ] Stage: Programs detection |- [ INFO ] Stage: Environment setup |- [ INFO ] Stage: Environment customization |- |- --== PRODUCT OPTIONS ==-- |- |- |- --== PACKAGES ==-- |- |- |- --== ALL IN ONE CONFIGURATION ==-- |- |- |- --== NETWORK CONFIGURATION ==-- |- |- [ INFO ] firewalld will be configured as firewall manager. |- |- --== DATABASE CONFIGURATION ==-- |- |- |- --== OVIRT ENGINE CONFIGURATION ==-- |- |- |- --== STORAGE CONFIGURATION ==-- |- |- |- --== PKI CONFIGURATION ==-- |- |- |- --== APACHE CONFIGURATION ==-- |- |- |- --== SYSTEM CONFIGURATION ==-- |- |- |- --== MISC CONFIGURATION ==-- |- |- |- --== END OF CONFIGURATION ==-- |- |- [ INFO ] Stage: Setup validation |- [WARNING] Cannot validate host name settings, reason: resolved host does not match any of the local addresses |- [WARNING] Less than 16384MB of memory is available |- |- --== CONFIGURATION PREVIEW ==-- |- |- Application mode : virt |- Default SAN wipe after delete : False |- Firewall manager : firewalld |- Update Firewall : True |- Host FQDN : ovirt.lutwyn.org |- Engine database secured connection : False |- Engine database host : localhost |- Engine database user name : engine |- Engine database name : engine |- Engine database port : 5432 |- Engine database host name validation : False |- Engine installation : True |- PKI organization : lutwyn.org |- Configure local Engine database : True |- Set application as default page : True |- Configure Apache SSL : True |- Configure VMConsole Proxy : True |- Engine Host FQDN : ovirt.lutwyn.org |- Configure WebSocket Proxy : True |- [ INFO ] Stage: Transaction setup |- [ INFO ] Stopping engine service |- [ INFO ] Stopping ovirt-fence-kdump-listener service |- [ INFO ] Stopping websocket-proxy service |- [ INFO ] Stage: Misc configuration |- [ INFO ] Stage: Package installation |- [ INFO ] Stage: Misc configuration |- [ INFO ] Initializing PostgreSQL |- [ INFO ] Creating PostgreSQL 'engine' database |- [ INFO ] Configuring PostgreSQL |- [ INFO ] Creating/refreshing Engine database schema |- [ INFO ] Creating/refreshing Engine 'internal' domain database schema |- [ INFO ] Upgrading CA |- [ INFO ] Creating CA |- [ INFO ] Setting up ovirt-vmconsole proxy helper PKI artifacts |- [ INFO ] Setting up ovirt-vmconsole SSH PKI artifacts |- [ INFO ] Configuring WebSocket Proxy |- [ INFO ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf' |- [ INFO ] Stage: Transaction commit |- [ INFO ] Stage: Closing up |- |- --== SUMMARY ==-- |- |- [WARNING] Less than 16384MB of memory is available |- SSH fingerprint: 13:dc:47:b9:38:66:15:eb:83:e8:ed:6f:71:cb:d4:ea |- Internal CA D8:4B:4C:74:45:EB:87:C4:8F:FE:DE:02:DF:99:5E:CB:AA:1B:D84 |- Note! If you want to gather statistical information you can install Reports and/or DWH: |- http://www.ovirt.org/Ovirt_DWH |- http://www.ovirt.org/Ovirt_Reports |- Web access is enabled at: |- http://ovirt.lutwyn.org:80/ovirt-engine |- https://ovirt.lutwyn.org:443/ovirt-engine |- Please use the user 'admin@internal' and password specified in order to login |- |- --== END OF SUMMARY ==-- |- |- [ INFO ] Starting engine service |- [ INFO ] Restarting httpd |- [ INFO ] Restarting ovirt-vmconsole proxy service |- [ INFO ] Stage: Clean up |- Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20160430234733-h0rns4.log |- [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20160430234907-setup.conf' |- [ INFO ] Stage: Pre-termination |- [ INFO ] Stage: Termination |- [ INFO ] Execution of setup completed successfully |- HE_APPLIANCE_ENGINE_SETUP_SUCCESS [ INFO ] Engine-setup successfully completed [ INFO ] Engine is still unreachable [ INFO ] Engine is still not reachable, waiting... [ INFO ] Engine is still unreachable [ INFO ] Engine is still not reachable, waiting... [ INFO ] Engine replied: DB Up!Welcome to Health Status! [ INFO ] Acquiring internal CA cert from the engine [ INFO ] The following CA certificate is going to be used, please immediately interrupt if not correct: [ INFO ] Issuer: C=US, O=lutwyn.org, CN=ovirt.lutwyn.org.36587, Subject: C=US, O=lutwyn.org, CN=ovirt.lutwyn.org.36587, Fingerprint (SHA-1): D84B4C7445EB87C48FFEDE02DF995ECBAA1BD84 [ INFO ] Connecting to the Engine [ INFO ] Waiting for the host to become operational in the engine. This may take several minutes... [ INFO ] The VDSM Host is now operational [ INFO ] Saving hosted-engine configuration on the shared storage domain [ INFO ] Shutting down the engine VM [ INFO ] Enabling and starting HA services [ INFO ] Stage: Clean up [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20160501015145.conf' [ INFO ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ INFO ] Hosted Engine successfully set up [root@ovirt01 ~]# _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users