Re: [ovirt-users] hosted engine setup failed for 10 minutes delay.. engine seems alive

On Mon, May 2, 2016 at 11:48 AM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: > On Mon, May 2, 2016 at 9:58 AM, Simone Tiraboschi wrote: >> >> >> >> hosted-engine-setup creates a fresh VM and inject a cloud-init script >> to configure it and execute there engine-setup to configure the engine >> as needed. >> Since engine-setup is running on the engine VM triggered by >> cloud-init, hosted-engine-setup has no way to really control its >> process status so we simply gather its output with a timeout of 10 >> minutes between each single output line. >> In nothing happens within 10 minutes (the value is easily >> customizable), hosted-engine-setup thinks that engine-setup is stuck. > > > > How can one customize the pre-set timeout?

> Could it be better to ask the user at the end of timeout if he/she wants to > wait again, instead of directly fail? Perhaps, can you please open a bz?

>> So the issue we have to understood is why this simple command took >> more than 10 minutes in your env: >> 2016-04-30 17:56:57 DEBUG >> otopi.plugins.ovirt_engine_setup.ovirt_engine.config.aaajdbc >> plugin.executeRaw:828 execute: ('/usr/bin/ovirt-aaa-jdbc-tool', >> '--db-config=/etc/ovirt-engine/aaa/internal.properties', 'user', >> 'password-reset', 'admin', '--password=env:pass', '--force', >> '--password-valid-to=2216-03-13 17:56:57Z'), executable='None', >> cwd='None', env={'LANG': 'en_US.UTF-8', 'SHLVL': '1', 'PYTHONPATH': >> '/usr/share/ovirt-engine/setup/bin/..::', 'pass': '**FILTERED**', >> 'OVIRT_ENGINE_JAVA_HOME_FORCE': '1', 'PWD': '/', >> 'OVIRT_ENGINE_JAVA_HOME': u'/usr/lib/jvm/jre', 'PATH': >> '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'OTOPI_LOGFILE': >> >> '/var/log/ovirt-engine/setup/ovirt-engine-setup-20160430175551-dttt2p.log', >> 'OVIRT_JBOSS_HOME': '/usr/share/ovirt-engine-wildfly', >> 'OTOPI_EXECDIR': '/'} > > > > > It seemed quite strange to me too (see below further info on this) > >> >> Can you please check the entropy value on your host? >> cat /proc/sys/kernel/random/entropy_avail >> > > I have not at hand now the server. I'll check soon and report > Do you mean entropy of the physical server that will operate as hypervisor?

That's a good question. Simone - do you know if we start the guest with virtio-rng?

This is another case of [1], perhaps we should reopen it. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1319827 > > >> >> > As a last question how to clean up things in case I have to start from >> > scratch. >> >> I'd recommend to redeploy from scratch instead of trying fixing it >> but, before that, we need to understand the root issue. >> > > So, trying restart the setup with generated answer file I got: > 1) if VM still powered on, an error about this condition > 2) if VM powered down, an error abut storage domain already in place and > restart not supported in this condition. > > I was able to continue with these steps: > > a) remove what inside the partially setup self hosted engine storage domain > rm -rf /SHE_DOMAIN/* > cd SHE_DOMAIN > mklost+found > > b) reboot the hypervisor > > c) stop vdsmd > > d) start the setup again with the answer file > It seems all went well and this time strangely the step that took more than > 10 minutes before lasted less than 2 seconds.... > > I was then able to deploy storage and iso domains without problems and self > hosted engine domain correctly detected and imported too. > Created two CentOS VMs without problems (6.7 and 7.2). > > See below the full output of deploy command > > > [root@ovirt01 ~]# hosted-engine --deploy > --config-append=/var/lib/ovirt-hosted-engine-setup/answers/answers-20160430200654.conf > [ INFO ] Stage: Initializing > [ INFO ] Generating a temporary VNC password. > [ INFO ] Stage: Environment setup > Configuration files: > ['/var/lib/ovirt-hosted-engine-setup/answers/answers-20160430200654.conf'] > Log file: > /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20160501014326-8frbxk.log > Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos) > [ INFO ] Hardware supports virtualization > [ INFO ] Bridge ovirtmgmt already created > [ INFO ] Stage: Environment packages setup > [ INFO ] Stage: Programs detection > [ INFO ] Stage: Environment setup > [ INFO ] Stage: Environment customization > > --== STORAGE CONFIGURATION ==-- > > During customization use CTRL-D to abort. > [ INFO ] Installing on first host > > --== SYSTEM CONFIGURATION ==-- > > > --== NETWORK CONFIGURATION ==-- > > > --== VM CONFIGURATION ==-- > > [ INFO ] Checking OVF archive content (could take a few minutes depending > on archive size) > [ INFO ] Checking OVF XML content (could take a few minutes depending on > archive size) > [WARNING] OVF does not contain a valid image description, using default. > Enter root password that will be used for the engine appliance > (leave it empty to skip): > Confirm appliance root password: > [ INFO ] The engine VM will be configured to use 192.168.1.212/24 > The following CPU types are supported by this host: > - model_Broadwell-noTSX: Intel Broadwell-noTSX Family > - model_Haswell-noTSX: Intel Haswell-noTSX Family > - model_SandyBridge: Intel SandyBridge Family > - model_Westmere: Intel Westmere Family > - model_Nehalem: Intel Nehalem Family > - model_Penryn: Intel Penryn Family > - model_Conroe: Intel Conroe Family > [WARNING] Minimum requirements for CPUs not met > [WARNING] Minimum requirements for disk size not met > > --== HOSTED ENGINE CONFIGURATION ==-- > > Enter the name which will be used to identify this host inside the > Administrator Portal [hosted_engine_1]: > Enter 'admin@internal' user password that will be used for > accessing the Administrator Portal: > Confirm 'admin@internal' user password: > [ INFO ] Stage: Setup validation > > --== CONFIGURATION PREVIEW ==-- > > Bridge interface : eth0 > Engine FQDN : ovirt.lutwyn.org > Bridge name : ovirtmgmt > Host address : ovirt01.lutwyn.org > SSH daemon port : 22 > Firewall manager : iptables > Gateway address : 192.168.1.1 > Host name for web application : hosted_engine_1 > Host ID : 1 > Image size GB : 10 > GlusterFS Share Name : hosted_engine_glusterfs > GlusterFS Brick Provisioning : False > Storage connection : > ovirt01.lutwyn.org:/SHE_DOMAIN > Console type : vnc > Memory size MB : 8192 > MAC address : 00:16:3e:2e:93:15 > Boot type : disk > Number of CPUs : 1 > OVF archive (for disk boot) : > /usr/share/ovirt-engine-appliance/ovirt-engine-appliance-3.6-20160420.1.el7.centos.ova > Restart engine VM after engine-setup: True > CPU Type : model_Broadwell-noTSX > [ INFO ] Stage: Transaction setup > [ INFO ] Stage: Misc configuration > [ INFO ] Stage: Package installation > [ INFO ] Stage: Misc configuration > [ INFO ] Configuring libvirt > [ INFO ] Configuring VDSM > [ INFO ] Starting vdsmd > [ INFO ] Waiting for VDSM hardware info > [ INFO ] Creating Storage Domain > [ INFO ] Creating Storage Pool > [ INFO ] Connecting Storage Pool > [ INFO ] Verifying sanlock lockspace initialization > [ INFO ] Creating Image for 'hosted-engine.lockspace' ... > [ INFO ] Image for 'hosted-engine.lockspace' created successfully > [ INFO ] Creating Image for 'hosted-engine.metadata' ... > [ INFO ] Image for 'hosted-engine.metadata' created successfully > [ INFO ] Creating VM Image > [ INFO ] Extracting disk image from OVF archive (could take a few minutes > depending on archive size) > [ INFO ] Validating pre-allocated volume size > [ INFO ] Uploading volume to data domain (could take a few minutes > depending on archive size) > [ INFO ] Image successfully imported from OVF > [ INFO ] Destroying Storage Pool > [ INFO ] Start monitoring domain > [ INFO ] Configuring VM > [ INFO ] Updating hosted-engine configuration > [ INFO ] Stage: Transaction commit > [ INFO ] Stage: Closing up > [ INFO ] Creating VM > You can now connect to the VM with the following command: > /bin/remote-viewer vnc://localhost:5900 > Use temporary password "4699ouyH" to connect to vnc console. > Please note that in order to use remote-viewer you need to be able > to run graphical applications. > This means that if you are using ssh you have to supply the -Y > flag (enables trusted X11 forwarding). > Otherwise you can run the command from a terminal in your > preferred desktop environment. > If you cannot run graphical applications you can connect to the > graphic console from another host or connect to the serial console using the > following command: > socat > UNIX-CONNECT:/var/run/ovirt-vmconsole-console/4dae6016-ff01-4a1f-87ea-09f049d0e58c.sock,user=ovirt-vmconsole > STDIO,raw,echo=0,escape=1 > Please ensure that your Guest OS is properly configured to support > serial console according to your distro documentation. > Follow > http://www.ovirt.org/Serial_Console_Setup#I_need_to_access_the_console_th... > for more info. > If you need to reboot the VM you will need to start it manually > using the command: > hosted-engine --vm-start > You can then set a temporary password using the command: > hosted-engine --add-console-password > [ INFO ] Running engine-setup on the appliance > |- [ INFO ] Stage: Initializing > |- [ INFO ] Stage: Environment setup > |- Configuration files: > ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', > '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', > '/root/ovirt-engine-answers', '/root/heanswers.conf'] > |- Log file: > /var/log/ovirt-engine/setup/ovirt-engine-setup-20160430234733-h0rns4.log > |- Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos) > |- [ INFO ] Stage: Environment packages setup > |- [ INFO ] Stage: Programs detection > |- [ INFO ] Stage: Environment setup > |- [ INFO ] Stage: Environment customization > |- > |- --== PRODUCT OPTIONS ==-- > |- > |- > |- --== PACKAGES ==-- > |- > |- > |- --== ALL IN ONE CONFIGURATION ==-- > |- > |- > |- --== NETWORK CONFIGURATION ==-- > |- > |- [ INFO ] firewalld will be configured as firewall manager. > |- > |- --== DATABASE CONFIGURATION ==-- > |- > |- > |- --== OVIRT ENGINE CONFIGURATION ==-- > |- > |- > |- --== STORAGE CONFIGURATION ==-- > |- > |- > |- --== PKI CONFIGURATION ==-- > |- > |- > |- --== APACHE CONFIGURATION ==-- > |- > |- > |- --== SYSTEM CONFIGURATION ==-- > |- > |- > |- --== MISC CONFIGURATION ==-- > |- > |- > |- --== END OF CONFIGURATION ==-- > |- > |- [ INFO ] Stage: Setup validation > |- [WARNING] Cannot validate host name settings, reason: resolved > host does not match any of the local addresses > |- [WARNING] Less than 16384MB of memory is available > |- > |- --== CONFIGURATION PREVIEW ==-- > |- > |- Application mode : virt > |- Default SAN wipe after delete : False > |- Firewall manager : firewalld > |- Update Firewall : True > |- Host FQDN : > ovirt.lutwyn.org > |- Engine database secured connection : False > |- Engine database host : localhost > |- Engine database user name : engine > |- Engine database name : engine > |- Engine database port : 5432 > |- Engine database host name validation : False > |- Engine installation : True > |- PKI organization : lutwyn.org > |- Configure local Engine database : True > |- Set application as default page : True > |- Configure Apache SSL : True > |- Configure VMConsole Proxy : True > |- Engine Host FQDN : > ovirt.lutwyn.org > |- Configure WebSocket Proxy : True > |- [ INFO ] Stage: Transaction setup > |- [ INFO ] Stopping engine service > |- [ INFO ] Stopping ovirt-fence-kdump-listener service > |- [ INFO ] Stopping websocket-proxy service > |- [ INFO ] Stage: Misc configuration > |- [ INFO ] Stage: Package installation > |- [ INFO ] Stage: Misc configuration > |- [ INFO ] Initializing PostgreSQL > |- [ INFO ] Creating PostgreSQL 'engine' database > |- [ INFO ] Configuring PostgreSQL > |- [ INFO ] Creating/refreshing Engine database schema > |- [ INFO ] Creating/refreshing Engine 'internal' domain database > schema > |- [ INFO ] Upgrading CA > |- [ INFO ] Creating CA > |- [ INFO ] Setting up ovirt-vmconsole proxy helper PKI artifacts > |- [ INFO ] Setting up ovirt-vmconsole SSH PKI artifacts > |- [ INFO ] Configuring WebSocket Proxy > |- [ INFO ] Generating post install configuration file > '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf' > |- [ INFO ] Stage: Transaction commit > |- [ INFO ] Stage: Closing up > |- > |- --== SUMMARY ==-- > |- > |- [WARNING] Less than 16384MB of memory is available > |- SSH fingerprint: > 13:dc:47:b9:38:66:15:eb:83:e8:ed:6f:71:cb:d4:ea > |- Internal CA > D8:4B:4C:74:45:EB:87:C4:8F:FE:DE:02:DF:99:5E:CB:AA:1B:D84 > |- Note! If you want to gather statistical information > you can install Reports and/or DWH: > |- http://www.ovirt.org/Ovirt_DWH > |- http://www.ovirt.org/Ovirt_Reports > |- Web access is enabled at: > |- http://ovirt.lutwyn.org:80/ovirt-engine > |- https://ovirt.lutwyn.org:443/ovirt-engine > |- Please use the user 'admin@internal' and password > specified in order to login > |- > |- --== END OF SUMMARY ==-- > |- > |- [ INFO ] Starting engine service > |- [ INFO ] Restarting httpd > |- [ INFO ] Restarting ovirt-vmconsole proxy service > |- [ INFO ] Stage: Clean up > |- Log file is located at > /var/log/ovirt-engine/setup/ovirt-engine-setup-20160430234733-h0rns4.log > |- [ INFO ] Generating answer file > '/var/lib/ovirt-engine/setup/answers/20160430234907-setup.conf' > |- [ INFO ] Stage: Pre-termination > |- [ INFO ] Stage: Termination > |- [ INFO ] Execution of setup completed successfully > |- HE_APPLIANCE_ENGINE_SETUP_SUCCESS > [ INFO ] Engine-setup successfully completed > [ INFO ] Engine is still unreachable > [ INFO ] Engine is still not reachable, waiting... > [ INFO ] Engine is still unreachable > [ INFO ] Engine is still not reachable, waiting... > [ INFO ] Engine replied: DB Up!Welcome to Health Status! > [ INFO ] Acquiring internal CA cert from the engine > [ INFO ] The following CA certificate is going to be used, please > immediately interrupt if not correct: > [ INFO ] Issuer: C=US, O=lutwyn.org, CN=ovirt.lutwyn.org.36587, Subject: > C=US, O=lutwyn.org, CN=ovirt.lutwyn.org.36587, Fingerprint (SHA-1): > D84B4C7445EB87C48FFEDE02DF995ECBAA1BD84 > [ INFO ] Connecting to the Engine > [ INFO ] Waiting for the host to become operational in the engine. This may > take several minutes... > [ INFO ] The VDSM Host is now operational > [ INFO ] Saving hosted-engine configuration on the shared storage domain > [ INFO ] Shutting down the engine VM > [ INFO ] Enabling and starting HA services > [ INFO ] Stage: Clean up > [ INFO ] Generating answer file > '/var/lib/ovirt-hosted-engine-setup/answers/answers-20160501015145.conf' > [ INFO ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf' > [ INFO ] Stage: Pre-termination > [ INFO ] Stage: Termination > [ INFO ] Hosted Engine successfully set up > [root@ovirt01 ~]# > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > -- Didi