[ovirt-users] hosted engine setup failed for 10 minutes delay.. engine seems alive

Yedidyah Bar David didi at redhat.com
Mon May 2 09:06:05 UTC 2016


On Mon, May 2, 2016 at 11:48 AM, Gianluca Cecchi
<gianluca.cecchi at gmail.com> wrote:
> On Mon, May 2, 2016 at 9:58 AM, Simone Tiraboschi wrote:
>>
>>
>>
>> hosted-engine-setup creates a fresh VM and inject a cloud-init script
>> to configure it and execute there engine-setup to configure the engine
>> as needed.
>> Since engine-setup is running on the engine VM triggered by
>> cloud-init, hosted-engine-setup has no way to really control its
>> process status so we simply gather its output with a timeout of 10
>> minutes between each single output line.
>> In nothing happens within 10 minutes (the value is easily
>> customizable), hosted-engine-setup thinks that engine-setup is stuck.
>
>
>
> How can one customize the pre-set timeout?
> Could it be better to ask the user at the end of timeout if he/she wants to
> wait again, instead of directly fail?

Perhaps, can you please open a bz?

>
>
>>
>> So the issue we have to understood is why this simple command took
>> more than 10 minutes in your env:
>> 2016-04-30 17:56:57 DEBUG
>> otopi.plugins.ovirt_engine_setup.ovirt_engine.config.aaajdbc
>> plugin.executeRaw:828 execute: ('/usr/bin/ovirt-aaa-jdbc-tool',
>> '--db-config=/etc/ovirt-engine/aaa/internal.properties', 'user',
>> 'password-reset', 'admin', '--password=env:pass', '--force',
>> '--password-valid-to=2216-03-13 17:56:57Z'), executable='None',
>> cwd='None', env={'LANG': 'en_US.UTF-8', 'SHLVL': '1', 'PYTHONPATH':
>> '/usr/share/ovirt-engine/setup/bin/..::', 'pass': '**FILTERED**',
>> 'OVIRT_ENGINE_JAVA_HOME_FORCE': '1', 'PWD': '/',
>> 'OVIRT_ENGINE_JAVA_HOME': u'/usr/lib/jvm/jre', 'PATH':
>> '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'OTOPI_LOGFILE':
>>
>> '/var/log/ovirt-engine/setup/ovirt-engine-setup-20160430175551-dttt2p.log',
>> 'OVIRT_JBOSS_HOME': '/usr/share/ovirt-engine-wildfly',
>> 'OTOPI_EXECDIR': '/'}
>
>
>
>
> It seemed quite strange to me too (see below further info on this)
>
>>
>> Can you please check the entropy value on your host?
>>  cat /proc/sys/kernel/random/entropy_avail
>>
>
> I have not at hand now the server. I'll check soon and report
> Do you mean entropy of the physical server that will operate as hypervisor?

That's a good question. Simone - do you know if we start the guest with
virtio-rng?

This is another case of [1], perhaps we should reopen it.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1319827

>
>
>>
>> > As a last question how to clean up things in case I have to start from
>> > scratch.
>>
>> I'd recommend to redeploy from scratch instead of trying fixing it
>> but, before that, we need to understand the root issue.
>>
>
> So, trying restart the setup with generated answer file I got:
> 1) if VM still powered on, an error about this condition
> 2) if VM powered down, an error abut storage domain already in place and
> restart not supported in this condition.
>
> I was able to continue with these steps:
>
> a) remove what inside the partially setup self hosted engine storage domain
> rm -rf /SHE_DOMAIN/*
> cd SHE_DOMAIN
> mklost+found
>
> b) reboot the hypervisor
>
> c) stop vdsmd
>
> d) start the setup again with the answer file
> It seems all went well and this time strangely the step that took more than
> 10 minutes before lasted less than 2 seconds....
>
> I was then able to deploy storage and iso domains without problems and self
> hosted engine domain correctly detected and imported too.
> Created two CentOS VMs without problems (6.7 and 7.2).
>
> See below the full output of deploy command
>
>
> [root at ovirt01 ~]# hosted-engine --deploy
> --config-append=/var/lib/ovirt-hosted-engine-setup/answers/answers-20160430200654.conf
> [ INFO  ] Stage: Initializing
> [ INFO  ] Generating a temporary VNC password.
> [ INFO  ] Stage: Environment setup
>           Configuration files:
> ['/var/lib/ovirt-hosted-engine-setup/answers/answers-20160430200654.conf']
>           Log file:
> /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20160501014326-8frbxk.log
>           Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos)
> [ INFO  ] Hardware supports virtualization
> [ INFO  ] Bridge ovirtmgmt already created
> [ INFO  ] Stage: Environment packages setup
> [ INFO  ] Stage: Programs detection
> [ INFO  ] Stage: Environment setup
> [ INFO  ] Stage: Environment customization
>
>           --== STORAGE CONFIGURATION ==--
>
>           During customization use CTRL-D to abort.
> [ INFO  ] Installing on first host
>
>           --== SYSTEM CONFIGURATION ==--
>
>
>           --== NETWORK CONFIGURATION ==--
>
>
>           --== VM CONFIGURATION ==--
>
> [ INFO  ] Checking OVF archive content (could take a few minutes depending
> on archive size)
> [ INFO  ] Checking OVF XML content (could take a few minutes depending on
> archive size)
> [WARNING] OVF does not contain a valid image description, using default.
>           Enter root password that will be used for the engine appliance
> (leave it empty to skip):
>           Confirm appliance root password:
> [ INFO  ] The engine VM will be configured to use 192.168.1.212/24
>           The following CPU types are supported by this host:
>           - model_Broadwell-noTSX: Intel Broadwell-noTSX Family
>           - model_Haswell-noTSX: Intel Haswell-noTSX Family
>           - model_SandyBridge: Intel SandyBridge Family
>           - model_Westmere: Intel Westmere Family
>           - model_Nehalem: Intel Nehalem Family
>           - model_Penryn: Intel Penryn Family
>           - model_Conroe: Intel Conroe Family
> [WARNING] Minimum requirements for CPUs not met
> [WARNING] Minimum requirements for disk size not met
>
>           --== HOSTED ENGINE CONFIGURATION ==--
>
>           Enter the name which will be used to identify this host inside the
> Administrator Portal [hosted_engine_1]:
>           Enter 'admin at internal' user password that will be used for
> accessing the Administrator Portal:
>           Confirm 'admin at internal' user password:
> [ INFO  ] Stage: Setup validation
>
>           --== CONFIGURATION PREVIEW ==--
>
>           Bridge interface                   : eth0
>           Engine FQDN                        : ovirt.lutwyn.org
>           Bridge name                        : ovirtmgmt
>           Host address                       : ovirt01.lutwyn.org
>           SSH daemon port                    : 22
>           Firewall manager                   : iptables
>           Gateway address                    : 192.168.1.1
>           Host name for web application      : hosted_engine_1
>           Host ID                            : 1
>           Image size GB                      : 10
>           GlusterFS Share Name               : hosted_engine_glusterfs
>           GlusterFS Brick Provisioning       : False
>           Storage connection                 :
> ovirt01.lutwyn.org:/SHE_DOMAIN
>           Console type                       : vnc
>           Memory size MB                     : 8192
>           MAC address                        : 00:16:3e:2e:93:15
>           Boot type                          : disk
>           Number of CPUs                     : 1
>           OVF archive (for disk boot)        :
> /usr/share/ovirt-engine-appliance/ovirt-engine-appliance-3.6-20160420.1.el7.centos.ova
>           Restart engine VM after engine-setup: True
>           CPU Type                           : model_Broadwell-noTSX
> [ INFO  ] Stage: Transaction setup
> [ INFO  ] Stage: Misc configuration
> [ INFO  ] Stage: Package installation
> [ INFO  ] Stage: Misc configuration
> [ INFO  ] Configuring libvirt
> [ INFO  ] Configuring VDSM
> [ INFO  ] Starting vdsmd
> [ INFO  ] Waiting for VDSM hardware info
> [ INFO  ] Creating Storage Domain
> [ INFO  ] Creating Storage Pool
> [ INFO  ] Connecting Storage Pool
> [ INFO  ] Verifying sanlock lockspace initialization
> [ INFO  ] Creating Image for 'hosted-engine.lockspace' ...
> [ INFO  ] Image for 'hosted-engine.lockspace' created successfully
> [ INFO  ] Creating Image for 'hosted-engine.metadata' ...
> [ INFO  ] Image for 'hosted-engine.metadata' created successfully
> [ INFO  ] Creating VM Image
> [ INFO  ] Extracting disk image from OVF archive (could take a few minutes
> depending on archive size)
> [ INFO  ] Validating pre-allocated volume size
> [ INFO  ] Uploading volume to data domain (could take a few minutes
> depending on archive size)
> [ INFO  ] Image successfully imported from OVF
> [ INFO  ] Destroying Storage Pool
> [ INFO  ] Start monitoring domain
> [ INFO  ] Configuring VM
> [ INFO  ] Updating hosted-engine configuration
> [ INFO  ] Stage: Transaction commit
> [ INFO  ] Stage: Closing up
> [ INFO  ] Creating VM
>           You can now connect to the VM with the following command:
>           /bin/remote-viewer vnc://localhost:5900
>           Use temporary password "4699ouyH" to connect to vnc console.
>           Please note that in order to use remote-viewer you need to be able
> to run graphical applications.
>           This means that if you are using ssh you have to supply the -Y
> flag (enables trusted X11 forwarding).
>           Otherwise you can run the command from a terminal in your
> preferred desktop environment.
>           If you cannot run graphical applications you can connect to the
> graphic console from another host or connect to the serial console using the
> following command:
>           socat
> UNIX-CONNECT:/var/run/ovirt-vmconsole-console/4dae6016-ff01-4a1f-87ea-09f049d0e58c.sock,user=ovirt-vmconsole
> STDIO,raw,echo=0,escape=1
>           Please ensure that your Guest OS is properly configured to support
> serial console according to your distro documentation.
>           Follow
> http://www.ovirt.org/Serial_Console_Setup#I_need_to_access_the_console_the_old_way
> for more info.
>           If you need to reboot the VM you will need to start it manually
> using the command:
>           hosted-engine --vm-start
>           You can then set a temporary password using the command:
>           hosted-engine --add-console-password
> [ INFO  ] Running engine-setup on the appliance
>           |- [ INFO  ] Stage: Initializing
>           |- [ INFO  ] Stage: Environment setup
>           |-           Configuration files:
> ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf',
> '/etc/ovirt-engine-setup.conf.d/10-packaging.conf',
> '/root/ovirt-engine-answers', '/root/heanswers.conf']
>           |-           Log file:
> /var/log/ovirt-engine/setup/ovirt-engine-setup-20160430234733-h0rns4.log
>           |-           Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos)
>           |- [ INFO  ] Stage: Environment packages setup
>           |- [ INFO  ] Stage: Programs detection
>           |- [ INFO  ] Stage: Environment setup
>           |- [ INFO  ] Stage: Environment customization
>           |-
>           |-           --== PRODUCT OPTIONS ==--
>           |-
>           |-
>           |-           --== PACKAGES ==--
>           |-
>           |-
>           |-           --== ALL IN ONE CONFIGURATION ==--
>           |-
>           |-
>           |-           --== NETWORK CONFIGURATION ==--
>           |-
>           |- [ INFO  ] firewalld will be configured as firewall manager.
>           |-
>           |-           --== DATABASE CONFIGURATION ==--
>           |-
>           |-
>           |-           --== OVIRT ENGINE CONFIGURATION ==--
>           |-
>           |-
>           |-           --== STORAGE CONFIGURATION ==--
>           |-
>           |-
>           |-           --== PKI CONFIGURATION ==--
>           |-
>           |-
>           |-           --== APACHE CONFIGURATION ==--
>           |-
>           |-
>           |-           --== SYSTEM CONFIGURATION ==--
>           |-
>           |-
>           |-           --== MISC CONFIGURATION ==--
>           |-
>           |-
>           |-           --== END OF CONFIGURATION ==--
>           |-
>           |- [ INFO  ] Stage: Setup validation
>           |- [WARNING] Cannot validate host name settings, reason: resolved
> host does not match any of the local addresses
>           |- [WARNING] Less than 16384MB of memory is available
>           |-
>           |-           --== CONFIGURATION PREVIEW ==--
>           |-
>           |-           Application mode                        : virt
>           |-           Default SAN wipe after delete           : False
>           |-           Firewall manager                        : firewalld
>           |-           Update Firewall                         : True
>           |-           Host FQDN                               :
> ovirt.lutwyn.org
>           |-           Engine database secured connection      : False
>           |-           Engine database host                    : localhost
>           |-           Engine database user name               : engine
>           |-           Engine database name                    : engine
>           |-           Engine database port                    : 5432
>           |-           Engine database host name validation    : False
>           |-           Engine installation                     : True
>           |-           PKI organization                        : lutwyn.org
>           |-           Configure local Engine database         : True
>           |-           Set application as default page         : True
>           |-           Configure Apache SSL                    : True
>           |-           Configure VMConsole Proxy               : True
>           |-           Engine Host FQDN                        :
> ovirt.lutwyn.org
>           |-           Configure WebSocket Proxy               : True
>           |- [ INFO  ] Stage: Transaction setup
>           |- [ INFO  ] Stopping engine service
>           |- [ INFO  ] Stopping ovirt-fence-kdump-listener service
>           |- [ INFO  ] Stopping websocket-proxy service
>           |- [ INFO  ] Stage: Misc configuration
>           |- [ INFO  ] Stage: Package installation
>           |- [ INFO  ] Stage: Misc configuration
>           |- [ INFO  ] Initializing PostgreSQL
>           |- [ INFO  ] Creating PostgreSQL 'engine' database
>           |- [ INFO  ] Configuring PostgreSQL
>           |- [ INFO  ] Creating/refreshing Engine database schema
>           |- [ INFO  ] Creating/refreshing Engine 'internal' domain database
> schema
>           |- [ INFO  ] Upgrading CA
>           |- [ INFO  ] Creating CA
>           |- [ INFO  ] Setting up ovirt-vmconsole proxy helper PKI artifacts
>           |- [ INFO  ] Setting up ovirt-vmconsole SSH PKI artifacts
>           |- [ INFO  ] Configuring WebSocket Proxy
>           |- [ INFO  ] Generating post install configuration file
> '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
>           |- [ INFO  ] Stage: Transaction commit
>           |- [ INFO  ] Stage: Closing up
>           |-
>           |-           --== SUMMARY ==--
>           |-
>           |- [WARNING] Less than 16384MB of memory is available
>           |-           SSH fingerprint:
> 13:dc:47:b9:38:66:15:eb:83:e8:ed:6f:71:cb:d4:ea
>           |-           Internal CA
> D8:4B:4C:74:45:EB:87:C4:8F:FE:DE:02:DF:99:5E:CB:AA:1B:D84
>           |-           Note! If you want to gather statistical information
> you can install Reports and/or DWH:
>           |-               http://www.ovirt.org/Ovirt_DWH
>           |-               http://www.ovirt.org/Ovirt_Reports
>           |-           Web access is enabled at:
>           |-               http://ovirt.lutwyn.org:80/ovirt-engine
>           |-               https://ovirt.lutwyn.org:443/ovirt-engine
>           |-           Please use the user 'admin at internal' and password
> specified in order to login
>           |-
>           |-           --== END OF SUMMARY ==--
>           |-
>           |- [ INFO  ] Starting engine service
>           |- [ INFO  ] Restarting httpd
>           |- [ INFO  ] Restarting ovirt-vmconsole proxy service
>           |- [ INFO  ] Stage: Clean up
>           |-           Log file is located at
> /var/log/ovirt-engine/setup/ovirt-engine-setup-20160430234733-h0rns4.log
>           |- [ INFO  ] Generating answer file
> '/var/lib/ovirt-engine/setup/answers/20160430234907-setup.conf'
>           |- [ INFO  ] Stage: Pre-termination
>           |- [ INFO  ] Stage: Termination
>           |- [ INFO  ] Execution of setup completed successfully
>           |- HE_APPLIANCE_ENGINE_SETUP_SUCCESS
> [ INFO  ] Engine-setup successfully completed
> [ INFO  ] Engine is still unreachable
> [ INFO  ] Engine is still not reachable, waiting...
> [ INFO  ] Engine is still unreachable
> [ INFO  ] Engine is still not reachable, waiting...
> [ INFO  ] Engine replied: DB Up!Welcome to Health Status!
> [ INFO  ] Acquiring internal CA cert from the engine
> [ INFO  ] The following CA certificate is going to be used, please
> immediately interrupt if not correct:
> [ INFO  ] Issuer: C=US, O=lutwyn.org, CN=ovirt.lutwyn.org.36587, Subject:
> C=US, O=lutwyn.org, CN=ovirt.lutwyn.org.36587, Fingerprint (SHA-1):
> D84B4C7445EB87C48FFEDE02DF995ECBAA1BD84
> [ INFO  ] Connecting to the Engine
> [ INFO  ] Waiting for the host to become operational in the engine. This may
> take several minutes...
> [ INFO  ] The VDSM Host is now operational
> [ INFO  ] Saving hosted-engine configuration on the shared storage domain
> [ INFO  ] Shutting down the engine VM
> [ INFO  ] Enabling and starting HA services
> [ INFO  ] Stage: Clean up
> [ INFO  ] Generating answer file
> '/var/lib/ovirt-hosted-engine-setup/answers/answers-20160501015145.conf'
> [ INFO  ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
> [ INFO  ] Stage: Pre-termination
> [ INFO  ] Stage: Termination
> [ INFO  ] Hosted Engine successfully set up
> [root at ovirt01 ~]#
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>



-- 
Didi



More information about the Users mailing list