[ovirt-users] hosted engine setup failed for 10 minutes delay.. engine seems alive

Simone Tiraboschi stirabos at redhat.com
Mon May 2 09:14:59 UTC 2016


On Mon, May 2, 2016 at 11:06 AM, Yedidyah Bar David <didi at redhat.com> wrote:
> On Mon, May 2, 2016 at 11:48 AM, Gianluca Cecchi
> <gianluca.cecchi at gmail.com> wrote:
>> On Mon, May 2, 2016 at 9:58 AM, Simone Tiraboschi wrote:
>>>
>>>
>>>
>>> hosted-engine-setup creates a fresh VM and inject a cloud-init script
>>> to configure it and execute there engine-setup to configure the engine
>>> as needed.
>>> Since engine-setup is running on the engine VM triggered by
>>> cloud-init, hosted-engine-setup has no way to really control its
>>> process status so we simply gather its output with a timeout of 10
>>> minutes between each single output line.
>>> In nothing happens within 10 minutes (the value is easily
>>> customizable), hosted-engine-setup thinks that engine-setup is stuck.
>>
>>
>>
>> How can one customize the pre-set timeout?

To set 20 minutes you can pass this
OVEHOSTED_ENGINE/engineSetupTimeout=int:1200


>> Could it be better to ask the user at the end of timeout if he/she wants to
>> wait again, instead of directly fail?
>
> Perhaps, can you please open a bz?

+1

>>> So the issue we have to understood is why this simple command took
>>> more than 10 minutes in your env:
>>> 2016-04-30 17:56:57 DEBUG
>>> otopi.plugins.ovirt_engine_setup.ovirt_engine.config.aaajdbc
>>> plugin.executeRaw:828 execute: ('/usr/bin/ovirt-aaa-jdbc-tool',
>>> '--db-config=/etc/ovirt-engine/aaa/internal.properties', 'user',
>>> 'password-reset', 'admin', '--password=env:pass', '--force',
>>> '--password-valid-to=2216-03-13 17:56:57Z'), executable='None',
>>> cwd='None', env={'LANG': 'en_US.UTF-8', 'SHLVL': '1', 'PYTHONPATH':
>>> '/usr/share/ovirt-engine/setup/bin/..::', 'pass': '**FILTERED**',
>>> 'OVIRT_ENGINE_JAVA_HOME_FORCE': '1', 'PWD': '/',
>>> 'OVIRT_ENGINE_JAVA_HOME': u'/usr/lib/jvm/jre', 'PATH':
>>> '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'OTOPI_LOGFILE':
>>>
>>> '/var/log/ovirt-engine/setup/ovirt-engine-setup-20160430175551-dttt2p.log',
>>> 'OVIRT_JBOSS_HOME': '/usr/share/ovirt-engine-wildfly',
>>> 'OTOPI_EXECDIR': '/'}
>>
>>
>>
>>
>> It seemed quite strange to me too (see below further info on this)
>>
>>>
>>> Can you please check the entropy value on your host?
>>>  cat /proc/sys/kernel/random/entropy_avail
>>>
>>
>> I have not at hand now the server. I'll check soon and report
>> Do you mean entropy of the physical server that will operate as hypervisor?

On the hypervisor

> That's a good question. Simone - do you know if we start the guest with
> virtio-rng?

AFAIK we are not.

> This is another case of [1], perhaps we should reopen it.
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1319827
>
>>
>>
>>>
>>> > As a last question how to clean up things in case I have to start from
>>> > scratch.
>>>
>>> I'd recommend to redeploy from scratch instead of trying fixing it
>>> but, before that, we need to understand the root issue.
>>>
>>
>> So, trying restart the setup with generated answer file I got:
>> 1) if VM still powered on, an error about this condition
>> 2) if VM powered down, an error abut storage domain already in place and
>> restart not supported in this condition.
>>
>> I was able to continue with these steps:
>>
>> a) remove what inside the partially setup self hosted engine storage domain
>> rm -rf /SHE_DOMAIN/*
>> cd SHE_DOMAIN
>> mklost+found
>>
>> b) reboot the hypervisor
>>
>> c) stop vdsmd
>>
>> d) start the setup again with the answer file
>> It seems all went well and this time strangely the step that took more than
>> 10 minutes before lasted less than 2 seconds....
>>
>> I was then able to deploy storage and iso domains without problems and self
>> hosted engine domain correctly detected and imported too.
>> Created two CentOS VMs without problems (6.7 and 7.2).
>>
>> See below the full output of deploy command
>>
>>
>> [root at ovirt01 ~]# hosted-engine --deploy
>> --config-append=/var/lib/ovirt-hosted-engine-setup/answers/answers-20160430200654.conf
>> [ INFO  ] Stage: Initializing
>> [ INFO  ] Generating a temporary VNC password.
>> [ INFO  ] Stage: Environment setup
>>           Configuration files:
>> ['/var/lib/ovirt-hosted-engine-setup/answers/answers-20160430200654.conf']
>>           Log file:
>> /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20160501014326-8frbxk.log
>>           Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos)
>> [ INFO  ] Hardware supports virtualization
>> [ INFO  ] Bridge ovirtmgmt already created
>> [ INFO  ] Stage: Environment packages setup
>> [ INFO  ] Stage: Programs detection
>> [ INFO  ] Stage: Environment setup
>> [ INFO  ] Stage: Environment customization
>>
>>           --== STORAGE CONFIGURATION ==--
>>
>>           During customization use CTRL-D to abort.
>> [ INFO  ] Installing on first host
>>
>>           --== SYSTEM CONFIGURATION ==--
>>
>>
>>           --== NETWORK CONFIGURATION ==--
>>
>>
>>           --== VM CONFIGURATION ==--
>>
>> [ INFO  ] Checking OVF archive content (could take a few minutes depending
>> on archive size)
>> [ INFO  ] Checking OVF XML content (could take a few minutes depending on
>> archive size)
>> [WARNING] OVF does not contain a valid image description, using default.
>>           Enter root password that will be used for the engine appliance
>> (leave it empty to skip):
>>           Confirm appliance root password:
>> [ INFO  ] The engine VM will be configured to use 192.168.1.212/24
>>           The following CPU types are supported by this host:
>>           - model_Broadwell-noTSX: Intel Broadwell-noTSX Family
>>           - model_Haswell-noTSX: Intel Haswell-noTSX Family
>>           - model_SandyBridge: Intel SandyBridge Family
>>           - model_Westmere: Intel Westmere Family
>>           - model_Nehalem: Intel Nehalem Family
>>           - model_Penryn: Intel Penryn Family
>>           - model_Conroe: Intel Conroe Family
>> [WARNING] Minimum requirements for CPUs not met
>> [WARNING] Minimum requirements for disk size not met
>>
>>           --== HOSTED ENGINE CONFIGURATION ==--
>>
>>           Enter the name which will be used to identify this host inside the
>> Administrator Portal [hosted_engine_1]:
>>           Enter 'admin at internal' user password that will be used for
>> accessing the Administrator Portal:
>>           Confirm 'admin at internal' user password:
>> [ INFO  ] Stage: Setup validation
>>
>>           --== CONFIGURATION PREVIEW ==--
>>
>>           Bridge interface                   : eth0
>>           Engine FQDN                        : ovirt.lutwyn.org
>>           Bridge name                        : ovirtmgmt
>>           Host address                       : ovirt01.lutwyn.org
>>           SSH daemon port                    : 22
>>           Firewall manager                   : iptables
>>           Gateway address                    : 192.168.1.1
>>           Host name for web application      : hosted_engine_1
>>           Host ID                            : 1
>>           Image size GB                      : 10
>>           GlusterFS Share Name               : hosted_engine_glusterfs
>>           GlusterFS Brick Provisioning       : False
>>           Storage connection                 :
>> ovirt01.lutwyn.org:/SHE_DOMAIN
>>           Console type                       : vnc
>>           Memory size MB                     : 8192
>>           MAC address                        : 00:16:3e:2e:93:15
>>           Boot type                          : disk
>>           Number of CPUs                     : 1
>>           OVF archive (for disk boot)        :
>> /usr/share/ovirt-engine-appliance/ovirt-engine-appliance-3.6-20160420.1.el7.centos.ova
>>           Restart engine VM after engine-setup: True
>>           CPU Type                           : model_Broadwell-noTSX
>> [ INFO  ] Stage: Transaction setup
>> [ INFO  ] Stage: Misc configuration
>> [ INFO  ] Stage: Package installation
>> [ INFO  ] Stage: Misc configuration
>> [ INFO  ] Configuring libvirt
>> [ INFO  ] Configuring VDSM
>> [ INFO  ] Starting vdsmd
>> [ INFO  ] Waiting for VDSM hardware info
>> [ INFO  ] Creating Storage Domain
>> [ INFO  ] Creating Storage Pool
>> [ INFO  ] Connecting Storage Pool
>> [ INFO  ] Verifying sanlock lockspace initialization
>> [ INFO  ] Creating Image for 'hosted-engine.lockspace' ...
>> [ INFO  ] Image for 'hosted-engine.lockspace' created successfully
>> [ INFO  ] Creating Image for 'hosted-engine.metadata' ...
>> [ INFO  ] Image for 'hosted-engine.metadata' created successfully
>> [ INFO  ] Creating VM Image
>> [ INFO  ] Extracting disk image from OVF archive (could take a few minutes
>> depending on archive size)
>> [ INFO  ] Validating pre-allocated volume size
>> [ INFO  ] Uploading volume to data domain (could take a few minutes
>> depending on archive size)
>> [ INFO  ] Image successfully imported from OVF
>> [ INFO  ] Destroying Storage Pool
>> [ INFO  ] Start monitoring domain
>> [ INFO  ] Configuring VM
>> [ INFO  ] Updating hosted-engine configuration
>> [ INFO  ] Stage: Transaction commit
>> [ INFO  ] Stage: Closing up
>> [ INFO  ] Creating VM
>>           You can now connect to the VM with the following command:
>>           /bin/remote-viewer vnc://localhost:5900
>>           Use temporary password "4699ouyH" to connect to vnc console.
>>           Please note that in order to use remote-viewer you need to be able
>> to run graphical applications.
>>           This means that if you are using ssh you have to supply the -Y
>> flag (enables trusted X11 forwarding).
>>           Otherwise you can run the command from a terminal in your
>> preferred desktop environment.
>>           If you cannot run graphical applications you can connect to the
>> graphic console from another host or connect to the serial console using the
>> following command:
>>           socat
>> UNIX-CONNECT:/var/run/ovirt-vmconsole-console/4dae6016-ff01-4a1f-87ea-09f049d0e58c.sock,user=ovirt-vmconsole
>> STDIO,raw,echo=0,escape=1
>>           Please ensure that your Guest OS is properly configured to support
>> serial console according to your distro documentation.
>>           Follow
>> http://www.ovirt.org/Serial_Console_Setup#I_need_to_access_the_console_the_old_way
>> for more info.
>>           If you need to reboot the VM you will need to start it manually
>> using the command:
>>           hosted-engine --vm-start
>>           You can then set a temporary password using the command:
>>           hosted-engine --add-console-password
>> [ INFO  ] Running engine-setup on the appliance
>>           |- [ INFO  ] Stage: Initializing
>>           |- [ INFO  ] Stage: Environment setup
>>           |-           Configuration files:
>> ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf',
>> '/etc/ovirt-engine-setup.conf.d/10-packaging.conf',
>> '/root/ovirt-engine-answers', '/root/heanswers.conf']
>>           |-           Log file:
>> /var/log/ovirt-engine/setup/ovirt-engine-setup-20160430234733-h0rns4.log
>>           |-           Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos)
>>           |- [ INFO  ] Stage: Environment packages setup
>>           |- [ INFO  ] Stage: Programs detection
>>           |- [ INFO  ] Stage: Environment setup
>>           |- [ INFO  ] Stage: Environment customization
>>           |-
>>           |-           --== PRODUCT OPTIONS ==--
>>           |-
>>           |-
>>           |-           --== PACKAGES ==--
>>           |-
>>           |-
>>           |-           --== ALL IN ONE CONFIGURATION ==--
>>           |-
>>           |-
>>           |-           --== NETWORK CONFIGURATION ==--
>>           |-
>>           |- [ INFO  ] firewalld will be configured as firewall manager.
>>           |-
>>           |-           --== DATABASE CONFIGURATION ==--
>>           |-
>>           |-
>>           |-           --== OVIRT ENGINE CONFIGURATION ==--
>>           |-
>>           |-
>>           |-           --== STORAGE CONFIGURATION ==--
>>           |-
>>           |-
>>           |-           --== PKI CONFIGURATION ==--
>>           |-
>>           |-
>>           |-           --== APACHE CONFIGURATION ==--
>>           |-
>>           |-
>>           |-           --== SYSTEM CONFIGURATION ==--
>>           |-
>>           |-
>>           |-           --== MISC CONFIGURATION ==--
>>           |-
>>           |-
>>           |-           --== END OF CONFIGURATION ==--
>>           |-
>>           |- [ INFO  ] Stage: Setup validation
>>           |- [WARNING] Cannot validate host name settings, reason: resolved
>> host does not match any of the local addresses
>>           |- [WARNING] Less than 16384MB of memory is available
>>           |-
>>           |-           --== CONFIGURATION PREVIEW ==--
>>           |-
>>           |-           Application mode                        : virt
>>           |-           Default SAN wipe after delete           : False
>>           |-           Firewall manager                        : firewalld
>>           |-           Update Firewall                         : True
>>           |-           Host FQDN                               :
>> ovirt.lutwyn.org
>>           |-           Engine database secured connection      : False
>>           |-           Engine database host                    : localhost
>>           |-           Engine database user name               : engine
>>           |-           Engine database name                    : engine
>>           |-           Engine database port                    : 5432
>>           |-           Engine database host name validation    : False
>>           |-           Engine installation                     : True
>>           |-           PKI organization                        : lutwyn.org
>>           |-           Configure local Engine database         : True
>>           |-           Set application as default page         : True
>>           |-           Configure Apache SSL                    : True
>>           |-           Configure VMConsole Proxy               : True
>>           |-           Engine Host FQDN                        :
>> ovirt.lutwyn.org
>>           |-           Configure WebSocket Proxy               : True
>>           |- [ INFO  ] Stage: Transaction setup
>>           |- [ INFO  ] Stopping engine service
>>           |- [ INFO  ] Stopping ovirt-fence-kdump-listener service
>>           |- [ INFO  ] Stopping websocket-proxy service
>>           |- [ INFO  ] Stage: Misc configuration
>>           |- [ INFO  ] Stage: Package installation
>>           |- [ INFO  ] Stage: Misc configuration
>>           |- [ INFO  ] Initializing PostgreSQL
>>           |- [ INFO  ] Creating PostgreSQL 'engine' database
>>           |- [ INFO  ] Configuring PostgreSQL
>>           |- [ INFO  ] Creating/refreshing Engine database schema
>>           |- [ INFO  ] Creating/refreshing Engine 'internal' domain database
>> schema
>>           |- [ INFO  ] Upgrading CA
>>           |- [ INFO  ] Creating CA
>>           |- [ INFO  ] Setting up ovirt-vmconsole proxy helper PKI artifacts
>>           |- [ INFO  ] Setting up ovirt-vmconsole SSH PKI artifacts
>>           |- [ INFO  ] Configuring WebSocket Proxy
>>           |- [ INFO  ] Generating post install configuration file
>> '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
>>           |- [ INFO  ] Stage: Transaction commit
>>           |- [ INFO  ] Stage: Closing up
>>           |-
>>           |-           --== SUMMARY ==--
>>           |-
>>           |- [WARNING] Less than 16384MB of memory is available
>>           |-           SSH fingerprint:
>> 13:dc:47:b9:38:66:15:eb:83:e8:ed:6f:71:cb:d4:ea
>>           |-           Internal CA
>> D8:4B:4C:74:45:EB:87:C4:8F:FE:DE:02:DF:99:5E:CB:AA:1B:D84
>>           |-           Note! If you want to gather statistical information
>> you can install Reports and/or DWH:
>>           |-               http://www.ovirt.org/Ovirt_DWH
>>           |-               http://www.ovirt.org/Ovirt_Reports
>>           |-           Web access is enabled at:
>>           |-               http://ovirt.lutwyn.org:80/ovirt-engine
>>           |-               https://ovirt.lutwyn.org:443/ovirt-engine
>>           |-           Please use the user 'admin at internal' and password
>> specified in order to login
>>           |-
>>           |-           --== END OF SUMMARY ==--
>>           |-
>>           |- [ INFO  ] Starting engine service
>>           |- [ INFO  ] Restarting httpd
>>           |- [ INFO  ] Restarting ovirt-vmconsole proxy service
>>           |- [ INFO  ] Stage: Clean up
>>           |-           Log file is located at
>> /var/log/ovirt-engine/setup/ovirt-engine-setup-20160430234733-h0rns4.log
>>           |- [ INFO  ] Generating answer file
>> '/var/lib/ovirt-engine/setup/answers/20160430234907-setup.conf'
>>           |- [ INFO  ] Stage: Pre-termination
>>           |- [ INFO  ] Stage: Termination
>>           |- [ INFO  ] Execution of setup completed successfully
>>           |- HE_APPLIANCE_ENGINE_SETUP_SUCCESS
>> [ INFO  ] Engine-setup successfully completed
>> [ INFO  ] Engine is still unreachable
>> [ INFO  ] Engine is still not reachable, waiting...
>> [ INFO  ] Engine is still unreachable
>> [ INFO  ] Engine is still not reachable, waiting...
>> [ INFO  ] Engine replied: DB Up!Welcome to Health Status!
>> [ INFO  ] Acquiring internal CA cert from the engine
>> [ INFO  ] The following CA certificate is going to be used, please
>> immediately interrupt if not correct:
>> [ INFO  ] Issuer: C=US, O=lutwyn.org, CN=ovirt.lutwyn.org.36587, Subject:
>> C=US, O=lutwyn.org, CN=ovirt.lutwyn.org.36587, Fingerprint (SHA-1):
>> D84B4C7445EB87C48FFEDE02DF995ECBAA1BD84
>> [ INFO  ] Connecting to the Engine
>> [ INFO  ] Waiting for the host to become operational in the engine. This may
>> take several minutes...
>> [ INFO  ] The VDSM Host is now operational
>> [ INFO  ] Saving hosted-engine configuration on the shared storage domain
>> [ INFO  ] Shutting down the engine VM
>> [ INFO  ] Enabling and starting HA services
>> [ INFO  ] Stage: Clean up
>> [ INFO  ] Generating answer file
>> '/var/lib/ovirt-hosted-engine-setup/answers/answers-20160501015145.conf'
>> [ INFO  ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
>> [ INFO  ] Stage: Pre-termination
>> [ INFO  ] Stage: Termination
>> [ INFO  ] Hosted Engine successfully set up
>> [root at ovirt01 ~]#
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
>
>
> --
> Didi



More information about the Users mailing list