[ovirt-users] hosted engine setup failed for 10 minutes delay.. engine seems alive
Yedidyah Bar David
didi at redhat.com
Mon May 2 09:06:05 UTC 2016
On Mon, May 2, 2016 at 11:48 AM, Gianluca Cecchi
<gianluca.cecchi at gmail.com> wrote:
> On Mon, May 2, 2016 at 9:58 AM, Simone Tiraboschi wrote:
>>
>>
>>
>> hosted-engine-setup creates a fresh VM and inject a cloud-init script
>> to configure it and execute there engine-setup to configure the engine
>> as needed.
>> Since engine-setup is running on the engine VM triggered by
>> cloud-init, hosted-engine-setup has no way to really control its
>> process status so we simply gather its output with a timeout of 10
>> minutes between each single output line.
>> In nothing happens within 10 minutes (the value is easily
>> customizable), hosted-engine-setup thinks that engine-setup is stuck.
>
>
>
> How can one customize the pre-set timeout?
> Could it be better to ask the user at the end of timeout if he/she wants to
> wait again, instead of directly fail?
Perhaps, can you please open a bz?
>
>
>>
>> So the issue we have to understood is why this simple command took
>> more than 10 minutes in your env:
>> 2016-04-30 17:56:57 DEBUG
>> otopi.plugins.ovirt_engine_setup.ovirt_engine.config.aaajdbc
>> plugin.executeRaw:828 execute: ('/usr/bin/ovirt-aaa-jdbc-tool',
>> '--db-config=/etc/ovirt-engine/aaa/internal.properties', 'user',
>> 'password-reset', 'admin', '--password=env:pass', '--force',
>> '--password-valid-to=2216-03-13 17:56:57Z'), executable='None',
>> cwd='None', env={'LANG': 'en_US.UTF-8', 'SHLVL': '1', 'PYTHONPATH':
>> '/usr/share/ovirt-engine/setup/bin/..::', 'pass': '**FILTERED**',
>> 'OVIRT_ENGINE_JAVA_HOME_FORCE': '1', 'PWD': '/',
>> 'OVIRT_ENGINE_JAVA_HOME': u'/usr/lib/jvm/jre', 'PATH':
>> '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'OTOPI_LOGFILE':
>>
>> '/var/log/ovirt-engine/setup/ovirt-engine-setup-20160430175551-dttt2p.log',
>> 'OVIRT_JBOSS_HOME': '/usr/share/ovirt-engine-wildfly',
>> 'OTOPI_EXECDIR': '/'}
>
>
>
>
> It seemed quite strange to me too (see below further info on this)
>
>>
>> Can you please check the entropy value on your host?
>> cat /proc/sys/kernel/random/entropy_avail
>>
>
> I have not at hand now the server. I'll check soon and report
> Do you mean entropy of the physical server that will operate as hypervisor?
That's a good question. Simone - do you know if we start the guest with
virtio-rng?
This is another case of [1], perhaps we should reopen it.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1319827
>
>
>>
>> > As a last question how to clean up things in case I have to start from
>> > scratch.
>>
>> I'd recommend to redeploy from scratch instead of trying fixing it
>> but, before that, we need to understand the root issue.
>>
>
> So, trying restart the setup with generated answer file I got:
> 1) if VM still powered on, an error about this condition
> 2) if VM powered down, an error abut storage domain already in place and
> restart not supported in this condition.
>
> I was able to continue with these steps:
>
> a) remove what inside the partially setup self hosted engine storage domain
> rm -rf /SHE_DOMAIN/*
> cd SHE_DOMAIN
> mklost+found
>
> b) reboot the hypervisor
>
> c) stop vdsmd
>
> d) start the setup again with the answer file
> It seems all went well and this time strangely the step that took more than
> 10 minutes before lasted less than 2 seconds....
>
> I was then able to deploy storage and iso domains without problems and self
> hosted engine domain correctly detected and imported too.
> Created two CentOS VMs without problems (6.7 and 7.2).
>
> See below the full output of deploy command
>
>
> [root at ovirt01 ~]# hosted-engine --deploy
> --config-append=/var/lib/ovirt-hosted-engine-setup/answers/answers-20160430200654.conf
> [ INFO ] Stage: Initializing
> [ INFO ] Generating a temporary VNC password.
> [ INFO ] Stage: Environment setup
> Configuration files:
> ['/var/lib/ovirt-hosted-engine-setup/answers/answers-20160430200654.conf']
> Log file:
> /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20160501014326-8frbxk.log
> Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos)
> [ INFO ] Hardware supports virtualization
> [ INFO ] Bridge ovirtmgmt already created
> [ INFO ] Stage: Environment packages setup
> [ INFO ] Stage: Programs detection
> [ INFO ] Stage: Environment setup
> [ INFO ] Stage: Environment customization
>
> --== STORAGE CONFIGURATION ==--
>
> During customization use CTRL-D to abort.
> [ INFO ] Installing on first host
>
> --== SYSTEM CONFIGURATION ==--
>
>
> --== NETWORK CONFIGURATION ==--
>
>
> --== VM CONFIGURATION ==--
>
> [ INFO ] Checking OVF archive content (could take a few minutes depending
> on archive size)
> [ INFO ] Checking OVF XML content (could take a few minutes depending on
> archive size)
> [WARNING] OVF does not contain a valid image description, using default.
> Enter root password that will be used for the engine appliance
> (leave it empty to skip):
> Confirm appliance root password:
> [ INFO ] The engine VM will be configured to use 192.168.1.212/24
> The following CPU types are supported by this host:
> - model_Broadwell-noTSX: Intel Broadwell-noTSX Family
> - model_Haswell-noTSX: Intel Haswell-noTSX Family
> - model_SandyBridge: Intel SandyBridge Family
> - model_Westmere: Intel Westmere Family
> - model_Nehalem: Intel Nehalem Family
> - model_Penryn: Intel Penryn Family
> - model_Conroe: Intel Conroe Family
> [WARNING] Minimum requirements for CPUs not met
> [WARNING] Minimum requirements for disk size not met
>
> --== HOSTED ENGINE CONFIGURATION ==--
>
> Enter the name which will be used to identify this host inside the
> Administrator Portal [hosted_engine_1]:
> Enter 'admin at internal' user password that will be used for
> accessing the Administrator Portal:
> Confirm 'admin at internal' user password:
> [ INFO ] Stage: Setup validation
>
> --== CONFIGURATION PREVIEW ==--
>
> Bridge interface : eth0
> Engine FQDN : ovirt.lutwyn.org
> Bridge name : ovirtmgmt
> Host address : ovirt01.lutwyn.org
> SSH daemon port : 22
> Firewall manager : iptables
> Gateway address : 192.168.1.1
> Host name for web application : hosted_engine_1
> Host ID : 1
> Image size GB : 10
> GlusterFS Share Name : hosted_engine_glusterfs
> GlusterFS Brick Provisioning : False
> Storage connection :
> ovirt01.lutwyn.org:/SHE_DOMAIN
> Console type : vnc
> Memory size MB : 8192
> MAC address : 00:16:3e:2e:93:15
> Boot type : disk
> Number of CPUs : 1
> OVF archive (for disk boot) :
> /usr/share/ovirt-engine-appliance/ovirt-engine-appliance-3.6-20160420.1.el7.centos.ova
> Restart engine VM after engine-setup: True
> CPU Type : model_Broadwell-noTSX
> [ INFO ] Stage: Transaction setup
> [ INFO ] Stage: Misc configuration
> [ INFO ] Stage: Package installation
> [ INFO ] Stage: Misc configuration
> [ INFO ] Configuring libvirt
> [ INFO ] Configuring VDSM
> [ INFO ] Starting vdsmd
> [ INFO ] Waiting for VDSM hardware info
> [ INFO ] Creating Storage Domain
> [ INFO ] Creating Storage Pool
> [ INFO ] Connecting Storage Pool
> [ INFO ] Verifying sanlock lockspace initialization
> [ INFO ] Creating Image for 'hosted-engine.lockspace' ...
> [ INFO ] Image for 'hosted-engine.lockspace' created successfully
> [ INFO ] Creating Image for 'hosted-engine.metadata' ...
> [ INFO ] Image for 'hosted-engine.metadata' created successfully
> [ INFO ] Creating VM Image
> [ INFO ] Extracting disk image from OVF archive (could take a few minutes
> depending on archive size)
> [ INFO ] Validating pre-allocated volume size
> [ INFO ] Uploading volume to data domain (could take a few minutes
> depending on archive size)
> [ INFO ] Image successfully imported from OVF
> [ INFO ] Destroying Storage Pool
> [ INFO ] Start monitoring domain
> [ INFO ] Configuring VM
> [ INFO ] Updating hosted-engine configuration
> [ INFO ] Stage: Transaction commit
> [ INFO ] Stage: Closing up
> [ INFO ] Creating VM
> You can now connect to the VM with the following command:
> /bin/remote-viewer vnc://localhost:5900
> Use temporary password "4699ouyH" to connect to vnc console.
> Please note that in order to use remote-viewer you need to be able
> to run graphical applications.
> This means that if you are using ssh you have to supply the -Y
> flag (enables trusted X11 forwarding).
> Otherwise you can run the command from a terminal in your
> preferred desktop environment.
> If you cannot run graphical applications you can connect to the
> graphic console from another host or connect to the serial console using the
> following command:
> socat
> UNIX-CONNECT:/var/run/ovirt-vmconsole-console/4dae6016-ff01-4a1f-87ea-09f049d0e58c.sock,user=ovirt-vmconsole
> STDIO,raw,echo=0,escape=1
> Please ensure that your Guest OS is properly configured to support
> serial console according to your distro documentation.
> Follow
> http://www.ovirt.org/Serial_Console_Setup#I_need_to_access_the_console_the_old_way
> for more info.
> If you need to reboot the VM you will need to start it manually
> using the command:
> hosted-engine --vm-start
> You can then set a temporary password using the command:
> hosted-engine --add-console-password
> [ INFO ] Running engine-setup on the appliance
> |- [ INFO ] Stage: Initializing
> |- [ INFO ] Stage: Environment setup
> |- Configuration files:
> ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf',
> '/etc/ovirt-engine-setup.conf.d/10-packaging.conf',
> '/root/ovirt-engine-answers', '/root/heanswers.conf']
> |- Log file:
> /var/log/ovirt-engine/setup/ovirt-engine-setup-20160430234733-h0rns4.log
> |- Version: otopi-1.4.1 (otopi-1.4.1-1.el7.centos)
> |- [ INFO ] Stage: Environment packages setup
> |- [ INFO ] Stage: Programs detection
> |- [ INFO ] Stage: Environment setup
> |- [ INFO ] Stage: Environment customization
> |-
> |- --== PRODUCT OPTIONS ==--
> |-
> |-
> |- --== PACKAGES ==--
> |-
> |-
> |- --== ALL IN ONE CONFIGURATION ==--
> |-
> |-
> |- --== NETWORK CONFIGURATION ==--
> |-
> |- [ INFO ] firewalld will be configured as firewall manager.
> |-
> |- --== DATABASE CONFIGURATION ==--
> |-
> |-
> |- --== OVIRT ENGINE CONFIGURATION ==--
> |-
> |-
> |- --== STORAGE CONFIGURATION ==--
> |-
> |-
> |- --== PKI CONFIGURATION ==--
> |-
> |-
> |- --== APACHE CONFIGURATION ==--
> |-
> |-
> |- --== SYSTEM CONFIGURATION ==--
> |-
> |-
> |- --== MISC CONFIGURATION ==--
> |-
> |-
> |- --== END OF CONFIGURATION ==--
> |-
> |- [ INFO ] Stage: Setup validation
> |- [WARNING] Cannot validate host name settings, reason: resolved
> host does not match any of the local addresses
> |- [WARNING] Less than 16384MB of memory is available
> |-
> |- --== CONFIGURATION PREVIEW ==--
> |-
> |- Application mode : virt
> |- Default SAN wipe after delete : False
> |- Firewall manager : firewalld
> |- Update Firewall : True
> |- Host FQDN :
> ovirt.lutwyn.org
> |- Engine database secured connection : False
> |- Engine database host : localhost
> |- Engine database user name : engine
> |- Engine database name : engine
> |- Engine database port : 5432
> |- Engine database host name validation : False
> |- Engine installation : True
> |- PKI organization : lutwyn.org
> |- Configure local Engine database : True
> |- Set application as default page : True
> |- Configure Apache SSL : True
> |- Configure VMConsole Proxy : True
> |- Engine Host FQDN :
> ovirt.lutwyn.org
> |- Configure WebSocket Proxy : True
> |- [ INFO ] Stage: Transaction setup
> |- [ INFO ] Stopping engine service
> |- [ INFO ] Stopping ovirt-fence-kdump-listener service
> |- [ INFO ] Stopping websocket-proxy service
> |- [ INFO ] Stage: Misc configuration
> |- [ INFO ] Stage: Package installation
> |- [ INFO ] Stage: Misc configuration
> |- [ INFO ] Initializing PostgreSQL
> |- [ INFO ] Creating PostgreSQL 'engine' database
> |- [ INFO ] Configuring PostgreSQL
> |- [ INFO ] Creating/refreshing Engine database schema
> |- [ INFO ] Creating/refreshing Engine 'internal' domain database
> schema
> |- [ INFO ] Upgrading CA
> |- [ INFO ] Creating CA
> |- [ INFO ] Setting up ovirt-vmconsole proxy helper PKI artifacts
> |- [ INFO ] Setting up ovirt-vmconsole SSH PKI artifacts
> |- [ INFO ] Configuring WebSocket Proxy
> |- [ INFO ] Generating post install configuration file
> '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
> |- [ INFO ] Stage: Transaction commit
> |- [ INFO ] Stage: Closing up
> |-
> |- --== SUMMARY ==--
> |-
> |- [WARNING] Less than 16384MB of memory is available
> |- SSH fingerprint:
> 13:dc:47:b9:38:66:15:eb:83:e8:ed:6f:71:cb:d4:ea
> |- Internal CA
> D8:4B:4C:74:45:EB:87:C4:8F:FE:DE:02:DF:99:5E:CB:AA:1B:D84
> |- Note! If you want to gather statistical information
> you can install Reports and/or DWH:
> |- http://www.ovirt.org/Ovirt_DWH
> |- http://www.ovirt.org/Ovirt_Reports
> |- Web access is enabled at:
> |- http://ovirt.lutwyn.org:80/ovirt-engine
> |- https://ovirt.lutwyn.org:443/ovirt-engine
> |- Please use the user 'admin at internal' and password
> specified in order to login
> |-
> |- --== END OF SUMMARY ==--
> |-
> |- [ INFO ] Starting engine service
> |- [ INFO ] Restarting httpd
> |- [ INFO ] Restarting ovirt-vmconsole proxy service
> |- [ INFO ] Stage: Clean up
> |- Log file is located at
> /var/log/ovirt-engine/setup/ovirt-engine-setup-20160430234733-h0rns4.log
> |- [ INFO ] Generating answer file
> '/var/lib/ovirt-engine/setup/answers/20160430234907-setup.conf'
> |- [ INFO ] Stage: Pre-termination
> |- [ INFO ] Stage: Termination
> |- [ INFO ] Execution of setup completed successfully
> |- HE_APPLIANCE_ENGINE_SETUP_SUCCESS
> [ INFO ] Engine-setup successfully completed
> [ INFO ] Engine is still unreachable
> [ INFO ] Engine is still not reachable, waiting...
> [ INFO ] Engine is still unreachable
> [ INFO ] Engine is still not reachable, waiting...
> [ INFO ] Engine replied: DB Up!Welcome to Health Status!
> [ INFO ] Acquiring internal CA cert from the engine
> [ INFO ] The following CA certificate is going to be used, please
> immediately interrupt if not correct:
> [ INFO ] Issuer: C=US, O=lutwyn.org, CN=ovirt.lutwyn.org.36587, Subject:
> C=US, O=lutwyn.org, CN=ovirt.lutwyn.org.36587, Fingerprint (SHA-1):
> D84B4C7445EB87C48FFEDE02DF995ECBAA1BD84
> [ INFO ] Connecting to the Engine
> [ INFO ] Waiting for the host to become operational in the engine. This may
> take several minutes...
> [ INFO ] The VDSM Host is now operational
> [ INFO ] Saving hosted-engine configuration on the shared storage domain
> [ INFO ] Shutting down the engine VM
> [ INFO ] Enabling and starting HA services
> [ INFO ] Stage: Clean up
> [ INFO ] Generating answer file
> '/var/lib/ovirt-hosted-engine-setup/answers/answers-20160501015145.conf'
> [ INFO ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
> [ INFO ] Stage: Pre-termination
> [ INFO ] Stage: Termination
> [ INFO ] Hosted Engine successfully set up
> [root at ovirt01 ~]#
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
--
Didi
More information about the Users
mailing list