Re: [ovirt-users] unsuccessful hosted engine install

As requested,

The output of ovirt-hosted-engine-cleanup [root@node-1 ~]# ovirt-hosted-engine-cleanup This will de-configure the host to run ovirt-hosted-engine-setup from scratch. Caution, this operation should be used with care. Are you sure you want to proceed? [y/n] y -=== Destroy hosted-engine VM ===- You must run deploy first -=== Stop HA services ===- -=== Shutdown sanlock ===- shutdown force 1 wait 0 shutdown done 0 -=== Disconnecting the hosted-engine storage domain ===- You must run deploy first -=== De-configure VDSM networks ===- -=== Stop other services ===- -=== De-configure external daemons ===- -=== Removing configuration files ===- ? /etc/init/libvirtd.conf already missing - removing /etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml ? /etc/ovirt-hosted-engine/answers.conf already missing ? /etc/ovirt-hosted-engine/hosted-engine.conf already missing - removing /etc/vdsm/vdsm.conf - removing /etc/pki/vdsm/certs/cacert.pem - removing /etc/pki/vdsm/certs/vdsmcert.pem - removing /etc/pki/vdsm/keys/vdsmkey.pem - removing /etc/pki/vdsm/libvirt-spice/ca-cert.pem - removing /etc/pki/vdsm/libvirt-spice/ca-key.pem - removing /etc/pki/vdsm/libvirt-spice/server-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-key.pem ? /etc/pki/CA/cacert.pem already missing ? /etc/pki/libvirt/*.pem already missing ? /etc/pki/libvirt/private/*.pem already missing ? /etc/pki/ovirt-vmconsole/*.pem already missing - removing /var/cache/libvirt/qemu ? /var/run/ovirt-hosted-engine-ha/* already missing [root@node-1 ~]# Output of sanlock client status: [root@node-1 ~]# sanlock client status [root@node-1 ~]# Thank you for your help! On Mon, Jun 5, 2017 at 7:25 AM, Simone Tiraboschi <stirabos(a)redhat.com&gt; wrote: > > > On Mon, Jun 5, 2017 at 3:57 PM, Brendan Hartzell <mrrex4(a)gmail.com&gt; > wrote: > >> After letting this sit for a few days, does anyone have any ideas as to >> how to deal with my situation? Would anyone like me to send the SOS report >> directly to them? It's a 9MB file. >> >> If nothing comes up, I'm going to try and sift through the SOS report >> tonight, but I won't know what I'm trying to find. >> >> Thank you for any and all help. >> >> On Thu, Jun 1, 2017 at 1:15 AM, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; >> wrote: >> >>> >>> >>> On Thu, Jun 1, 2017 at 6:36 AM, Brendan Hartzell <mrrex4(a)gmail.com&gt; >>> wrote: >>> >>>> Ran the 4 commands listed above, no errors on the screen. >>>> >>>> Started the hosted-engine standard setup from the web-UI. >>>> >>>> Using iSCSI for the storage. >>>> >>>> Using mostly default options, I got these errors in the web-UI. >>>> >>>> Error creating Volume Group: Failed to initialize physical device: >>>> ("[u'/dev/mapper/36589cfc000000de7482638fcfcebbbb4']",) >>>> Failed to execute stage 'Misc configuration': Failed to initialize >>>> physical device: ("[u'/dev/mapper/36589cfc00000 >>>> 0de7482638fcfcebbbb4']",) >>>> Hosted Engine deployment failed: this system is not reliable, please >>>> check the issue,fix and redeploy >>>> >>>> I rebuilt my iSCSI (I don't think I cleaned it up from a previous >>>> install). >>>> Re-ran the above 4 commands. >>>> Restarted hosted engine standard setup from web-UI. >>>> Install moved past "Connecting Storage Pool" so I believe the above >>>> was my fault. >>>> >>>> These are the last messages displayed on the web-UI. >>>> Creating Storage Pool >>>> Connecting Storage Pool >>>> Verifying sanlock lockspace initialization >>>> Creating Image for 'hosted-engine.lockspace' ... >>>> Image for 'hosted-engine.lockspace' created successfully >>>> Creating Image for 'hosted-engine.metadata' ... >>>> Image for 'hosted-engine.metadata' created successfully >>>> Creating VM Image >>>> Extracting disk image from OVF archive (could take a few minutes >>>> depending on archive size) >>>> Validating pre-allocated volume size >>>> Uploading volume to data domain (could take a few minutes depending on >>>> archive size) >>>> >>>> At the host terminal, I got the error "watchdog watchdog0: watchdog >>>> did not stop!" >>>> Then the host restarted. >>>> >>> >>> Simone, can you help here? >>> >>> > Ok, sorry for the delay. > The second installation attempt seams fine but it seams that > ovirt-hosted-engine-cleanup failed stopping sanlock and so the > watchdog kick in rebooting your system in the middle of deployment attempt. > > could you please post the output of > ovirt-hosted-engine-cleanup > sanlock client status > ? > > > >> >>> >>> >>>> >>>> This is as far as I've gotten in previous attempts. >>>> >>>> Attaching the hosted-engine-setup log. >>>> >>>> The SOS report is 9MB and the ovirt users group will drop the email. >>>> >>>> On Wed, May 31, 2017 at 6:59 AM, Sandro Bonazzola <sbonazzo(a)redhat.com >>>> > wrote: >>>> >>>>> >>>>> >>>>> On Wed, May 31, 2017 at 3:10 PM, Brendan Hartzell <mrrex4(a)gmail.com&gt; >>>>> wrote: >>>>> >>>>>> Now that you have identified the problem, should I run the following >>>>>> commands and send you another SOS? >>>>>> >>>>>> ovirt-hosted-engine-cleanup >>>>>> vdsm-tool configure --force >>>>>> systemctl restart libvirtd >>>>>> systemctl restart vdsm >>>>>> >>>>>> Or is there a different plan in mind? >>>>>> >>>>> >>>>> I would have expected someone from virt team to follow up for further >>>>> investigations :-) >>>>> above commands should work. >>>>> >>>>> >>>>> >>>>>> >>>>>> Thank you, >>>>>> >>>>>> Brendan >>>>>> >>>>>> On Tue, May 30, 2017 at 11:42 PM, Sandro Bonazzola < >>>>>> sbonazzo(a)redhat.com&gt; wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, May 31, 2017 at 4:45 AM, Brendan Hartzell <mrrex4(a)gmail.com >>>>>>> > wrote: >>>>>>> >>>>>>>> Can you please elaborate about the failure you see here and how >>>>>>>> are you trying to manually partition the host? >>>>>>>> >>>>>>>> Sure, I will start from the beginning. >>>>>>>> - Using: ovirt-node-ng-installer-ovirt-4.1-2017052604 >>>>>>>> <(201)%20705-2604>.iso >>>>>>>> - During installation I setup one of the two interfaces and check >>>>>>>> the box to automatically use the connection. >>>>>>>> - I'm currently providing a host name of node-1.test.net until I >>>>>>>> have a successful process. >>>>>>>> - I configure date and time for my timezone and to use an internal >>>>>>>> NTP server. >>>>>>>> - On Installation Destination, I pick my 128GB USB3.0 SanDisk >>>>>>>> flash drive, check the box that I would like to make additional space, and >>>>>>>> click done. In the reclaim disk space window, I click delete all, and then >>>>>>>> reclaim space. I go back into the Installation Destination, select that I >>>>>>>> will configure partitioning, and click done. The Manual Partitioning >>>>>>>> window opens, I use the option to automatically create mount points. >>>>>>>> >>>>>>> >>>>>>> In this screen, please change partitioning scheme from LVM to LVM >>>>>>> Thin Provisioning: it should solve your following error. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> At this point, /boot is 1024MB, /var is 15GB, / is 88.11 GB, and >>>>>>>> swap is 11.57GB. I then change / to 23.11 GB, update settings, change /var >>>>>>>> to 80GB, update settings again, and click done. I accept the changes and >>>>>>>> begin installation. >>>>>>>> >>>>>>>> I tried these changes based on this article: http://www.ovirt.org/ >>>>>>>> documentation/self-hosted/chap-Deploying_Self-Hosted_Engine/ >>>>>>>> >>>>>>>> The article does say that you can specify a different directory >>>>>>>> than /var/tmp, but I don't recall seeing that option. >>>>>>>> >>>>>>> >>>>>>> If the setup detects not enough space in /var/tmp for extracting >>>>>>> the appliance it will ask about a different directory. >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> After some time, I get the following error: >>>>>>>> There was an error running the kickstart script at line 7. This >>>>>>>> is a fatal error and installation will be aborted. The details of this >>>>>>>> error are: >>>>>>>> >>>>>>>> [INFO] Trying to create a manageable base from '/' >>>>>>>> [ERROR] LVM Thin Provisioning partitioning scheme is required. >>>>>>>> For autoinstall via Kickstart with LVM Thin Provisioning check options >>>>>>>> --thinpool and --grow. Please consult documentation for details. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> ^^ this one should be solved by the LVM Thin Provisioning scheme >>>>>>> mentioned above.. >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Traceback (most recent call last): >>>>>>>> File "/usr/lib64/python2.7/runpy.py", line 162, in >>>>>>>> _run_module_as_main "__main__", fname, loader, pkg_name) >>>>>>>> File "/usr/lib64/python2.7runpy.py", line 72, in _run_code exec >>>>>>>> code in run_globals >>>>>>>> File "/usr/lib/python2.7/site-packages/imgbased/__main__.py", >>>>>>>> line 51, in <module> CliApplication() >>>>>>>> File "/usr/lib/python2.7/site-packages/imgbased/__init__.py", >>>>>>>> line 82, in CliApplication() >>>>>>>> File "/usr/lib/python2.7/site-packages/imgbased/hooks.py", line >>>>>>>> 120, in emit cb(self.context, *args) >>>>>>>> File "/usr/lib/python2.7/site-packages/imgbased/plugins/core.py", >>>>>>>> line 169, in post_argparse layout.initialize(args.source, args.init_nvr) >>>>>>>> File "/usr/lib/python2.7/site-packages/imgbased/plugins/core.py", >>>>>>>> line 216, in initialize self.app.imgbase.init_layout_from(source, >>>>>>>> init_nvr) >>>>>>>> File "/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line >>>>>>>> 271, in init_layout_from self.init_tags_on(existing_lv) >>>>>>>> File "/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line >>>>>>>> 243, in init_tags_on pool = lv.thinpool() >>>>>>>> File "/usr/lib/python2.7/site-packages/imgbased/lvm.py", line >>>>>>>> 250, in thinpool raise MissingLvmThinPool() imgbased.lvm.MissingLvmThinPoo >>>>>>>> l >>>>>>>> >>>>>>>> At this point, the only option is to exit the installer. >>>>>>>> >>>>>>>> **************************** >>>>>>>> >>>>>>>> Being this a new install, please use 4.1. oVirt 4.0 is not >>>>>>>> supported anymore. >>>>>>>> >>>>>>>> Not a problem. >>>>>>>> >>>>>>>> **************************** >>>>>>>> >>>>>>>> Can you please provide hosted engine setup logs or better a full >>>>>>>> sos report? (sosreport -a) >>>>>>>> >>>>>>>> Again, the process I'm following: >>>>>>>> - Using: ovirt-node-ng-installer-ovirt-4.1-2017052604 >>>>>>>> <(201)%20705-2604>.iso >>>>>>>> - During installation I setup one of the two interfaces and check >>>>>>>> the box to automatically use the connection. >>>>>>>> - I'm currently providing a host name of node-1.test.net until I >>>>>>>> have a successful process. >>>>>>>> - I configure date and time for my timezone and to use an internal >>>>>>>> NTP server. >>>>>>>> - On Installation Destination, I pick my 128GB USB3.0 SanDisk >>>>>>>> flash drive, check the box that I would like to make additional space, and >>>>>>>> click done. In the reclaim disk space window, I click delete all, and then >>>>>>>> reclaim space. >>>>>>>> - Begin Installation and set a root password. >>>>>>>> - Perform a yum update - no packages marked for update (as >>>>>>>> expected) >>>>>>>> - Use vi to update /etc/hosts with a reference for node-1.test.net >>>>>>>> and engine.test.net >>>>>>>> - First attempt at hosted-engine from web-UI >>>>>>>> - Setup downloads and installs ovirt-engine-applianc >>>>>>>> e-4.1-20170523.1.el7.centos.noarch.rpm >>>>>>>> *Failed to execute stage 'Environment setup': Failed to >>>>>>>> reconfigure libvirt for VDSM >>>>>>>> *Hosted Engine deployment failed >>>>>>>> - Attached SOS report >>>>>>>> The checksum is: aa56097edc0b63c49caaf1a1fde021bc >>>>>>>> >>>>>>>> At this point, I would run ovirt-hosted-engine-cleanup and I >>>>>>>> would get further along in the install process. However, because this is a >>>>>>>> fresh install, I'm going to leave things here for now so you can review the >>>>>>>> SOS. >>>>>>>> >>>>>>> >>>>>>> Thanks for the SOS report! >>>>>>> Hosted Engine setup fails on: >>>>>>> >>>>>>> 2017-05-30 19:24:39 DEBUG otopi.plugins.gr_he_setup.system.vdsmenv >>>>>>> plugin.execute:921 execute-output: ('/bin/vdsm-tool', 'configure', >>>>>>> '--force') stdout: >>>>>>> >>>>>>> Checking configuration status... >>>>>>> >>>>>>> Current revision of multipath.conf detected, preserving >>>>>>> lvm is configured for vdsm >>>>>>> libvirt is already configured for vdsm >>>>>>> SUCCESS: ssl configured to true. No conflicts >>>>>>> >>>>>>> Running configure... >>>>>>> Reconfiguration of libvirt is done. >>>>>>> >>>>>>> 2017-05-30 19:24:39 DEBUG otopi.plugins.gr_he_setup.system.vdsmenv >>>>>>> plugin.execute:926 execute-output: ('/bin/vdsm-tool', 'configure', >>>>>>> '--force') stderr: >>>>>>> Error: ServiceOperationError: _systemctlStart failed >>>>>>> Job for libvirtd.service failed because the control process exited >>>>>>> with error code. See "systemctl status libvirtd.service" and "journalctl >>>>>>> -xe" for details. >>>>>>> >>>>>>> At the same time journalctl shows: >>>>>>> >>>>>>> May 30 19:24:39 node-1.test.net libvirtd[20954]: libvirt version: >>>>>>> 2.0.0, package: 10.el7_3.5 (CentOS BuildSystem < >>>>>>> http://bugs.centos.org>;, 2017-03-03-02:09:45, c1bm.rdu2.centos.org) >>>>>>> May 30 19:24:39 node-1.test.net libvirtd[20954]: hostname: >>>>>>> node-1.test.net >>>>>>> May 30 19:24:39 node-1.test.net libvirtd[20954]: The server >>>>>>> certificate /etc/pki/vdsm/certs/vdsmcert.pem is not yet active >>>>>>> May 30 19:24:39 node-1.test.net systemd[1]: libvirtd.service: main >>>>>>> process exited, code=exited, status=6/NOTCONFIGURED >>>>>>> May 30 19:24:39 node-1.test.net systemd[1]: Failed to start >>>>>>> Virtualization daemon. >>>>>>> May 30 19:24:39 node-1.test.net systemd[1]: Unit libvirtd.service >>>>>>> entered failed state. >>>>>>> May 30 19:24:39 node-1.test.net systemd[1]: libvirtd.service >>>>>>> failed. >>>>>>> May 30 19:24:39 node-1.test.net systemd[1]: libvirtd.service >>>>>>> holdoff time over, scheduling restart. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> **************************** >>>>>>>> >>>>>>>> I'd like to understand the issues you faced before suggesting to >>>>>>>> restart from scratch. >>>>>>>> >>>>>>>> Too late... I did two re-installs to get a more accurate account >>>>>>>> of my install process for above. >>>>>>>> >>>>>>>> **************************** >>>>>>>> >>>>>>>> Thank you for your help! >>>>>>>> >>>>>>>> Brendan >>>>>>>> >>>>>>>> On Tue, May 30, 2017 at 12:17 AM, Sandro Bonazzola < >>>>>>>> sbonazzo(a)redhat.com&gt; wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, May 30, 2017 at 6:49 AM, Brendan Hartzell < >>>>>>>>> mrrex4(a)gmail.com&gt; wrote: >>>>>>>>> >>>>>>>>>> oVirt users list, >>>>>>>>>> >>>>>>>>>> Long story short, I've been spending weeks on this project for >>>>>>>>>> my home lab with no success. >>>>>>>>>> >>>>>>>>>> I would like to successfully install two nodes that host a >>>>>>>>>> highly available engine with an iSCSI storage back-end. >>>>>>>>>> >>>>>>>>>> I have read through most, if not all, of the guides on ovirt.org >>>>>>>>>> with no substantial help. >>>>>>>>>> >>>>>>>>>> Successfully, I have done the following: >>>>>>>>>> Install oVirt Engine on a bare metal system, added a node, and >>>>>>>>>> started exploring - not desired. >>>>>>>>>> Install oVirt Node 4.0.6 on a bare metal system - fails if >>>>>>>>>> partitions are not done automatically. >>>>>>>>>> Install oVirt Node 4.1.2 on a bare metal system - fails if >>>>>>>>>> partitions are not done automatically. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Can you please elaborate about the failure you see here and how >>>>>>>>> are you trying to manually partition the host? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> My process after installing a Node: >>>>>>>>>> Run a yum update - just to be sure, but I am using latest iso >>>>>>>>>> images from downloads section. >>>>>>>>>> Edit /etc/hosts for local name resolution - the goal is to host >>>>>>>>>> DNS as a virtual machine, eventually. >>>>>>>>>> On 4.1 if I install ovirt-engine-appliance from yum, it does >>>>>>>>>> simplify one step in the hosted engine setup. If I do this on 4.0 it >>>>>>>>>> discards the image and uses the default. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Being this a new install, please use 4.1. oVirt 4.0 is not >>>>>>>>> supported anymore. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 4.1 the hosted engine setup fails immediately unless I run >>>>>>>>>> the hosted engine cleanup from the shell. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Can you please provide hosted engine setup logs or better a full >>>>>>>>> sos report? (sosreport -a) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> If I do this, I can typically get to the point of installing. >>>>>>>>>> >>>>>>>>>> When I do get to the installation phase, I get to a point just >>>>>>>>>> after extracting the OVA that I get a message on the shell saying something >>>>>>>>>> about the watchdog running the whole time and then the node reboots. >>>>>>>>>> >>>>>>>>>> I found one email thread that sounded like my issue and >>>>>>>>>> suggested the following commands: >>>>>>>>>> vdsm-tool configure --force >>>>>>>>>> systemctl restart libvirtd >>>>>>>>>> systemctl restart vdsmd >>>>>>>>>> >>>>>>>>>> Unfortunately, these commands did not help my situation like the >>>>>>>>>> other individual. >>>>>>>>>> >>>>>>>>>> What log file would everyone like to see first? Given that I >>>>>>>>>> still consider myself relatively new to Linux, please identify the path for >>>>>>>>>> the log file requested. >>>>>>>>>> >>>>>>>>> >>>>>>>>> See above >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Also, because I plan on performing a clean install for this >>>>>>>>>> thread using my process from above (I'm not expecting my outcome to be any >>>>>>>>>> different), are there any tips and tricks that might result in a success? >>>>>>>>>> >>>>>>>>> >>>>>>>>> I'd like to understand the issues you faced before suggesting to >>>>>>>>> restart from scratch. >>>>>>>>> Adding some people who may help as well. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thank you for any and all help, >>>>>>>>>> Brendan >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list >>>>>>>>>> Users(a)ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> SANDRO BONAZZOLA >>>>>>>>> >>>>>>>>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION >>>>>>>>> R&D >>>>>>>>> >>>>>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>>>>> <https://red.ht/sig> >>>>>>>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> SANDRO BONAZZOLA >>>>>>> >>>>>>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D >>>>>>> >>>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>>> <https://red.ht/sig> >>>>>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> SANDRO BONAZZOLA >>>>> >>>>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D >>>>> >>>>> Red Hat EMEA <https://www.redhat.com/> >>>>> <https://red.ht/sig> >>>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>>>> >>>> >>>> >>> >>> >>> -- >>> >>> SANDRO BONAZZOLA >>> >>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D >>> >>> Red Hat EMEA <https://www.redhat.com/> >>> <https://red.ht/sig> >>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>> >> >> >