[ovirt-users] Hosted Engine won't start, how to debug?
Martin Sivak
msivak at redhat.com
Mon Apr 23 14:37:19 UTC 2018
Hi,
the VM configuration is stored in the engine database and we generate
the vm.conf using the data the engine exports. So you are right, we
overwrite the file.
You might be able to simply edit the VM in the webadmin and enable the
VNC (or Spice) console there. I am not entirely sure if we allow that
atm, but we might.
If that is not allowed, then you will have to edit the DB. The best
way would be to change the vm_static table's origin (should be 6 -
managed hosted engine) field to 3 - ovirt. The webadmin will then
allow you to edit the VM fully. Then change the origin back to 6. All
that for a VM with vm_name = "HostedEngine".
So the fallback procedure for a default install of webadmin (the
database name = engine):
- put hosted engine into global maintenance
- ssh into the engine VM
- sudo -i -u postgres
- psql engine -c "UPDATE vm_static SET origin=3 WHERE vm_name='HostedEngine'"
- edit the VM in webadmin (it should lose its crown status after a
while) and add VNC
- psql engine -c "UPDATE vm_static SET origin=6 WHERE vm_name='HostedEngine'"
- remove global maintenance
All usual disclaimers apply. I recommend backup if it is a production
environment and I can only propose this because I used it repeatedly
during development tests. But production use is on your own risk.
Best regards
Martin Sivak
On Mon, Apr 23, 2018 at 3:45 PM, Thomas Klute <klute at ingenit.com> wrote:
> Dear Martin,
>
> a follow up question regarding this case:
> we managed to get the hosted engine up and running again, but we'd like
> to permanently add the missing graphics device to the vm.conf (instead
> of starting with the vm-custom.conf in case of trouble).
> How do I add this line to the vm.conf config file (it seems to be
> overwritten after a few seconds - I suppose it's copied from somewhere?)
>
> Thanks,
> Thomas
>
>> Hi,
>>
>> the vnc device is there by default (I copied it out of my own hosted
>> engine instance), I do not know why it was missing in your case.
>>
>> Best regards
>>
>> Martin Sivak
>>
>> On Fri, Apr 13, 2018 at 5:13 PM, Thomas Klute <klute at ingenit.com> wrote:
>>> Dear Martin,
>>>
>>> yes, that worked. Thank you so much!!
>>> We were able to see that the latest kernel update failed and did not
>>> create the initramfs file and thus the boot process failed with a kernel
>>> panic.
>>>
>>> Debugging this problem took us many hours... It felt so complicated to
>>> connect a vnc to this vm - compared to a bare metal setup with display
>>> and keyboard.
>>> Wouldn't it be a good idea to have the vnc device in the config by default?
>>>
>>> Best regards,
>>> Thomas
>>>
>>>> You need to be in global maintenance, but I think you already know that.
>>>> Then try updating the vm.conf like you already did and add this line:
>>>>
>>>> devices={device:vnc,type:graphics,deviceId:f1d0394e-b077-4ea6-99e5-b9b6b8fe073c,address:None}
>>>>
>>>> Then restart the VM using hosted-engine commands and try the VNC approach again.
>>>>
>>>> Best regards
>>>>
>>>> Martin Sivak
>>>>
>>>> On Fri, Apr 13, 2018 at 2:35 PM, Thomas Klute <klute at ingenit.com> wrote:
>>>>> Dear Martin,
>>>>>
>>>>> thanks for the feedback.
>>>>> We already read this and tried it.
>>>>> It seems to me that the graphics device was removed from the hosted
>>>>> engine by some ovirt release.
>>>>>
>>>>> If I try to set a console password I see this message:
>>>>>
>>>>> hosted-engine --add-console-password
>>>>> Enter password:
>>>>> no graphics devices configured
>>>>>
>>>>> Furthermore, there is nothing listening on port 5900 after that.
>>>>> The HostedEngine qemu process shows a " -display none " as parameter and
>>>>> I have no idea where this can be changed.
>>>>>
>>>>> I already created a /var/run/ovirt-hosted-engine-ha/vm-custom.conf
>>>>> containing:
>>>>> display=vnc
>>>>> kvmEnable=true
>>>>>
>>>>> But nothing changed.
>>>>> I also edited the HostedEngine VM config using virsh and added a vnc
>>>>> display:
>>>>> /usr/bin/virsh -c
>>>>> qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf edit
>>>>> HostedEngine
>>>>>
>>>>> <graphics type='vnc' port='12355' autoport='no' listen='0.0.0.0'>
>>>>> <listen type='address' address='0.0.0.0'/>
>>>>> </graphics>
>>>>> <video>
>>>>> <model type='qxl' ram='65536' vram='65536' vgamem='16384'
>>>>> heads='1' primary='yes'/>
>>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
>>>>> function='0x0'/>
>>>>> </video>
>>>>>
>>>>> But there is still "display none" passed as command line parameter to
>>>>> qemu and thus, I suppose, there's no display.
>>>>>
>>>>> Any help is appreciated, thanks,
>>>>> Thomas
>>>>>
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> the serial console will show output if the kernel knows to use it.
>>>>>>
>>>>>> The VNC approach is also possible and I believe we already have a
>>>>>> graphical device present. What you are looking for is probably this
>>>>>> (VNC method is described there):
>>>>>> https://www.ovirt.org/documentation/how-to/hosted-engine/#handle-engine-vm-boot-problems
>>>>>>
>>>>>> Best regards
>>>>>>
>>>>>> --
>>>>>> Martin Sivak
>>>>>> SLA / oVirt
>>>>>>
>>>>>> On Fri, Apr 13, 2018 at 11:26 AM, Thomas Klute <klute at ingenit.com> wrote:
>>>>>>> Dear oVirt Team,
>>>>>>>
>>>>>>> after trying to reboot a hosted engine setup on oVirt 4.2 the VM won't
>>>>>>> come up anymore.
>>>>>>> The qemu-kvm process is there but we're unable to access the VM using
>>>>>>> - the serial console (simply does not show anything, does not react to
>>>>>>> characters typed)
>>>>>>> - VNC / Spice because the hosted engine vm.conf does not contain any
>>>>>>> graphics device.
>>>>>>>
>>>>>>> Before trying to reinstall, we'd like to recover and debug what is going on.
>>>>>>> We mounted a Centos7 install .iso and started the VM using
>>>>>>> hosted-engine --vm-start
>>>>>>> --vm-conf=/var/run/ovirt-hosted-engine-ha/vm-custom.conf
>>>>>>> But we still have to problem, that the serial console does not show
>>>>>>> anything and there is no way to connect using VNC.
>>>>>>>
>>>>>>> So, what is the recommended way to move forward in such situation?
>>>>>>> IMHO the classical way would be to add a graphics device and connect via
>>>>>>> VNC?
>>>>>>> I did not use the serial console much, up to now. Should the serial
>>>>>>> console show any output during boot?
>>>>>>>
>>>>>>> Thanks for your help,
>>>>>>> Thomas
>>>>>>>
>>>>>>> --
>>>>>>> ________________________________________________________________________
>>>>>>>
>>>>>>> Dipl.-Inform. Thomas Klute klute at ingenit.com
>>>>>>> Geschäftsführer / CEO
>>>>>>> ----------------------------------------------------------------------
>>>>>>> ingenit GmbH & Co. KG Tel. +49 (0)231 58 698-120
>>>>>>> Emil-Figge-Strasse 76-80 Fax. +49 (0)231 58 698-121
>>>>>>> D-44227 Dortmund www.ingenit.com
>>>>>>>
>>>>>>> Registergericht: Amtsgericht Dortmund, HRA 13 914
>>>>>>> Gesellschafter : Thomas Klute, Marc-Christian Schröer
>>>>>>> ________________________________________________________________________
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Users mailing list
>>>>>>> Users at ovirt.org
>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>> Sollten noch Fragen offen sein, stehen wir Ihnen natürlich jederzeit
>>>>> gerne zur Verfügung.
>>>>>
>>>>> Mit Grüßen aus Dortmund,
>>>>> Thomas Klute
>>>>>
>>>>> --
>>>>> ________________________________________________________________________
>>>>>
>>>>> Dipl.-Inform. Thomas Klute klute at ingenit.com
>>>>> Geschäftsführer / CEO
>>>>> ----------------------------------------------------------------------
>>>>> ingenit GmbH & Co. KG Tel. +49 (0)231 58 698-120
>>>>> Emil-Figge-Strasse 76-80 Fax. +49 (0)231 58 698-121
>>>>> D-44227 Dortmund www.ingenit.com
>>>>>
>>>>> Registergericht: Amtsgericht Dortmund, HRA 13 914
>>>>> Gesellschafter : Thomas Klute, Marc-Christian Schröer
>>>>> ________________________________________________________________________
>>>>>
>>> Sollten noch Fragen offen sein, stehen wir Ihnen natürlich jederzeit
>>> gerne zur Verfügung.
>>>
>>> Mit Grüßen aus Dortmund,
>>> Thomas Klute
>>>
>>> --
>>> ________________________________________________________________________
>>>
>>> Dipl.-Inform. Thomas Klute klute at ingenit.com
>>> Geschäftsführer / CEO
>>> ----------------------------------------------------------------------
>>> ingenit GmbH & Co. KG Tel. +49 (0)231 58 698-120
>>> Emil-Figge-Strasse 76-80 Fax. +49 (0)231 58 698-121
>>> D-44227 Dortmund www.ingenit.com
>>>
>>> Registergericht: Amtsgericht Dortmund, HRA 13 914
>>> Gesellschafter : Thomas Klute, Marc-Christian Schröer
>>> ________________________________________________________________________
>>>
>>
>
> Sollten noch Fragen offen sein, stehen wir Ihnen natürlich jederzeit
> gerne zur Verfügung.
>
> Mit Grüßen aus Dortmund,
> Thomas Klute
>
> --
> ________________________________________________________________________
>
> Dipl.-Inform. Thomas Klute klute at ingenit.com
> Geschäftsführer / CEO
> ----------------------------------------------------------------------
> ingenit GmbH & Co. KG Tel. +49 (0)231 58 698-120
> Emil-Figge-Strasse 76-80 Fax. +49 (0)231 58 698-121
> D-44227 Dortmund www.ingenit.com
>
> Registergericht: Amtsgericht Dortmund, HRA 13 914
> Gesellschafter : Thomas Klute, Marc-Christian Schröer
> ________________________________________________________________________
>
More information about the Users
mailing list