Hi, It looks like after host restart my hosted engine VM is not accessible any more. The storage is glusterfs. The gluster volume is healthy. The VM status is: {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"} hosted-engine --console The engine VM is running on this host Connected to domain HostedEngine Escape character is ^] error: internal error: cannot find character device <null> I tried to boot it from cdrom by changing vm.conf file and I'm not sure if the syntax is correct: 1. I removed bootOrder:1 from the index:0 device devices={index:0,iface:virtio,format:raw,address:{type:pci,slot:0x07,bus:0x00,domain:0x0000,function:0x0},volumeID:8d823e33-4260-4004-a468-cf477d7b1f5b,imageID:41342181-5c8f-4544-878b-44fdaa40dddc,readonly:false,domainID:beb954e7-61b7-4437-bd21-4b268e1a26e5,deviceId:41342181-5c8f-4544-878b-44fdaa40dddc,poolID:00000000-0000-0000-0000-000000000000,device:disk,shared:exclusive,propagateErrors:off,type:disk} 2. in index:2 device I addedd bootOrder:1 and path to the iso image: devices={index:2,iface:ide,shared:false,readonly:true,bootOrder:1,deviceId:8c3179ac-b322-4f5c-9449-c52e3665e0ae,address:{controller:0,target:0,unit:0,bus:1,type:drive},device:cdrom,path:/opt/iso/CentOS-7-x86_64-DVD-1804.iso,type:disk} The outcome is the same - no console connection. Now I also get: hosted-engine --vm-start Command VM.getStats with args {'vmID': '1e3aa9cf-8708-40a0-bc86-1127df01047a'} failed: (code=1, message=Virtual machine does not exist: {'vmId': u'1e3aa9cf-8708-40a0-bc86-1127df01047a'}) Unfortunately I do not have any backup. Is there a way to redeploy hosted engine and import current configuration or any other way to fix it?
On Tue, Nov 6, 2018 at 4:16 PM Jarosław Prokopowski <jprokopowski@gmail.com> wrote:
Hi,
It looks like after host restart my hosted engine VM is not accessible any more. The storage is glusterfs. The gluster volume is healthy.
The VM status is: {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}
hosted-engine --console The engine VM is running on this host Connected to domain HostedEngine Escape character is ^] error: internal error: cannot find character device <null>
The serial console requires a correctly working systemd instance on guest side to spawn a getty process on tty connections. VNC console seams more reliable/useful for troubleshooting activities on systems that fails the boot process. I'd suggest to try: hosted-engine --add-console-password to force a temporary VNC password and then connect it with a VNC client.
I tried to boot it from cdrom by changing vm.conf file and I'm not sure if the syntax is correct: 1. I removed bootOrder:1 from the index:0 device
devices={index:0,iface:virtio,format:raw,address:{type:pci,slot:0x07,bus:0x00,domain:0x0000,function:0x0},volumeID:8d823e33-4260-4004-a468-cf477d7b1f5b,imageID:41342181-5c8f-4544-878b-44fdaa40dddc,readonly:false,domainID:beb954e7-61b7-4437-bd21-4b268e1a26e5,deviceId:41342181-5c8f-4544-878b-44fdaa40dddc,poolID:00000000-0000-0000-0000-000000000000,device:disk,shared:exclusive,propagateErrors:off,type:disk}
2. in index:2 device I addedd bootOrder:1 and path to the iso image:
devices={index:2,iface:ide,shared:false,readonly:true,bootOrder:1,deviceId:8c3179ac-b322-4f5c-9449-c52e3665e0ae,address:{controller:0,target:0,unit:0,bus:1,type:drive},device:cdrom,path:/opt/iso/CentOS-7-x86_64-DVD-1804.iso,type:disk}
The outcome is the same - no console connection.
Now I also get:
hosted-engine --vm-start Command VM.getStats with args {'vmID': '1e3aa9cf-8708-40a0-bc86-1127df01047a'} failed: (code=1, message=Virtual machine does not exist: {'vmId': u'1e3aa9cf-8708-40a0-bc86-1127df01047a'})
Unfortunately I do not have any backup. Is there a way to redeploy hosted engine and import current configuration or any other way to fix it?
My first advice is to try fixing it via VNC connection. If not feasible, we now have an ansible procedure to automatically redeploy hosted-engine restoring a backup on the fly but you still need a backup took with engine-backup. Last option is to deploy a new engine instance; shutdown all the VMs at guest level and try importing other storage domains from the new engine.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/HOD22UUEVQQDCS...
Thanks for your responses. VM was up but network was down in it so I could not use ssh. I didn't know the trick with with VNC password. Thanks Simone it may come handy in the future. I recreated the self hosted engine and imported the Data storage domain. Now I have another problem. The VMs are visible in the "VM Import" tab but there is nothing in "Template import" tab. When I imported the first VM with "Allow partial" option selected it just disappeared after the import and I cannot find it. How can I use it now? Thanks Jarek On Wed, Nov 7, 2018 at 11:07 AM <vadq-esq@yandex.ru> wrote:
The VM status is:
"vm": "up",
"detail": "Up"}
VM is up try to use ssh _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/JVVLSNHTPFVWGB...
Actually the VM is in Compute -> Virtual Machines but it does not have any Instance Images attached. How can I attach its original disk image? On Wed, Nov 7, 2018 at 12:12 PM Jarosław Prokopowski <jprokopowski@gmail.com> wrote:
Thanks for your responses.
VM was up but network was down in it so I could not use ssh. I didn't know the trick with with VNC password. Thanks Simone it may come handy in the future.
I recreated the self hosted engine and imported the Data storage domain. Now I have another problem. The VMs are visible in the "VM Import" tab but there is nothing in "Template import" tab. When I imported the first VM with "Allow partial" option selected it just disappeared after the import and I cannot find it. How can I use it now?
Thanks Jarek
On Wed, Nov 7, 2018 at 11:07 AM <vadq-esq@yandex.ru> wrote:
The VM status is:
"vm": "up",
"detail": "Up"}
VM is up try to use ssh _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/JVVLSNHTPFVWGB...
I tried the option to "Scan Disks" for the imported storage domain but I get this error in engine.log and does not change anything: 2018-11-07 16:41:25,440+01 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (default task-34) [51450801-5353-4e8b-b72a-66ddd71ba6b2] Failed in 'GetVolumeInfoVDS' method 2018-11-07 16:41:25,440+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (default task-34) [51450801-5353-4e8b-b72a-66ddd71ba6b2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand' return value ' VolumeInfoReturn:{status='Status [code=201, message=Volume does not exist: (u'37bc4b45-69f9-4dc4-a9b8-c6f4582c6cbe',)]'} I was able to find the correct disk images in the glusterfs volume and uploaded one of them through the web interface and attached to corresponding imported virtual machine. The VM does not start completely. It complains now about disk size: Device /dev/sda2 has size of 22528 sectors which is smaller than corresponding PV size of 166746122 This is thin provision disk. Do you have any ideas? On Wed, Nov 7, 2018 at 12:22 PM Jarosław Prokopowski <jprokopowski@gmail.com> wrote:
Actually the VM is in Compute -> Virtual Machines but it does not have any Instance Images attached. How can I attach its original disk image?
On Wed, Nov 7, 2018 at 12:12 PM Jarosław Prokopowski < jprokopowski@gmail.com> wrote:
Thanks for your responses.
VM was up but network was down in it so I could not use ssh. I didn't know the trick with with VNC password. Thanks Simone it may come handy in the future.
I recreated the self hosted engine and imported the Data storage domain. Now I have another problem. The VMs are visible in the "VM Import" tab but there is nothing in "Template import" tab. When I imported the first VM with "Allow partial" option selected it just disappeared after the import and I cannot find it. How can I use it now?
Thanks Jarek
On Wed, Nov 7, 2018 at 11:07 AM <vadq-esq@yandex.ru> wrote:
The VM status is:
"vm": "up",
"detail": "Up"}
VM is up try to use ssh _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/JVVLSNHTPFVWGB...
On Wed, 2018-11-07 at 16:05 +0000, Jarosław Prokopowski wrote:
I was able to find the correct disk images in the glusterfs volume and uploaded one of them through the web interface and attached to corresponding imported virtual machine. The VM does not start completely. It complains now about disk size: Device /dev/sda2 has size of 22528 sectors which is smaller than corresponding PV size of 166746122 This is thin provision disk.
We had a very similar issue where the disk image on gluster got corrupted and ended-up being a similar size. I'm interested in the fix for this. -- Met vriendelijke groet, Rob Epping | NOC | Afdeling: +31 88 773 25 86 +31 88 77 325 87 | www.speakup.nl Institutenweg 6 7521 PK Enschede | Postbus 1330 7500 BH Enschede
I ended up creating the VMs from scratch but I had only 3 so it wasn't that bad. I lost few days of work though. I'm sure the virtual disk was ok but oVirt could not recognize it correctly. I was able to import VMs only partially from connected storage domain but it did not import the disks. When I found the correct disk image in Glusterfs and uploaded it there was the size issue. The VM did not recognize the thin provision disk correctly. I also would like to know how to deal with such issues in case it happens again. On Wed, Nov 14, 2018 at 4:07 PM Rob Epping <r.epping@speakup.nl> wrote:
On Wed, 2018-11-07 at 16:05 +0000, Jarosław Prokopowski wrote:
I was able to find the correct disk images in the glusterfs volume and uploaded one of them through the web interface and attached to corresponding imported virtual machine. The VM does not start completely. It complains now about disk size: Device /dev/sda2 has size of 22528 sectors which is smaller than corresponding PV size of 166746122 <16%20674%2061%2022> This is thin provision disk.
We had a very similar issue where the disk image on gluster got corrupted and ended-up being a similar size. I'm interested in the fix for this. -- Met vriendelijke groet, Rob Epping | NOC | Afdeling: +31 88 773 25 86
+31 88 77 325 87 | www.speakup.nl Institutenweg 6 7521 PK Enschede | Postbus 1330 7500 BH Enschede
participants (4)
-
Jarosław Prokopowski -
Rob Epping -
Simone Tiraboschi -
vadq-esq＠yandex.ru