[ovirt-users] hosted-engine vm-status stale data and cluster seems "broken"

Sahina Bose sabose at redhat.com
Wed Jun 15 05:09:45 UTC 2016


Are the ovirt-ha-agent and ovirt-ha-broker services running on all the 
nodes? If they are, check the agent.log and broker.log for errors.

On 06/14/2016 05:29 PM, Charles Kozler wrote:
> Anyone have any other possible information on this? I've noticed this 
> issue before and usually it just takes a bit of time for the cluster 
> to 'settle' after some node reboots but its been a few days and its 
> still marked as stale.
>
>
>
> --== Host 1 status ==--
>
> Status up-to-date                  : False
> Hostname                           : njsevcnp01
> Host ID                            : 1
> Engine status                      : unknown stale-data
> Score                              : 0
> stopped                            : True
> Local maintenance                  : False
> crc32                              : 260dbf06
> Host timestamp                     : 327
>
>
> --== Host 2 status ==--
>
> Status up-to-date                  : False
> Hostname                           : njsevcnp02
> Host ID                            : 2
> Engine status                      : unknown stale-data
> Score                              : 0
> stopped                            : True
> Local maintenance                  : False
> crc32                              : 25da07df
> Host timestamp                     : 3030
>
>
> --== Host 3 status ==--
>
> Status up-to-date                  : False
> Hostname                           : njsevcnp03
> Host ID                            : 3
> Engine status                      : unknown stale-data
> Score                              : 0
> stopped                            : True
> Local maintenance                  : False
> crc32                              : c67818cb
> Host timestamp                     : 10877406
>
>
> && vdsClient on node2 showing hosted engine is up on node 2
>
> 48207078-8cb0-413c-8984-40aa772f4d94
> Status = Up
> nicModel = rtl8139,pv
> statusTime = 4540044460
> emulatedMachine = pc
> pid = 30571
> vmName = HostedEngine
> devices = [{'device': 'memballoon', 'specParams': {'model': 'none'}, 
> 'type': 'balloon', 'alias': 'balloon0'}, {'alias': 'scsi0', 
> 'deviceId': '17f10db1-2e9e-4422-9ea5-61a628072e29', 'address': 
> {'slot': '0x04', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 
> 'function': '0x0'}, 'device': 'scsi', 'model': 'virtio-scsi', 'type': 
> 'controller'}, {'device': 'usb', 'alias': 'usb', 'type': 'controller', 
> 'deviceId': '9be34ac0-7d00-4a95-bdfe-5b328fc1355b', 'address': 
> {'slot': '0x01', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 
> 'function': '0x2'}}, {'device': 'ide', 'alias': 'ide', 'type': 
> 'controller', 'deviceId': '222629a8-0dd6-4e8e-9b42-43aac314c0c2', 
> 'address': {'slot': '0x01', 'bus': '0x00', 'domain': '0x0000', 'type': 
> 'pci', 'function': '0x1'}}, {'device': 'virtio-serial', 'alias': 
> 'virtio-serial0', 'type': 'controller', 'deviceId': 
> '7cbccd04-853a-408f-94c2-5b10b641b7af', 'address': {'slot': '0x05', 
> 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}}, 
> {'device': 'vnc', 'specParams': {'spiceSecureChannels': 
> 'smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir', 
> 'displayIp': '0'}, 'type': 'graphics', 'port': '5900'}, {'nicModel': 
> 'pv', 'macAddr': '00:16:3e:16:83:91', 'linkActive': True, 'network': 
> 'ovirtmgmt', 'alias': 'net0', 'deviceId': 
> '3f679659-142c-41f3-a69d-4264d7234fbc', 'address': {'slot': '0x03', 
> 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 
> 'device': 'bridge', 'type': 'interface', 'name': 'vnet0'}, {'address': 
> {'slot': '0x06', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 
> 'function': '0x0'}, 'volumeInfo': {'domainID': 
> 'c6323975-2966-409d-b9e0-48370a513a98', 'volType': 'path', 
> 'leaseOffset': 0, 'volumeID': 'aa66d378-5a5f-490c-b0ab-993b79838d95', 
> 'leasePath': 
> '/rhev/data-center/mnt/nas01:_volume1_vm__os_ovirt36__engine/c6323975-2966-409d-b9e0-48370a513a98/images/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95.lease', 
> 'imageID': '8518ef4a-7b17-4291-856c-81875ba4e264', 'path': 
> '/rhev/data-center/mnt/nas01:_volume1_vm__os_ovirt36__engine/c6323975-2966-409d-b9e0-48370a513a98/images/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95'}, 
> 'index': '0', 'iface': 'virtio', 'apparentsize': '10737418240', 
> 'imageID': '8518ef4a-7b17-4291-856c-81875ba4e264', 'readonly': 
> 'False', 'shared': 'exclusive', 'truesize': '6899802112', 'type': 
> 'disk', 'domainID': 'c6323975-2966-409d-b9e0-48370a513a98', 'reqsize': 
> '0', 'format': 'raw', 'deviceId': 
> '8518ef4a-7b17-4291-856c-81875ba4e264', 'poolID': 
> '00000000-0000-0000-0000-000000000000', 'device': 'disk', 'path': 
> '/var/run/vdsm/storage/c6323975-2966-409d-b9e0-48370a513a98/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95', 
> 'propagateErrors': 'off', 'name': 'vda', 'bootOrder': '1', 'volumeID': 
> 'aa66d378-5a5f-490c-b0ab-993b79838d95', 'alias': 'virtio-disk0', 
> 'volumeChain': [{'domainID': 'c6323975-2966-409d-b9e0-48370a513a98', 
> 'volType': 'path', 'leaseOffset': 0, 'volumeID': 
> 'aa66d378-5a5f-490c-b0ab-993b79838d95', 'leasePath': 
> '/rhev/data-center/mnt/nas01:_volume1_vm__os_ovirt36__engine/c6323975-2966-409d-b9e0-48370a513a98/images/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95.lease', 
> 'imageID': '8518ef4a-7b17-4291-856c-81875ba4e264', 'path': 
> '/rhev/data-center/mnt/nas01:_volume1_vm__os_ovirt36__engine/c6323975-2966-409d-b9e0-48370a513a98/images/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95'}]}, 
> {'index': '2', 'iface': 'ide', 'name': 'hdc', 'alias': 'ide0-1-0', 
> 'readonly': 'True', 'deviceId': 
> '8c3179ac-b322-4f5c-9449-c52e3665e0ae', 'address': {'bus': '1', 
> 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 
> 'device': 'cdrom', 'shared': 'false', 'path': '', 'type': 'disk'}, 
> {'device': 'unix', 'alias': 'channel0', 'type': 'channel', 'address': 
> {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': 
> '1'}}, {'device': 'unix', 'alias': 'channel1', 'type': 'channel', 
> 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 
> 'port': '2'}}, {'device': 'unix', 'alias': 'channel2', 'type': 
> 'channel', 'address': {'bus': '0', 'controller': '0', 'type': 
> 'virtio-serial', 'port': '3'}}, {'device': '', 'alias': 'video0', 
> 'type': 'video', 'address': {'slot': '0x02', 'bus': '0x00', 'domain': 
> '0x0000', 'type': 'pci', 'function': '0x0'}}]
> guestDiskMapping = {'8518ef4a-7b17-4291-8': {'name': '/dev/vda'}, 
> 'QEMU_DVD-ROM_QM00003': {'name': '/dev/sr0'}}
> vmType = kvm
> displaySecurePort = -1
> memSize = 4096
> displayPort = 5900
> clientIp =
> spiceSecureChannels = 
> smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir
> smp = 4
> displayIp = 0
> display = vnc
> pauseCode = NOERR
>
>
>
>
> On Mon, Jun 13, 2016 at 8:25 AM, Charles Kozler <charles at fixflyer.com 
> <mailto:charles at fixflyer.com>> wrote:
>
>     It is up. I can do "ps -Aef | grep -i qemu-kvm | grep -i hosted"
>     and see it running. I also forcefully shut it down with
>     hosted-engine --vm-stop when it was on node1 and then did
>     --vm-start on node 2 and it came up. Also the Web UI is reachable
>     so thats how I also know the hosted engine VM is running
>
>     On Mon, Jun 13, 2016 at 8:24 AM, Alexis HAUSER
>     <alexis.hauser at telecom-bretagne.eu
>     <mailto:alexis.hauser at telecom-bretagne.eu>> wrote:
>
>
>         > http://imgur.com/a/6xkaS
>
>         I had similar errors with one single host and a hosted-engine VM.
>         My case should be totally different, but one thing you could
>         try first is to check VM is really up.
>         In my issues, VM was shown by hosted-engine command as up, but
>         was down. with vdsClient command, you can check if it's status
>         with more details.
>
>         What is the result for you of the following command ?
>
>          vdsClient -s 0 list
>
>
>
>
>     -- 
>
>     *Charles Kozler*
>     /Vice President, IT Operations/
>     FIX Flyer, LLC
>     225 Broadway | Suite 1600 | New York, NY 10007
>     1-888-349-3593 <tel:1-888-349-3593>
>     http://www.fixflyer.com <http://fixflyer.com>
>     NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED
>     RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL
>     INFORMATION WHICH IS PROPRIETARY TO FIX FLYER LLC.  ANY
>     UNAUTHORIZED USE, COPYING, DISTRIBUTION, OR DISSEMINATION IS
>     STRICTLY PROHIBITED.  ALL RIGHTS TO THIS INFORMATION IS RESERVED
>     BY FIX FLYER LLC.  IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE
>     CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL
>     FROM YOUR SYSTEM AND DESTROY ANY COPIES.
>
>
>
>
> -- 
>
> *Charles Kozler*
> /Vice President, IT Operations/
> FIX Flyer, LLC
> 225 Broadway | Suite 1600 | New York, NY 10007
> 1-888-349-3593
> http://www.fixflyer.com <http://fixflyer.com>
> NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED 
> RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL 
> INFORMATION WHICH IS PROPRIETARY TO FIX FLYER LLC.  ANY UNAUTHORIZED 
> USE, COPYING, DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED.  
> ALL RIGHTS TO THIS INFORMATION IS RESERVED BY FIX FLYER LLC.  IF YOU 
> ARE NOT THE INTENDED RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY 
> E-MAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR SYSTEM AND DESTROY ANY 
> COPIES.
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160615/6c01195c/attachment-0001.html>


More information about the Users mailing list