[ovirt-users] hosted-engine deploy fails at "Wait for the engine to come up on the target VM" step
Simone Tiraboschi
stirabos at redhat.com
Mon Mar 12 17:55:57 UTC 2018
On Mon, Mar 12, 2018 at 6:44 PM, Kristian Petersen <nesretep at chem.byu.edu>
wrote:
> I think I accidentally sent that reply before I was really finished with
> it. The error said the the VM mentioned in the conf file didn't exist. I
> included the log file as requested.
>
As far as I can see from the logs, the engine VM went up as expected now:
2018-03-12 11:15:00,183-0600 INFO (jsonrpc/5) [api.virt] FINISH getStats
return={'status': {'message': 'Done', 'code': 0}, 'statsList':
[{'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': u'vnc',
'port': '5900'}], 'memUsage': '17', 'acpiEnable': 'true', 'guestFQDN': u'
rhv-engine.cpms.byu.edu', 'vmId': 'cbe9b80f-9c18-409c-b7b1-54d95f4734ca',
'session': 'Unknown', 'displaySecurePort': '-1', 'timeOffset': '0',
'balloonInfo': {'balloon_max': '16777216', 'balloon_min': '0',
'balloon_target': '16777216', 'balloon_cur': '16777216'}, 'pauseCode':
'NOERR', 'disksUsage': [{u'path': u'/', u'total': '6565134336', u'fs':
u'xfs', u'used': '1950617600'}, {u'path': u'/boot', u'total': '1063256064',
u'fs': u'xfs', u'used': '170590208'}, {u'path': u'/home', u'total':
'1063256064', u'fs': u'xfs', u'used': '33792000'}, {u'path': u'/var',
u'total': '21464350720', u'fs': u'xfs', u'used': '396513280'}, {u'path':
u'/var/log', u'total': '10726932480', u'fs': u'xfs', u'used': '42823680'},
{u'path': u'/tmp', u'total': '2136997888', u'fs': u'xfs', u'used':
'34058240'}, {u'path': u'/var/log/audit', u'total': '1063256064', u'fs':
u'xfs', u'used': '34586624'}], 'network': {'vnet0': {'macAddr':
u'00:16:3e:54:f3:8e', 'rxDropped': '0', 'tx': '30742', 'rxErrors': '0',
'txDropped': '0', 'rx': '167904', 'txErrors': '0', 'state': 'unknown',
'sampleTime': 9385842.59, 'speed': '1000', 'name': 'vnet0'}}, 'vmJobs': {},
'cpuUser': '7.43', 'elapsedTime': '81', 'memoryStats': {'swap_out': '0',
'majflt': '0', 'mem_cached': '452216', 'mem_free': '13404476',
'mem_buffers': '2104', 'swap_in': '0', 'pageflt': '418', 'mem_total':
'16263704', 'mem_unused': '13404476'}, 'cpuSys': '1.67', 'appsList':
(u'ovirt-guest-agent-common-1.0.14-1.el7', u'kernel-3.10.0-693.17.1.el7',
u'cloud-init-0.7.9-9.el7.centos.2'), 'guestOs':
u'3.10.0-693.17.1.el7.x86_64', 'vmName': 'HostedEngine', 'displayType':
'vnc', 'vcpuCount': '4', 'clientIp': '', 'hash': '-7630705381253994604',
'guestCPUCount': 4, 'vmType': 'kvm', 'displayIp': '0', 'cpuUsage':
'9110000000', 'vcpuPeriod': 100000L, 'displayPort': '5900',
'guestTimezone': {u'zone': u'America/Denver', u'offset': -420},
'vcpuQuota': '-1', 'statusTime': '9385842590', 'kvmEnable': 'true',
'disks': {'vda': {'readLatency': '387098', 'writtenBytes': '36851200',
'writeOps': '465', 'apparentsize': '125627793408', 'readOps': '15711',
'writeLatency': '1931806', 'imageID':
u'ec964354-ac01-4799-9c20-4bc923d285d4', 'readBytes': '480176128',
'flushLatency': '237153', 'readRate': '545.769487017', 'truesize':
'2503184384', 'writeRate': '76646.5023329'}, 'hdc': {'readLatency': '0',
'writtenBytes': '0', 'writeOps': '0', 'apparentsize': '0', 'readOps': '4',
'writeLatency': '0', 'readBytes': '152', 'flushLatency': '0', 'readRate':
'0.0', 'truesize': '0', 'writeRate': '0.0'}}, 'monitorResponse': '0',
'guestOsInfo': {u'kernel': u'3.10.0-693.17.1.el7.x86_64', u'arch':
u'x86_64', u'version': u'7.4.1708', u'distribution': u'CentOS Linux',
u'type': u'linux', u'codename': u'Core'}, 'username': u'None', 'guestName':
u'rhv-engine.cpms.byu.edu', 'status': 'Up', 'lastLogin': 1520874832.982759,
'guestIPs': u'192.168.1.22', 'guestContainers': [], 'netIfaces':
[{u'inet6': [u'fe80::216:3eff:fe54:f38e'], u'hw': u'00:16:3e:54:f3:8e',
u'inet': [u'192.168.1.22'], u'name': u'eth0'}]}]} from=::1,52334 (api:52)
>
>
> On Mon, Mar 12, 2018 at 11:36 AM, Simone Tiraboschi <stirabos at redhat.com>
> wrote:
>
>>
>>
>> On Mon, Mar 12, 2018 at 6:27 PM, Kristian Petersen <nesretep at chem.byu.edu
>> > wrote:
>>
>>> I tried using my customized vm.conf with the fix in the CPU name as you
>>> suggested. When I ran hosted-engine --vm-start --vm-conf=/root/myvm.conf
>>> and that failed.
>>>
>>
>> This is fine if the VM doesn't exist.
>> Can you please share your vdsm.log?
>>
>>
>>> It said the vm didn't exist. It sounds like I might need to get the
>>> updated package from the ovirt-4.2-pre repo and try deploying again.
>>>
>>> On Mon, Mar 12, 2018 at 10:31 AM, Simone Tiraboschi <stirabos at redhat.com
>>> > wrote:
>>>
>>>>
>>>>
>>>> On Mon, Mar 12, 2018 at 5:25 PM, Kristian Petersen <
>>>> nesretep at chem.byu.edu> wrote:
>>>>
>>>>> I'm guessing that v2.2.10 is not in the oVirt repo yet. When I looked
>>>>> at vm.conf, the CPU name has a space in it like the one mentioned in the
>>>>> link you included. So replacing that space with an underscore should do
>>>>> the trick prehaps?
>>>>>
>>>>
>>>> v2.2.12 is in -pre repo.
>>>>
>>>> You should replace the space with a dash: Broadwell-IBRS
>>>>
>>>>
>>>>>
>>>>> On Mon, Mar 12, 2018 at 10:00 AM, Kristian Petersen <
>>>>> nesretep at chem.byu.edu> wrote:
>>>>>
>>>>>> I have v2.2.9 of ovirt-hosted-engine-setup currently installed. I'll
>>>>>> try out the other suggestion you made also. Thanks for the help.
>>>>>>
>>>>>> On Fri, Mar 9, 2018 at 4:26 PM, Simone Tiraboschi <
>>>>>> stirabos at redhat.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Mar 9, 2018 at 8:33 PM, Kristian Petersen <
>>>>>>> nesretep at chem.byu.edu> wrote:
>>>>>>>
>>>>>>>> I have attached the relevant log files as requested.
>>>>>>>> vdsm.log.1
>>>>>>>> <https://drive.google.com/a/chem.byu.edu/file/d/1ibJG_SEjK9NSEPft_HCkZzQO25dGgMVo/view?usp=drive_web>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The real issue is here:
>>>>>>>
>>>>>>> <cpu match="exact">
>>>>>>> <model>BroadwellIBRS</model>
>>>>>>> </cpu>
>>>>>>> <on_poweroff>destroy</on_poweroff><on_reboot>destroy</on_reb
>>>>>>> oot><on_crash>destroy</on_crash></domain> (vm:2751)
>>>>>>> 2018-03-08 08:04:13,757-0700 ERROR (vm/9a1e133d) [virt.vm]
>>>>>>> (vmId='9a1e133d-13d8-4613-b1a5-fd3ca81ffcc3') The vm start process
>>>>>>> failed (vm:927)
>>>>>>> Traceback (most recent call last):
>>>>>>> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>>>>>>> 856, in _startUnderlyingVm
>>>>>>> self._run()
>>>>>>> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>>>>>>> 2756, in _run
>>>>>>> dom.createWithFlags(flags)
>>>>>>> File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py",
>>>>>>> line 130, in wrapper
>>>>>>> ret = f(*args, **kwargs)
>>>>>>> File "/usr/lib/python2.7/site-packages/vdsm/common/function.py",
>>>>>>> line 92, in wrapper
>>>>>>> return func(inst, *args, **kwargs)
>>>>>>> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069,
>>>>>>> in createWithFlags
>>>>>>> if ret == -1: raise libvirtError ('virDomainCreateWithFlags()
>>>>>>> failed', dom=self)
>>>>>>> libvirtError: internal error: Unknown CPU model BroadwellIBRS
>>>>>>>
>>>>>>> Indeed it should be Broadwell-IBRS
>>>>>>>
>>>>>>> Can you please report which rpm version of ovirt-hosted-engine-setup
>>>>>>> did you used?
>>>>>>>
>>>>>>> You can fix it in this way:
>>>>>>> copy /var/run/ovirt-hosted-engine-ha/vm.conf somewhere, edit it and
>>>>>>> update the cpuType field.
>>>>>>>
>>>>>>> Then start the engine VM with your custom vm.conf with something
>>>>>>> like:
>>>>>>> hosted-engine --vm-start --vm-conf=/root/my_vm.conf
>>>>>>> keep the engine up for at least one hour and it will generate the
>>>>>>> OVF_STORE disks with the right configuration for the hosted-engine VM.
>>>>>>>
>>>>>>> It failed really at the end of the setup so anything else should be
>>>>>>> fine.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Mar 9, 2018 at 1:21 AM, Simone Tiraboschi <
>>>>>>>> stirabos at redhat.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Mar 8, 2018 at 7:28 PM, Kristian Petersen <
>>>>>>>>> nesretep at chem.byu.edu> wrote:
>>>>>>>>>
>>>>>>>>>> I am trying to deploy oVirt with a self-hosted engine and the
>>>>>>>>>> setup seems to go well until near the very end when the status message says:
>>>>>>>>>> [ INFO ] TASK [Wait for the engine to come up on the target VM]
>>>>>>>>>>
>>>>>>>>>> [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 120,
>>>>>>>>>> "changed": true, "cmd": ["hosted-engine", "--vm-status", "--json"],
>>>>>>>>>> "delta": "0:0
>>>>>>>>>> 0:00.216412", "end": "2018-03-07 16:02:02.677478", "rc": 0,
>>>>>>>>>> "start": "2018-03-07 16:02:02.461066", "stderr": "", "stderr_lines": [],
>>>>>>>>>> "stdout
>>>>>>>>>> ": "{\"1\": {\"conf_on_shared_storage\": true, \"live-data\":
>>>>>>>>>> true, \"extra\": \"metadata_parse_version=1\\nm
>>>>>>>>>> etadata_feature_version=1\\ntim
>>>>>>>>>> estamp=4679955 (Wed Mar 7 16:01:50 2018)\\nhost-id=1\\nscore=3400
>>>>>>>>>> \\nvm_conf_refresh_time=4679956 (Wed Mar 7 16:01:51
>>>>>>>>>> 2018)\\nconf_on_share
>>>>>>>>>> d_storage=True\\nmaintenance=False\\nstate=EngineStarting\\nstopped=False\\n\",
>>>>>>>>>> \"hostname\": \"rhv1.cpms.byu.edu\", \"host-id\": 1, \"engin
>>>>>>>>>> e-status\": {\"reason\": \"vm not running on this host\",
>>>>>>>>>> \"health\": \"bad\", \"vm\": \"down\", \"detail\": \"unknown\"}, \"score\":
>>>>>>>>>> 3400,
>>>>>>>>>> \"stopped\": false, \"maintenance\": false, \"crc32\":
>>>>>>>>>> \"d3a67cf7\", \"local_conf_timestamp\": 4679956, \"host-ts\": 4679955},
>>>>>>>>>> \"global_main
>>>>>>>>>> tenance\": false}", "stdout_lines": ["{\"1\":
>>>>>>>>>> {\"conf_on_shared_storage\": true, \"live-data\": true, \"extra\":
>>>>>>>>>> \"metadata_parse_version=1\
>>>>>>>>>> \nmetadata_feature_version=1\\ntimestamp=4679955 (Wed Mar 7
>>>>>>>>>> 16:01:50 2018)\\nhost-id=1\\nscore=3400
>>>>>>>>>> \\nvm_conf_refresh_time=4679956 (Wed Mar
>>>>>>>>>> 7 16:01:51 2018)\\nconf_on_shared_storage
>>>>>>>>>> =True\\nmaintenance=False\\nstate=EngineStarting\\nstopped=False\\n\",
>>>>>>>>>> \"hostname\": \"rhv1.cpms.
>>>>>>>>>> byu.edu\", \"host-id\": 1, \"engine-status\": {\"reason\": \"vm
>>>>>>>>>> not running on this host\", \"health\": \"bad\", \"vm\": \"down\", \"detail\
>>>>>>>>>> ": \"unknown\"}, \"score\": 3400, \"stopped\": false,
>>>>>>>>>> \"maintenance\": false, \"crc32\": \"d3a67cf7\", \"local_conf_timestamp\":
>>>>>>>>>> 4679956, \"
>>>>>>>>>> host-ts\": 4679955}, \"global_maintenance\": false}"]}
>>>>>>>>>> [ ERROR ] Failed to execute stage 'Closing up': Failed executing
>>>>>>>>>> ansible-playbook
>>>>>>>>>>
>>>>>>>>>> Any ideas that might help?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Kristian,
>>>>>>>>> {\"reason\": \"vm not running on this host\" sonds really bad.
>>>>>>>>> I means that ovirt-ha-agent (in charge of restarting the engine
>>>>>>>>> VM) think that another host took over but at that stage you should have
>>>>>>>>> just one host.
>>>>>>>>>
>>>>>>>>> Could you please attach /var/log/ovirt-hosted-engine-ha/agent.log
>>>>>>>>> and /var/log/vdsm/vdsm.log for the relevant time frame?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Kristian Petersen
>>>>>>>>>> System Administrator
>>>>>>>>>> Dept. of Chemistry and Biochemistry
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Users mailing list
>>>>>>>>>> Users at ovirt.org
>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Kristian Petersen
>>>>>>>> System Administrator
>>>>>>>> Dept. of Chemistry and Biochemistry
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Kristian Petersen
>>>>>> System Administrator
>>>>>> BYU Dept. of Chemistry and Biochemistry
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Kristian Petersen
>>>>> System Administrator
>>>>> BYU Dept. of Chemistry and Biochemistry
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Kristian Petersen
>>> System Administrator
>>> BYU Dept. of Chemistry and Biochemistry
>>>
>>
>>
>
>
> --
> Kristian Petersen
> System Administrator
> BYU Dept. of Chemistry and Biochemistry
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20180312/86da1883/attachment.html>
More information about the Users
mailing list