Non-responsive vm's due to crashed host and hosted vm liveliness check fails
by clam2718＠gmail.com
Dear Community:
The local drive on a host running ovirt-node-ng-4.1.9 in a three node cluster failed. I have production JIRA and Postres running on it at the time, not in HA, just simple vm's. Storage is via NFS on a Synology NAS. Hosted Engine was on a different host, but JIRA and Postgres vm's showed nonresponsive. I tried different things but then stupidly thought that upgrading my Hosted Engine would reinitialize the vm's that were on the failed host. Update of HE seemed to go well (output below) but now my Hosted Engine, while up, fails the liveliness check and the web management console is unavailable. I cannot console into the HE from the host it is running on. Below are the results of my attempts to console into the Hosted Engine. Please help! I have search forums, lists and Google but have not been able to fix this. My coworkers and manager are anxious.
---
When I try "hosted-engine --console" after setting the console password I get
The engine VM is running on this host
Connected to domain HostedEngine
Escape character is ^]
_
The prompt is non-responsive except for the escape character key combo.
---
"virsh -r list" gives ID 3, Name: HostedEngine, State: running
"virsh -r console HostedEngine" gives
Connected to domain HostedEngine
Escape character is ^]
error: operation forbidden: read only access prevents virDomainOpenConsole
"virsh -r vncdisplay HostedEngine" gives "0:0" and returns me to prompt
---
I am SSHed into the host running my Hosted Engine from a CentOS7 minimal install with packages xorg-x11-server-Xorg, xorg-x11-xauth and xorg-x11-apps installed. The result of "grep -i X11Forwarding /etc/ssh/sshd_config" shows it set to "Yes".
I SSH into the host using "ssh -Y root(a)xxx.xxx.xxx.xxx" I am logged into the CentOS7 minimal install as root. I know root is poor practice but was trying to minimize anything that could be causing an issue.
---
Below are the results of my attempt to update Hosted Engine (slightly redacted to remove personal info):
--== CONFIGURATION PREVIEW ==--
Default SAN wipe after delete : False
Firewall manager : firewalld
Update Firewall : True
Host FQDN : ovengineint.xdomainx.tld
Upgrade packages : True
Engine database secured connection : False
Engine database user name : engine
Engine database name : engine
Engine database host : localhost
Engine database port : 5432
Engine database host name validation : False
Engine installation : True
PKI organization : xdomainx.tld
Set up ovirt-provider-ovn : True
Configure WebSocket Proxy : True
DWH installation : True
DWH database secured connection : False
DWH database host : localhost
DWH database user name : ovirt_engine_history
DWH database name : ovirt_engine_history
DWH database port : 5432
DWH database host name validation : False
Configure Image I/O Proxy : True
Configure VMConsole Proxy : True
--== SUMMARY ==--
[ INFO ] Restarting httpd
Web access is enabled at:
http://ovengineint.xdomainx.tld:80/ovirt-engine
https://ovengineint.xdomainx.tld:443/ovirt-engine
Internal CA XX:XX:XX:XX...
SSH fingerprint: SHA256:xxxxxxxxxx...
--== END OF SUMMARY ==--
[ INFO ] Stage: Clean up
Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20180502165652-88pkpi.log
[ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20180502170149-setup.conf'
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ INFO ] Execution of setup completed successfully
Again, thank you so very much for any suggestions! I have found many answers on this mailing list archive to be of great insight and help.
Respectfully,
Charles
6 years, 5 months
User agent for Ovirt 4.2
by Jim Kusznir
Hi:
I haven't managed to find the new / current repo/source for the ovirt guest
agent for the 4.2 upgrade. All my VMs now say that they need the agent.
Googles keep referring me to old / broke / non-existent repos. Where do I
find the 4.2 agent (or does the 4.2 agent even exist?)
Thanks!
--Jim
6 years, 5 months
Re: OVA export/import
by RabidCicada
Third attempt to send to list also
> Yaniv,
> Most of this is obe...but I'll answer the questions. It's been resolved
> as in my other email to this chain.
>
>
> On Thu, Jun 14, 2018, 4:56 PM Yaniv Kaul <ykaul(a)redhat.com> wrote:
>
>>
>>
>> On Wed, Jun 13, 2018, 4:05 PM RabidCicada <rabidcicada(a)gmail.com> wrote:
>>
>>> All,
>>> I recently tried to used the OVA export/import functionality. It
>>> seems I misunderstood the intentions. I expected OVA export/import to
>>> be reciprocal functionalities. I expected what we export as OVA to be
>>> importable as OVA in ovirt.
>>>
>>> What I have found is the following:
>>>
>>> * It seems OVA export does export an OVA, though the format is not
>>> spec compliant
>>>
>>
>> Which spec?
>>
> I believe every version of the dmtd ovf specification.
>
>>
>> * We use 'disk/' in the HostResource xml field instead of '/disk/'
>>> for example in the ovf.
>>>
>>
>> Who's 'we'?
>>
> Ovirt software/group. I was including myself in a friendly manner soas to
> not sound accusatory. I know it's a group effort :).
>
>
>> * It seems OVA import is intended specifically for VMWare OVA's?
>>>
>>
>> That's a common case, but we can look at other OVAs. The problem is that
>> the spec is quite loose.
>> Y.
>>
>
> Agreed. I know that's why it's hard to be inter-compatible.
>
6 years, 5 months
OVA export/import
by RabidCicada
All,
I recently tried to used the OVA export/import functionality. It
seems I misunderstood the intentions. I expected OVA export/import to
be reciprocal functionalities. I expected what we export as OVA to be
importable as OVA in ovirt.
What I have found is the following:
* It seems OVA export does export an OVA, though the format is not
spec compliant
* We use 'disk/' in the HostResource xml field instead of '/disk/'
for example in the ovf.
* It seems OVA import is intended specifically for VMWare OVA's?
* I thought I read about export domains being deprecated, and
incorrectly assumed the recently added OVA export/import was to make
things better for one off import/export.
Can anyone clarify the following:
* Is OVA import really just VMWare OVA's?
* Are we really not spec compliant with the OVA/OVF format as my
example above shows? OR do I misunderstand something?
* After spending my time yesterday, I recall now that the deprecation
of export domain is in favor of detaching and re-attaching ANY data
domain.
* What is the purpose of our OVA export if we ourselves indeed cannot
import our own OVA? Or can you tell me how we import our own if I'm
wrong.
Please feel free to overshare details :), as any context will help
prevent more confusion on my part.
~Kyle
6 years, 5 months
Ovirt-Hosted-Engine-Install-Purge Ansible Script Snippet
by RabidCicada
All,
I put together a full purge ansible script for CentOS. I figured I'd
share it here. I ran into many issues with spurious failures of
installs when I would try to reinstall ovirt on CentOS after having
"removed" it.
I was developing an automated install of our own software along with
ovirt but along the way I had to reinstall on the same box numerous
times. Initially I'd get spurious failures from leftover cruft that
would get in the way. I eventually found
this:https://www.ovirt.org/documentation/how-to/hosted-engine/#recoving-f...
and then crafted my own ansible purge script.
It uses ovirts own engine cleanup script first if it exists, then
fully cleans every other thing that can affect it.
Tested on CentOS, probably will work on other distros without too much
trouble. Hopefully someone saves themselves some grief
- name: Clean Old Install
#This attempts to remove all old cruft from previous install attempts
#The reason we include the ovirt packages is so that they can be reinstall
#At potentially newer versions along with dependency packages
block:
- name: Detect existing cleanup script
shell: which ovirt-hosted-engine-cleanup | cat
register: ohes_cleanup
- name: Debug ohes_cleanup.stdout
debug:
var: ohes_cleanup.stdout
- name: Run Ovirt's Hosted Engine Cleanup Script
shell: ovirt-hosted-engine-cleanup -q
when: ohes_cleanup.stdout != ""
- name: Clean old packages
package:
name: "{{item}}"
state: absent
with_items:
- "*vdsm*"
- "*ovirt*"
- "*libvirt*"
- "*cockpit*"
- name: Remove old configs etc
shell: "rm -rf /etc/{{item}}"
args:
warn: False
with_items:
- "/etc/*ovirt*"
- "/etc/*vdsm*"
- "/etc/libvirt/qemu/HostedEngine*"
- "/etc/*libvirt*"
- "/etc/guacamole"
- "/etc/pki/vdsm"
- "/etc/pki/libvirt"
- "/etc/pki/CA"
- "/etc/pki/keystore"
- "/etc/ovirt-hosted-engine"
- "/var/lib/libvirt/"
- "/var/lib/vdsm/"
- "/var/lib/ovirt-hosted-engine-*"
- "/var/log/ovirt-hosted-engine-setup/"
- "/var/cache/libvirt/"
- "/etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml"
- "/var/cache/tomcat/temp/*"
- "/var/cache/tomcat/work/Catalina/localhost/*"
- "/usr/share/tomcat/webapps/guacamole"
- "/usr/share/tomcat/webapps/guacamole.war"
- "/etc/guacamole/extensions/guacamole*"
- name: Clean old repo files
shell: "rm -rf /etc/yum.repos.d/{{item}}"
args:
warn: False
with_items:
- "ovirt*"
- "virt*"
- name: clean interface configs
shell: "rm -rf /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt"
args:
warn: False
- name: clean network stuff
shell: "{{item}}"
args:
warn: False
with_items:
- "brctl delbr ovirtmgmt | cat"
- "ip link del ovirtmgmt | cat"
- "ip link del dummy0 | cat"
- "ip link del virbr0 | cat"
- "ip link del virbr0-nic | cat"
- 'ip link del \;vdsmdummy\; | cat'
6 years, 5 months
LDAP logins do not work
by Michael Watters
I've ran the ovirt-engine-extension-aaa-ldap-setup command to configure
LDAP authentication using Active Directory however I am unable to
authenticate using valid credentials. Here is the output show while
testing the login flow.
[ INFO ] Executing login sequence...
Login output:
2018-06-13 11:27:17,931-04 INFO
========================================================================
2018-06-13 11:27:17,960-04 INFO
============================ Initialization ============================
2018-06-13 11:27:17,960-04 INFO
========================================================================
2018-06-13 11:27:17,999-04 INFO Loading extension
'example.com-authn'
2018-06-13 11:27:18,072-04 INFO Extension
'example.com-authn' loaded
2018-06-13 11:27:18,077-04 INFO Loading extension
'example.com-authz'
2018-06-13 11:27:18,089-04 INFO Extension
'example.com-authz' loaded
2018-06-13 11:27:18,090-04 INFO Initializing extension
'example.com-authn'
2018-06-13 11:27:18,091-04 INFO
[ovirt-engine-extension-aaa-ldap.authn::example.com-authn] Creating LDAP
pool 'authz'
2018-06-13 11:27:19,574-04 WARNING Exception: 80090308:
LdapErr: DSID-0C09042A, comment: AcceptSecurityContext error, data 52e,
v3839
2018-06-13 11:27:19,576-04 INFO
[ovirt-engine-extension-aaa-ldap.authn::example.com-authn] Creating LDAP
pool 'authn'
2018-06-13 11:27:20,668-04 INFO
[ovirt-engine-extension-aaa-ldap.authn::example.com-authn] LDAP pool
'authn' information: vendor='null' version='null'
2018-06-13 11:27:20,674-04 WARNING Ignoring records from pool:
'authz'
2018-06-13 11:27:20,676-04 WARNING Ignoring records from pool:
'authz'
2018-06-13 11:27:20,676-04 INFO Extension
'example.com-authn' initialized
2018-06-13 11:27:20,677-04 INFO Initializing extension
'example.com-authz'
2018-06-13 11:27:20,679-04 INFO
[ovirt-engine-extension-aaa-ldap.authz::example.com-authz] Creating LDAP
pool 'authz'
2018-06-13 11:27:21,270-04 WARNING Exception: 80090308:
LdapErr: DSID-0C09042A, comment: AcceptSecurityContext error, data 52e,
v3839
2018-06-13 11:27:21,273-04 INFO
[ovirt-engine-extension-aaa-ldap.authz::example.com-authz] Creating LDAP
pool 'gc'
2018-06-13 11:27:22,065-04 WARNING Exception: 80090308:
LdapErr: DSID-0C090400, comment: AcceptSecurityContext error, data 52e,
v1db1
2018-06-13 11:27:22,069-04 WARNING Ignoring records from pool:
'authz'
2018-06-13 11:27:22,072-04 WARNING Ignoring records from pool:
'authz'
2018-06-13 11:27:22,085-04 WARNING Ignoring records from pool:
'authz'
2018-06-13 11:27:22,086-04 INFO
[ovirt-engine-extension-aaa-ldap.authz::example.com-authz] Available
Namespaces: []
2018-06-13 11:27:22,087-04 INFO Extension
'example.com-authz' initialized
2018-06-13 11:27:22,088-04 INFO Start of enabled extensions
list
2018-06-13 11:27:22,089-04 INFO Instance name:
'example.com-authz', Extension name:
'ovirt-engine-extension-aaa-ldap.authz', Version: '1.3.7', Notes:
'Display name: ovirt-engine-extension-aaa-ldap-1.3.7-1.el7.centos',
License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt
Project', Build interface Version: '0', File:
'/tmp/tmpPQluAI/extensions.d/example.com-authz.properties', Initialized:
'true'
2018-06-13 11:27:22,089-04 INFO Instance name:
'example.com-authn', Extension name:
'ovirt-engine-extension-aaa-ldap.authn', Version: '1.3.7', Notes:
'Display name: ovirt-engine-extension-aaa-ldap-1.3.7-1.el7.centos',
License: 'ASL 2.0', Home: 'http://www.ovirt.org', Author 'The oVirt
Project', Build interface Version: '0', File:
'/tmp/tmpPQluAI/extensions.d/example.com-authn.properties', Initialized:
'true'
2018-06-13 11:27:22,090-04 INFO End of enabled extensions list
2018-06-13 11:27:22,090-04 INFO
========================================================================
2018-06-13 11:27:22,090-04 INFO
============================== Execution ===============================
2018-06-13 11:27:22,091-04 INFO
========================================================================
2018-06-13 11:27:22,091-04 INFO Iteration: 0
2018-06-13 11:27:22,093-04 INFO Profile='example.com'
authn='example.com-authn' authz='example.com-authz' mapping='null'
2018-06-13 11:27:22,094-04 INFO API:
-->Authn.InvokeCommands.AUTHENTICATE_CREDENTIALS profile='example.com'
user='d861703'
2018-06-13 11:27:22,251-04 INFO API:
<--Authn.InvokeCommands.AUTHENTICATE_CREDENTIALS profile='example.com'
result=CREDENTIALS_INCORRECT
2018-06-13 11:27:22,262-04 SEVERE Authn.Result code is:
CREDENTIALS_INCORRECT
[ ERROR ] Login sequence failed
Does anybody know what LdapErr: DSID-0C09042A, comment:
AcceptSecurityContext error, data 52e, v3839 means? Is this a TLS
issue? I am quite certain the password I'm using is correct.
6 years, 5 months
VM created is reachable from only oVirt node
by sweettriplek＠gmail.com
VM running with centos is not reaching to any other network other than oVirt node. I tried to apply no network filer, clean-filter while creating vNIC interfaces but no luck.
Setup configurations:
- oVirt Engine and Node running on different machines
- OVirt running on cent OS
Please help me to resolve this issue, thanks in advance.
6 years, 5 months
guests crashing during live migration(NUMA config issue)
by Balg, Andreas
Hello,
during updates of our physical nodes running ovirt 4.2.3 I had to live-
migrate all VMs to evacuate them from the hosts. This caused roughly
10% of guests to end up crashed /shutdown after live migration.
Errors in the Logs are:
2018-05-31T15:15:51.273805Z qemu-kvm: warning: All CPU(s) up to maxcpus
should be described in NUMA config, ability to start up with partial
NUMA mappings is obsoleted and will be removed in future
2018-05-31T15:16:54.596554Z qemu-kvm: Unknown combination of migration
flags: 0
2018-05-31T15:16:54.597196Z qemu-kvm: error while loading state section
id 3(ram)
2018-05-31T15:16:54.598491Z qemu-kvm: load of migration failed: Invalid
argument
2018-05-31 15:16:55.010+0000: shutting down, reason=crashed
is there anything I can do about this? The hardware of all nodes is
100% identical
--
Andreas Balg
6 years, 5 months
