Unsynced entries do not self-heal during upgrade from oVirt 4.2 -> 4.3
by Goorkate, B.J.
Hi all,
I'm in the process of upgrading oVirt-nodes from 4.2 to 4.3.
After upgrading the first of 3 oVirt/gluster nodes, there are between 600-1200 unsynced entries for a week now on 1 upgraded node and one not-yet-upgraded node. The third node (also not-yet-upgraded) says it's OK (no unsynced entries).
The cluster doesn't seem to be very busy, but somehow self-heal doesn't complete.
Is this because of different gluster versions across the nodes and will it resolve as soon as I upgraded all nodes? Since it's our production cluster, I don't want to take any risk...
Does anybody recognise this problem? Of course I can provide more information if necessary.
Any hints on troubleshooting the unsynced entries are more than welcome!
Thanks in advance!
Regards,
Bertjan
------------------------------------------------------------------------------
De informatie opgenomen in dit bericht kan vertrouwelijk zijn en is
uitsluitend bestemd voor de geadresseerde. Indien u dit bericht onterecht
ontvangt, wordt u verzocht de inhoud niet te gebruiken en de afzender direct
te informeren door het bericht te retourneren. Het Universitair Medisch
Centrum Utrecht is een publiekrechtelijke rechtspersoon in de zin van de W.H.W.
(Wet Hoger Onderwijs en Wetenschappelijk Onderzoek) en staat geregistreerd bij
de Kamer van Koophandel voor Midden-Nederland onder nr. 30244197.
Denk s.v.p aan het milieu voor u deze e-mail afdrukt.
------------------------------------------------------------------------------
This message may contain confidential information and is intended exclusively
for the addressee. If you receive this message unintentionally, please do not
use the contents but notify the sender immediately by return e-mail. University
Medical Center Utrecht is a legal person by public law and is registered at
the Chamber of Commerce for Midden-Nederland under no. 30244197.
Please consider the environment before printing this e-mail.
3 years, 1 month
Vm suddenly paused with error "vm has paused due to unknown storage error"
by Jasper Siero
Hi all,
Since we upgraded our Ovirt nodes to CentOS 7 a vm (not a specific one but never more then one) will sometimes pause suddenly with the error "VM ... has paused due to unknown storage error". It happens now two times in a month.
The Ovirt node uses san storage for the vm's running on it. When a specific vm is pausing with an error the other vm's keeps running without problems.
The vm runs without problems after unpausing it.
Versions:
CentOS Linux release 7.1.1503
vdsm-4.14.17-0
libvirt-daemon-1.2.8-16
vdsm.log:
VM Channels Listener::DEBUG::2015-10-25 07:43:54,382::vmChannels::95::vds::(_handle_timeouts) Timeout on fileno 78.
libvirtEventLoop::INFO::2015-10-25 07:43:56,177::vm::4602::vm.Vm::(_onIOError) vmId=`77f07ae0-cc3e-4ae2-90ec-7fba7b11deeb`::abnormal vm stop device virtio-disk0 error eother
libvirtEventLoop::DEBUG::2015-10-25 07:43:56,178::vm::5204::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`77f07ae0-cc3e-4ae2-90ec-7fba7b11deeb`::event Suspended detail 2 opaque None
libvirtEventLoop::INFO::2015-10-25 07:43:56,178::vm::4602::vm.Vm::(_onIOError) vmId=`77f07ae0-cc3e-4ae2-90ec-7fba7b11deeb`::abnormal vm stop device virtio-disk0 error eother
...........
libvirtEventLoop::INFO::2015-10-25 07:43:56,180::vm::4602::vm.Vm::(_onIOError) vmId=`77f07ae0-cc3e-4ae2-90ec-7fba7b11deeb`::abnormal vm stop device virtio-disk0 error eother
specific error part in libvirt vm log:
block I/O error in device 'drive-virtio-disk0': Unknown error 32758 (32758)
...........
block I/O error in device 'drive-virtio-disk0': Unknown error 32758 (32758)
engine.log:
2015-10-25 07:44:48,945 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) [a43dcc8] VM diataal-prod-cas1 77f07ae0-cc3e-4ae2-90ec-7fba7b11deeb moved from
Up --> Paused
2015-10-25 07:44:49,003 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [a43dcc8] Correlation ID: null, Call Stack: null, Custom Event
ID: -1, Message: VM diataal-prod-cas1 has paused due to unknown storage error.
Has anyone experienced the same problem or knows a way to solve this?
Kind regards,
Jasper
3 years, 1 month
hosted-engine --deploy fails after "Wait for the host to be up" task
by Fredy Sanchez
*Hi all,*
*[root@bric-ovirt-1 ~]# cat /etc/*release**
CentOS Linux release 7.7.1908 (Core)
*[root@bric-ovirt-1 ~]# yum info ovirt-engine-appliance*
Installed Packages
Name : ovirt-engine-appliance
Arch : x86_64
Version : 4.3
Release : 20191121.1.el7
Size : 1.0 G
Repo : installed
From repo : ovirt-4.3
*Same situation as https://bugzilla.redhat.com/show_bug.cgi?id=1787267
<https://bugzilla.redhat.com/show_bug.cgi?id=1787267>. The error message
almost everywhere is some red herring message about ansible*
[ INFO ] TASK [ovirt.hosted_engine_setup : Wait for the host to be up]
[ ERROR ] fatal: [localhost]: FAILED! => {"ansible_facts": {"ovirt_hosts":
[]}, "attempts": 120, "changed": false, "deprecations": [{"msg": "The
'ovirt_host_facts' module has been renamed to 'ovirt_host_info', and the
renamed one no longer returns ansible_facts", "version": "2.13"}]}
[ INFO ] TASK [ovirt.hosted_engine_setup : Notify the user about a failure]
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The
system may not be provisioned according to the playbook results: please
check the logs for the issue, fix accordingly or re-deploy from scratch.\n"}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing
ansible-playbook
[ INFO ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: please check the logs for the
issue, fix accordingly or re-deploy from scratch.
Log file is located at
/var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20200126170315-req4qb.log
*But the "real" problem seems to be SSH related, as you can see below*
*[root@bric-ovirt-1 ovirt-engine]# pwd*
/var/log/ovirt-hosted-engine-setup/engine-logs-2020-01-26T17:19:28Z/ovirt-engine
*[root@bric-ovirt-1 ovirt-engine]# grep -i error engine.log*
2020-01-26 17:26:50,178Z ERROR
[org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand] (default task-1)
[2341fd23-f0c7-4f1c-ad48-88af20c2d04b] Failed to establish session with
host 'bric-ovirt-1.corp.modmed.com': SSH session closed during connection '
root(a)bric-ovirt-1.corp.modmed.com'
2020-01-26 17:26:50,205Z ERROR
[org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default
task-1) [] Operation Failed: [Cannot add Host. Connecting to host via SSH
has failed, verify that the host is reachable (IP address, routable address
etc.) You may refer to the engine.log file for further details.]
*The funny thing is that the engine can indeed ssh to bric-ovirt-1
(physical host). See below*
*[root@bric-ovirt-1 ovirt-hosted-engine-setup]# cat /etc/hosts*
192.168.1.52 bric-ovirt-engine.corp.modmed.com # temporary entry added by
hosted-engine-setup for the bootstrap VM
127.0.0.1 localhost localhost.localdomain localhost4
localhost4.localdomain4
#::1 localhost localhost.localdomain localhost6
localhost6.localdomain6
10.130.0.50 bric-ovirt-engine bric-ovirt-engine.corp.modmed.com
10.130.0.51 bric-ovirt-1 bric-ovirt-1.corp.modmed.com
10.130.0.52 bric-ovirt-2 bric-ovirt-2.corp.modmed.com
10.130.0.53 bric-ovirt-3 bric-ovirt-3.corp.modmed.com
192.168.0.1 bric-ovirt-1gluster bric-ovirt-1gluster.corp.modmed.com
192.168.0.2 bric-ovirt-2gluster bric-ovirt-2gluster.corp.modmed.com
192.168.0.3 bric-ovirt-3gluster bric-ovirt-3gluster.corp.modmed.com
[root@bric-ovirt-1 ovirt-hosted-engine-setup]#
*[root@bric-ovirt-1 ~]# ssh 192.168.1.52*
Last login: Sun Jan 26 17:55:20 2020 from 192.168.1.1
[root@bric-ovirt-engine ~]#
[root@bric-ovirt-engine ~]#
*[root@bric-ovirt-engine ~]# ssh bric-ovirt-1*
Password:
Password:
Last failed login: Sun Jan 26 18:17:16 UTC 2020 from 192.168.1.52 on
ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Sun Jan 26 18:16:46 2020
###################################################################
# UNAUTHORIZED ACCESS TO THIS SYSTEM IS PROHIBITED #
# #
# This system is the property of Modernizing Medicine, Inc. #
# It is for authorized Company business purposes only. #
# All connections are monitored and recorded. #
# Disconnect IMMEDIATELY if you are not an authorized user! #
###################################################################
[root@bric-ovirt-1 ~]#
[root@bric-ovirt-1 ~]#
[root@bric-ovirt-1 ~]# exit
logout
Connection to bric-ovirt-1 closed.
[root@bric-ovirt-engine ~]#
[root@bric-ovirt-engine ~]#
*[root@bric-ovirt-engine ~]# ssh bric-ovirt-1.corp.modmed.com
<http://bric-ovirt-1.corp.modmed.com>*
Password:
Last login: Sun Jan 26 18:17:22 2020 from 192.168.1.52
###################################################################
# UNAUTHORIZED ACCESS TO THIS SYSTEM IS PROHIBITED #
# #
# This system is the property of Modernizing Medicine, Inc. #
# It is for authorized Company business purposes only. #
# All connections are monitored and recorded. #
# Disconnect IMMEDIATELY if you are not an authorized user! #
###################################################################
[root@bric-ovirt-1 ~]# exit
logout
Connection to bric-ovirt-1.corp.modmed.com closed.
[root@bric-ovirt-engine ~]#
[root@bric-ovirt-engine ~]#
[root@bric-ovirt-engine ~]# exit
logout
Connection to 192.168.1.52 closed.
[root@bric-ovirt-1 ~]#
*So, what gives? I already disabled all ssh security in the physical host,
and whitelisted all potential IPs from the engine using firewalld.
Regardless, the engine can ssh to the host as root :-(. Is there maybe
another user that's used for the "Wait for the host to be up" SSH test?
Yes, I tried both passwords and certificates.*
*Maybe what's really happening is that engine is not getting the right IP?
bric-ovirt-engine is supposed to get 10.130.0.50, instead it never gets
there, getting 192.168.1.52 from virbr0 in bric-ovirt-1. See below.*
--== HOST NETWORK CONFIGURATION ==--
Please indicate the gateway IP address [10.130.0.1]
Please indicate a nic to set ovirtmgmt bridge on: (p4p1, p5p1)
[p4p1]:
--== VM CONFIGURATION ==--
You may specify a unicast MAC address for the VM or accept a randomly
generated default [00:16:3e:17:1d:f8]:
How should the engine VM network be configured (DHCP,
Static)[DHCP]? static
Please enter the IP address to be used for the engine VM []:
10.130.0.50
[ INFO ] The engine VM will be configured to use 10.130.0.50/25
Please provide a comma-separated list (max 3) of IP addresses of
domain name servers for the engine VM
Engine VM DNS (leave it empty to skip) [10.130.0.2,10.130.0.3]:
Add lines for the appliance itself and for this host to
/etc/hosts on the engine VM?
Note: ensuring that this host could resolve the engine VM
hostname is still up to you
(Yes, No)[No] Yes
*[root@bric-ovirt-1 ~]# ip addr*
3: p4p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group
default qlen 1000
link/ether 00:0a:f7:f1:c6:80 brd ff:ff:ff:ff:ff:ff
inet 10.130.0.51/25 brd 10.130.0.127 scope global noprefixroute p4p1
valid_lft forever preferred_lft forever
28: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
UP group default qlen 1000
link/ether 52:54:00:25:7b:6f brd ff:ff:ff:ff:ff:ff
inet 192.168.1.1/24 brd 192.168.1.255 scope global virbr0
valid_lft forever preferred_lft forever
29: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master
virbr0 state DOWN group default qlen 1000
link/ether 52:54:00:25:7b:6f brd ff:ff:ff:ff:ff:ff
30: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UNKNOWN group default qlen 1000
link/ether fe:16:3e:17:1d:f8 brd ff:ff:ff:ff:ff:ff
*The newly created engine VM does remain up even after hosted-engine
--deploy errors out; just at the wrong IP. I haven't been able to make it
get its real IP. At any rate, thank you very much for taking a look at my
very long email. Any and all help would be really appreciated.*
Cheers,
--
Fredy
--
*CONFIDENTIALITY NOTICE:* This e-mail message may contain material
protected by the Health Insurance Portability and Accountability Act of
1996 and its implementing regulations and other state and federal laws and
legal privileges. This message is only for the personal and confidential
use of the individuals or organization to whom the message is addressed. If
you are an unintended recipient, you have received this message in error,
and any reading, distributing, copying or disclosure is unauthorized and
strictly prohibited. All recipients are hereby notified that any
unauthorized receipt does not waive any confidentiality obligations or
privileges. If you have received this message in error, please notify the
sender immediately at the above email address and confirm that you have
deleted or destroyed the message.
3 years, 1 month
Re: Cannot Increase Hosted Engine VM Memory
by Serhiy Morhun
Hello, did anyone find a resolution for this issue? I'm having exactly the
same problem:
Hosted Engine VM is running with 5344MB of RAM, when trying to increase to
8192 it would not accept the change because the difference is not divisible
by 256.
When trying to increase to 8160 the change is accepted but log shows
"Hotset memory: changed the amount of memory on VM HostedEngine from 5344
to 5344" at the same time amount of guaranteed memory does increase to 8160
which, in turn, starts generating error messages that VM does not have all
the guaranteed RAM.
Serhiy Morhun
--
*-----------------------------------------------------------------------------------***
*THE INFORMATION CONTAINED IN THIS MESSAGE (E-MAIL AND ANY ATTACHMENTS) IS
INTENDED ONLY FOR THE INDIVIDUAL AND CONFIDENTIAL USE OF THE DESIGNATED
RECIPIENT(S).*
If any reader of this message is not an intended recipient
or any agent responsible for delivering it to an intended recipient, you
are hereby notified that you have received this document in error, and that
any review, dissemination, distribution, copying or other use of this
message is prohibited. If you have received this message in error, please
notify us immediately by reply e-mail message or by telephone and delete
the original message from your e-mail system and/or computer database.
Thank you.
*-----------------------------------------------------------------------------------*
**NOTICE**:
*You are advised that e-mail correspondence and attachments
between the public and the Ridgewood Board of Education are obtainable by
any person who files a request under the NJ Open Public Records Act (OPRA)
unless it is subject to a specific OPRA exception. You should have no
expectation that the content of e-mails sent to or from school district
e-mail addresses, or between the public and school district officials and
employees, will remain private.*
*-----------------------------------------------------------------------------------*
3 years, 1 month
[ANN] oVirt 4.3.9 First Release Candidate is now available for testing
by Sandro Bonazzola
The oVirt Project is pleased to announce the availability of the oVirt
4.3.9 First Release Candidate for testing, as of January 30th, 2020.
This update is a release candidate of the nineth in a series of
stabilization updates to the 4.3 series.
This is pre-release software. This pre-release should not to be used in
production.
This release is available now on x86_64 architecture for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
This release supports Hypervisor Hosts on x86_64 and ppc64le architectures
for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
* oVirt Node 4.3 (available for x86_64 only) has been built consuming
CentOS 7.7 Release
See the release notes [1] for known issues, new features and bugs fixed.
Notes:
- oVirt Appliance is already available
- oVirt Node will be available soon
Additional Resources:
* Read more about the oVirt 4.3.9 release highlights:
http://www.ovirt.org/release/4.3.9/
* Get more oVirt Project updates on Twitter: https://twitter.com/ovirt
* Check out the latest project news on the oVirt blog:
http://www.ovirt.org/blog/
[1] http://www.ovirt.org/release/4.3.9/
[2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*Red Hat respects your work life balance.
Therefore there is no need to answer this email out of your office hours.*
3 years, 1 month
oVirt MAC Pool question
by Vrgotic, Marko
Dear oVirt,
While investigating and DHCP & DDNS collision issues between two VM servers from different oVirt clusters, I noticed that oVirt assigns same default MAC range for each of it’s managed clusters.
Question1: Does oVirt-Engine keep separate place in DB or … for MAC addresses assigned per cluster or it keeps them all in same place?
Question2: Would there be an harming effect on existing VMs if the default mac pool would be changed?
Additional info:
Self Hosted ovirt-engine – 4.3.4 and 4.3.7
-----
kind regards/met vriendelijke groeten
Marko Vrgotic
ActiveVideo
3 years, 1 month
Deploy Hosted Engine fails at "Set VLAN ID at datacenter level"
by Guillaume Pavese
Hi,
Trying to deploy ovirt 4.3-stable Hosted Engine with cockpit
This fails with the following :
[ INFO ] TASK [ovirt.hosted_engine_setup : Set VLAN ID at datacenter level]
[ ERROR ] Exception: Entity 'None' was not found.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Entity
'None' was not found."}
Any idea?
Guillaume Pavese
Ingénieur Système et Réseau
Interactiv-Group
--
Ce message et toutes les pièces jointes (ci-après le “message”) sont
établis à l’intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur, merci de le détruire et d’en avertir
immédiatement l’expéditeur. Toute utilisation de ce message non conforme a
sa destination, toute diffusion ou toute publication, totale ou partielle,
est interdite, sauf autorisation expresse. L’internet ne permettant pas
d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales)
décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse
ou il aurait été modifié. IT, ES, UK.
<https://interactiv-group.com/disclaimer.html>
3 years, 1 month
Understanding ovirt memory management which appears incorrect
by divan＠santanas.co.za
Hi All,
A question regarding memory management with ovirt. I know memory can
be complicated hence I'm asking the experts. :)
Two examples of where it looks - to me - that memory management from
ovirt perspective is incorrect. This is resulting in us not getting as
much out of a host as we'd expect.
## Example 1:
host: dev-cluster-04
I understand the mem on the host to be:
128G total (physical)
68G used
53G available
56G buff/cache
I understand therefore 53G should still be available to allocate
(approximately, minus a few things).
```
DEV [root@dev-cluster-04:~] # free -m
total used free shared buff/cache available
Mem: 128741 68295 4429 4078 56016 53422
Swap: 12111 1578 10533
DEV [root@dev-cluster-04:~] # cat /proc/meminfo
MemTotal: 131831292 kB
MemFree: 4540852 kB
MemAvailable: 54709832 kB
Buffers: 3104 kB
Cached: 5174136 kB
SwapCached: 835012 kB
Active: 66943552 kB
Inactive: 5980340 kB
Active(anon): 66236968 kB
Inactive(anon): 5713972 kB
Active(file): 706584 kB
Inactive(file): 266368 kB
Unevictable: 50036 kB
Mlocked: 54132 kB
SwapTotal: 12402684 kB
SwapFree: 10786688 kB
Dirty: 812 kB
Writeback: 0 kB
AnonPages: 67068548 kB
Mapped: 143880 kB
Shmem: 4176328 kB
Slab: 52183680 kB
SReclaimable: 49822156 kB
SUnreclaim: 2361524 kB
KernelStack: 20000 kB
PageTables: 213628 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 78318328 kB
Committed_AS: 110589076 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 859104 kB
VmallocChunk: 34291324976 kB
HardwareCorrupted: 0 kB
AnonHugePages: 583680 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 621088 kB
DirectMap2M: 44439552 kB
DirectMap1G: 91226112 kB
```
The ovirt engine, compute -> hosts view shows s4-dev-cluster-01 as 93%
memory utilised.
Clicking on the node says:
Physical Memory: 128741 MB total, 119729 MB used, 9012 MB free
So ovirt engine says 9G free. The OS reports 4G free but 53G
available. Surely ovirt should be looking at available memory?
This is a problem, for instance, when trying to run a VM, called
dev-cassandra-01, with mem size 24576, max mem 24576 and mem
guarantee set to 10240 on this host it fails with:
```
Cannot run VM. There is no host that satisfies current scheduling
constraints. See below for details:
The host dev-cluster-04.fnb.co.za did not satisfy internal filter
Memory because its available memory is too low (19884 MB) to run the
VM.
```
To me this looks blatantly wrong. The host has 53G available according
to free -m.
Guessing I'm missing something, unless this is some sort of bug?
versions:
```
engine: 4.3.7.2-1.el7
host:
OS Version: RHEL - 7 - 6.1810.2.el7.centos
OS Description: CentOS Linux 7 (Core)
Kernel Version: 3.10.0 - 957.12.1.el7.x86_64
KVM Version: 2.12.0 - 18.el7_6.3.1
LIBVIRT Version: libvirt-4.5.0-10.el7_6.7
VDSM Version: vdsm-4.30.13-1.el7
SPICE Version: 0.14.0 - 6.el7_6.1
GlusterFS Version: [N/A]
CEPH Version: librbd1-10.2.5-4.el7
Open vSwitch Version: openvswitch-2.10.1-3.el7
Kernel Features: PTI: 1, IBRS: 0, RETP: 1, SSBD: 3
VNC Encryption: Disabled
```
## Example 2:
A ovirt host with two VMs:
According to the host, it has 128G of physical memory of which 56G is
used, 69G is buff/cache and 65G is available.
As is shown here:
```
LIVE [root@prod-cluster-01:~] # cat /proc/meminfo
MemTotal: 131326836 kB
MemFree: 2630812 kB
MemAvailable: 66573596 kB
Buffers: 2376 kB
Cached: 5670628 kB
SwapCached: 151072 kB
Active: 59106140 kB
Inactive: 2744176 kB
Active(anon): 58099732 kB
Inactive(anon): 2327428 kB
Active(file): 1006408 kB
Inactive(file): 416748 kB
Unevictable: 40004 kB
Mlocked: 42052 kB
SwapTotal: 4194300 kB
SwapFree: 3579492 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 56085040 kB
Mapped: 121816 kB
Shmem: 4231808 kB
Slab: 65143868 kB
SReclaimable: 63145684 kB
SUnreclaim: 1998184 kB
KernelStack: 25296 kB
PageTables: 148336 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 69857716 kB
Committed_AS: 76533164 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 842296 kB
VmallocChunk: 34291404724 kB
HardwareCorrupted: 0 kB
AnonHugePages: 55296 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 722208 kB
DirectMap2M: 48031744 kB
DirectMap1G: 87031808 kB
LIVE [root@prod-cluster-01:~] # free -m
total used free shared buff/cache available
Mem: 128248 56522 2569 4132 69157 65013
Swap: 4095 600 3495
```
However the compute -> hosts ovirt screen shows this node as 94%
memory.
Clicking compute -> hosts -> prod-cluster-01 -> general says:
Physical Memory: 128248 MB total, 120553 MB used, 7695 MB free
Swap Size: 4095 MB total, 600 MB used, 3495 MB free
The physical memory in the above makes no sense to me. Unless it
includes caches which I would think it shouldn't.
This host has just two VMs:
LIVE [root@prod-cluster-01:~] # virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf list
Id Name State
----------------------------------------------------
35 prod-box-18 running
36 prod-box-11 running
Moreover each VM has 32G memory set, in every possible place - from
what I can see.
```
LIVE [root@prod-cluster-01:~] # virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf dumpxml prod-box-11|grep -i mem
<ovirt-vm:memGuaranteedSize type="int">32768</ovirt-vm:memGuaranteedSize>
<ovirt-vm:minGuaranteedMemoryMb type="int">32768</ovirt-vm:minGuaranteedMemoryMb>
<memory unit='KiB'>33554432</memory>
<currentMemory unit='KiB'>33554432</currentMemory>
<cell id='0' cpus='0-27' memory='33554432' unit='KiB'/>
<suspend-to-mem enabled='no'/>
<model type='qxl' ram='65536' vram='32768' vgamem='16384' heads='1' primary='yes'/>
<memballoon model='virtio'>
</memballoon>
```
prod-box-11 is however set as high performance VM. That could cause a
problem.
Same for the other VM:
```
LIVE [root@prod-cluster-01:~] # virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf dumpxml prod-box-18|grep -i mem
<ovirt-vm:memGuaranteedSize type="int">32768</ovirt-vm:memGuaranteedSize>
<ovirt-vm:minGuaranteedMemoryMb type="int">32768</ovirt-vm:minGuaranteedMemoryMb>
<memory unit='KiB'>33554432</memory>
<currentMemory unit='KiB'>33554432</currentMemory>
<cell id='0' cpus='0-27' memory='33554432' unit='KiB'/>
<suspend-to-mem enabled='no'/>
<model type='qxl' ram='65536' vram='32768' vgamem='16384' heads='1' primary='yes'/>
<memballoon model='virtio'>
</memballoon>
```
So I understand that two VMs each having allocated 32G of ram should
consume approx 64G of ram on the host. The host has 128G of ram, so
usage should be at approx 50%. However ovirt is reporting 94% usage.
Versions:
```
engine: 4.3.5.5-1.el7
host:
OS Version: RHEL - 7 - 6.1810.2.el7.centos
OS Description: CentOS Linux 7 (Core)
Kernel Version: 3.10.0 - 957.10.1.el7.x86_64
KVM Version: 2.12.0 - 18.el7_6.3.1
LIBVIRT Version: libvirt-4.5.0-10.el7_6.6
VDSM Version: vdsm-4.30.11-1.el7
SPICE Version: 0.14.0 - 6.el7_6.1
GlusterFS Version: [N/A]
CEPH Version: librbd1-10.2.5-4.el7
Open vSwitch Version: openvswitch-2.10.1-3.el7
Kernel Features: PTI: 1, IBRS: 0, RETP: 1
VNC Encryption: Disabled
```
Thanks for any insights!
--
Divan Santana
https://divansantana.com
3 years, 1 month
Re: Failed to synchronize networks of Provider ovirt-provider-ovn
by Mail SET Inc. Group
Yes, i use same manual to change WebUI SSL.
ovirt-ca-file= is a same SSL file which use WebUI.
Yes, i restart ovirt-provider-ovn, i restart engine, i restart all what i can restart. Nothing...
> 12 сент. 2018 г., в 16:11, Dominik Holler <dholler(a)redhat.com> написал(а):
>
> On Wed, 12 Sep 2018 14:23:54 +0300
> "Mail SET Inc. Group" <mail(a)set-pro.net> wrote:
>
>> Ok!
>
> Not exactly, please use users(a)ovirt.org for such questions.
> Other should benefit from this questions, too.
> Please write the next mail to users(a)ovirt.org and keep me in CC.
>
>> What i did:
>>
>> 1) install oVirt «from box» (4.2.5.2-1.el7);
>> 2) generate own ssl for my engine using my FreeIPA CA, Install it and
>
> What means "Install it"? You can use the doc from the following link
> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/...
>
> Ensure that ovirt-ca-file= in
> /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf
> points to the correct file and ovirt-provider-ovn is restarted.
>
>> get tis issue;
>>
>>
>> [root@engine ~]# tail -n 50 /var/log/ovirt-provider-ovn.log
>> 2018-09-12 14:10:23,828 root [SSL: CERTIFICATE_VERIFY_FAILED]
>> certificate verify failed (_ssl.c:579) Traceback (most recent call
>> last): File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py",
>> line 133, in _handle_request method, path_parts, content
>> File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py",
>> line 175, in handle_request return
>> self.call_response_handler(handler, content, parameters) File
>> "/usr/share/ovirt-provider-ovn/handlers/keystone.py", line 33, in
>> call_response_handler return response_handler(content, parameters)
>> File "/usr/share/ovirt-provider-ovn/handlers/keystone_responses.py",
>> line 62, in post_tokens user_password=user_password) File
>> "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 26, in
>> create_token return auth.core.plugin.create_token(user_at_domain,
>> user_password) File
>> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/plugin.py", line
>> 48, in create_token timeout=self._timeout()) File
>> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 75,
>> in create_token username, password, engine_url, ca_file, timeout)
>> File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line
>> 91, in _get_sso_token timeout=timeout File
>> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54,
>> in wrapper response = func(*args, **kwargs) File
>> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47,
>> in wrapper raise BadGateway(e) BadGateway: [SSL:
>> CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
>>
>>
>> [root@engine ~]# tail -n 20 /var/log/ovirt-engine/engine.log
>> 2018-09-12 14:10:23,773+03 INFO
>> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
>> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Lock
>> Acquired to object
>> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]',
>> sharedLocks=''}' 2018-09-12 14:10:23,778+03 INFO
>> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
>> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685]
>> Running command: SyncNetworkProviderCommand internal: true.
>> 2018-09-12 14:10:23,836+03 ERROR
>> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
>> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685]
>> Command
>> 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand'
>> failed: EngineException: (Failed with error Bad Gateway and code
>> 5050) 2018-09-12 14:10:23,837+03 INFO
>> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
>> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Lock
>> freed to object
>> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]',
>> sharedLocks=''}' 2018-09-12 14:14:12,477+03 INFO
>> [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default
>> task-6) [] User admin@internal successfully logged in with scopes:
>> ovirt-app-admin ovirt-app-api ovirt-app-portal
>> ovirt-ext=auth:sequence-priority=~ ovirt-ext=revoke:revoke-all
>> ovirt-ext=token-info:authz-search
>> ovirt-ext=token-info:public-authz-search
>> ovirt-ext=token-info:validate ovirt-ext=token:password-access
>> 2018-09-12 14:14:12,587+03 INFO
>> [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default
>> task-6) [1bf1b763] Running command: CreateUserSessionCommand
>> internal: false. 2018-09-12 14:14:12,628+03 INFO
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (default task-6) [1bf1b763] EVENT_ID: USER_VDC_LOGIN(30), User
>> admin@internal-authz connecting from '10.0.3.61' using session
>> 's8jAm7BUJGlicthm6yZBA3CUM8QpRdtwFaK3M/IppfhB3fHFB9gmNf0cAlbl1xIhcJ2WX+ww7e71Ri+MxJSsIg=='
>> logged in. 2018-09-12 14:14:30,972+03 INFO
>> [org.ovirt.engine.core.bll.provider.ImportProviderCertificateCommand]
>> (default task-6) [ee3cc8a7-4485-4fdf-a0c2-e9d67b5cfcd3] Running
>> command: ImportProviderCertificateCommand internal: false. Entities
>> affected : ID: aaa00000-0000-0000-0000-123456789aaa Type:
>> SystemAction group CREATE_STORAGE_POOL with role type ADMIN
>> 2018-09-12 14:14:30,982+03 INFO
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (default task-6) [ee3cc8a7-4485-4fdf-a0c2-e9d67b5cfcd3] EVENT_ID:
>> PROVIDER_CERTIFICATE_IMPORTED(213), Certificate for provider
>> ovirt-provider-ovn was imported. (User: admin@internal-authz)
>> 2018-09-12 14:14:31,006+03 INFO
>> [org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand]
>> (default task-6) [a48d94ab-b0b2-42a2-a667-0525b4c652ea] Running
>> command: TestProviderConnectivityCommand internal: false. Entities
>> affected : ID: aaa00000-0000-0000-0000-123456789aaa Type:
>> SystemAction group CREATE_STORAGE_POOL with role type ADMIN
>> 2018-09-12 14:14:31,058+03 ERROR
>> [org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand]
>> (default task-6) [a48d94ab-b0b2-42a2-a667-0525b4c652ea] Command
>> 'org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand'
>> failed: EngineException: (Failed with error Bad Gateway and code
>> 5050) 2018-09-12 14:15:10,954+03 INFO
>> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
>> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread
>> pool 'default' is using 0 threads out of 1, 5 threads waiting for
>> tasks. 2018-09-12 14:15:10,954+03 INFO
>> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
>> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread
>> pool 'engine' is using 0 threads out of 500, 16 threads waiting for
>> tasks and 0 tasks in queue. 2018-09-12 14:15:10,954+03 INFO
>> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
>> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread
>> pool 'engineScheduled' is using 0 threads out of 100, 100 threads
>> waiting for tasks. 2018-09-12 14:15:10,954+03 INFO
>> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
>> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread
>> pool 'engineThreadMonitoring' is using 1 threads out of 1, 0 threads
>> waiting for tasks. 2018-09-12 14:15:10,954+03 INFO
>> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
>> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread
>> pool 'hostUpdatesChecker' is using 0 threads out of 5, 2 threads
>> waiting for tasks. 2018-09-12 14:15:23,843+03 INFO
>> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
>> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Lock
>> Acquired to object
>> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]',
>> sharedLocks=''}' 2018-09-12 14:15:23,849+03 INFO
>> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
>> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f]
>> Running command: SyncNetworkProviderCommand internal: true.
>> 2018-09-12 14:15:23,900+03 ERROR
>> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
>> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f]
>> Command
>> 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand'
>> failed: EngineException: (Failed with error Bad Gateway and code
>> 5050) 2018-09-12 14:15:23,901+03 INFO
>> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
>> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Lock
>> freed to object
>> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]',
>> sharedLocks=''}'
>>
>>
>> [root@engine ~]#
>> cat /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf #
>> This file is automatically generated by engine-setup. Please do not
>> edit manually [OVN REMOTE] ovn-remote=ssl:127.0.0.1:6641
>> [SSL]
>> https-enabled=true
>> ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem
>> ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer
>> ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass
>> [OVIRT]
>> ovirt-sso-client-secret=Ms7Gw9qNT6IkXu7oA54tDmxaZDIukABV
>> ovirt-host=https://engine.set.local:443
>> ovirt-sso-client-id=ovirt-provider-ovn
>> ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem
>> [PROVIDER]
>> provider-host=engine.set.local
>>
>>
>>> 12 сент. 2018 г., в 13:59, Dominik Holler <dholler(a)redhat.com>
>>> написал(а):
>>>
>>> On Wed, 12 Sep 2018 13:04:53 +0300
>>> "Mail SET Inc. Group" <mail(a)set-pro.net> wrote:
>>>
>>>> Hello Dominik!
>>>> I have a same issue with OVN provider and SSL
>>>> https://www.mail-archive.com/users@ovirt.org/msg47020.html
>>>> <https://www.mail-archive.com/users@ovirt.org/msg47020.html> But
>>>> certificate changes not helps to resolve it. Maybe you can help me
>>>> with this?
>>>
>>> Sure. Can you please share the relevant lines of
>>> ovirt-provider-ovn.log and engine.log, and the information if you
>>> are using the certificates generated by engine-setup with
>>> users(a)ovirt.org ? Thanks,
>>> Dominik
>>>
>>
>
>
3 years, 1 month
Two host cluster without hyperconverged
by Göker Dalar
Hello everyone,
I want to get an idea in this topic.
I have two servers with the same capabilities and 8 same physical disc per
node. I want to setup a cluster using a redundant disc.I dont have another
server for gluster hyperconverged . How i should build for this
structure ?
Thanks in advance,
Göker
3 years, 1 month
