What happens in loss of network scenarios?
by David White
I tested oVirt (4.3? I can't remember) last fall on a single host (hyperconverged).
Now, I'm getting ready to deploy to a 3 physical node (possibly 4) hyperconverged cluster, and I guess I'll go ahead and go with 4.4.
Although Red Hat's recent shift of CentOS 8 to the Stream model, as well as the announcement that RHV is going away makes me nervous. I really don't see any other virtualization software doing quite the same stuff as oVirt at the moment.
One of my questions is around the back end out-of-band network for data replication.
What happens if all 3 servers are healthy and the normal network is fine for serving traffic to the VM consumers, but the switching network for data replication goes down? Is it possible to configure oVirt to "fail over" to the front-end network?I'm also wondering if its possible to do away with a switch all together, and just link the physical hosts together directly (like a cross-over cable) for the data replication.
I'm also wondering what would happen in the following scenario:
- All 3 servers are healthy
- The out-of-band data replication network is healthy
- 1 or 2 of the servers suddenly lost network connectivity on the front-end network
What then? Would everything just keep working, and network traffic be forced to go out the healthy interface(s) on the remaining hosts?
Sent with ProtonMail Secure Email.
4 years
Cloud-init confusion
by Shantur Rathore
Hi Users,
I am new to oVirt and having some confusion with cloud-init.
I have a VM which supports Cloud Init and when I enable Cloud Init under
Initial boot settings of VM, the cloud-init config disk isn't attached to
the VM.
If I use Run-Once and enable Cloud-Init the config disk is attached to the
VM and works as expected.
Is cloud-init only supposed to work with Run-Once or am I missing something?
Thanks in advance.
Shantur
4 years
Template backing disks on different storage
by Shantur Rathore
Hi Users,
I am evaluating oVirt coming from a VMware background.
I have a test cluster with 6 machines running oVirt Node providing
compute + gluster services.
There are 2 storage domains
1. Gluster backed
2. iSCSI SAN
I use iSCSI SAN for the templates (backing disks) and would like to use
Gluster storage domain for template instances.
In VMware I could create a linked clone on one storage linked to a template
on another.
Is this possible with oVirt? via GUI or API?
Thanks in advance,
Shantur
4 years
node IPs can't be pingued on a fresh 4.4.4 installation
by Nathanaël Blanchet
Hello
On a fresh installation of a 4.4.4 node, I configured network so as to
get dhcp delivered IP for gluster and migration services. After IPs have
been delivered, they can't be pingued and vms migrations fail.
All work as expected once I reactivate each bridge with nmcli/nmtui...
This bug is present on each new node installation.
--
Nathanaël Blanchet
Supervision réseau
SIRE
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5
Tél. 33 (0)4 67 54 84 55
Fax 33 (0)4 67 54 84 14
blanchet(a)abes.fr
4 years
Right Glusterfs Config on HA Server
by Marcel d'Heureuse
Hi Guys or Girls,
I have a problem in a productive system. We use three Servers which have
a Raid 5 with four Harddrives SATA 8 TB 7200 rpms. The Systems are
interconnected for Glusterfs with 1 GB/s.
The Network Load between all three servers are between 5 and 8 % of the
network. The Glusterfs has a separate VLAN which complete isolated from
other networks.
Find attached a small drawing.
On that server we rum as HA and there are 8 VMs installed. Most of the
VMs working well but if there is high load on the system some of the VM
can't write the Data in a "good" speed on the disks. The Glusterfs is
configured as Replica on the Raid5 disk which is as Jbob in Glusterfs
defined.
If i take a look on the iostats i can see that the Harddisk have round
50 open connections some of the glusterfs and a lot of the vms. In side
the VMs i can see long iowaiting times from 300 up to 600 ms. With is on
case of a Posgresql database very long.
Should we configure the Raid5 controller out that the controller only
should 4 disks with 8 TB and Glusterfs should connect directly to the
/dev/sd[b-e]?
What do you think? Will SSD HDD will help on performance but the same
config with Raid5 and Jbod could be the bigger problem.
Thanks
Marcel
4 years
Newly uploaded disk stuck in 'Finalizing Failure' status
by n.bouwhuis＠speakup.nl
Hey there,
A few days ago, I tried to create a new template using one of the Ansible playbooks I frequently use. This time, the upload of the disk failed, possibly due to an unrelated error.
The disk is now stuck in Ovirt. It has the status 'Finalizing Failure' and it's locked, thus I cannot remove it. Trying to unlock the disk with the 'unlock_entity.sh' helper utility does not help. The utility says it has completed it task successfully but the disk remains in this state.
I decided to check the logs, and see that `engine.log` gets spammed with these lines every 10 seconds
```
2021-03-03 09:05:38,335Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-45) [redacted] Failed to stop image transfer session. Ticket does not exist for image '[disk uuid]'
```
How can I force Ovirt to remove this disk?
I presume the upload failure itself is a different error, possibly related to our setup. Hence this is not the topic of this thread. I just want to know how I can remove these disks.
Thanks,
Nick
4 years
SHE upgrade from 4.3 to 4.4 doubts
by Gianluca Cecchi
Hello,
I would like to get some clarifications about the SHE environment upgrade
from 4.3.10 to 4.4.4.
Example scenario two hosts, both configured as HE hosts.
Interested both in iSCSI environment and in FC environment.
Going through here:
https://www.ovirt.org/documentation/upgrade_guide/#Upgrading_the_Manager_...
I have some doubts:
1) general one
between step 5 and 6 in section 4.4, actually the current 4.3 engine VM is
still running its OS (even if with ovirt-engine service stopped) and the VM
is running on the host you are going to scratch. So I think before
scratching the host and reinstalling it (step 6), it should be clarified
some step like
- shutdown engine VM
- shutdown host (is it ok in clean way or do I have to simply power off
it....?)
- install from scratch ovirt-ng node 4.4.4 on host
2) general one
why in external engine upgrade guide it is not specified to stop
ovirt-engine service before executing the backup, while in SHE it is
specified?
See steps on section 2.3 here:
https://www.ovirt.org/documentation/upgrade_guide/#Upgrading_from_4-3
expected different behaviour? Or any change needed in the doc?
3) SHE when iSCSI and FC
It is clear that one new domain is to be provided, intended for the new
engine 4.4 VM
It is clear that during install of the new host in 4.4 it is safe to have
shared LUNs masked / not accessible so that you don't risk to corrupt
current 4.3 data
It is not clear in my opinion what to do before running the step 8 of 4.4
section:
hosted-engine --deploy --restore-from-file=backup.bck
a) iSCSI
in general the configured iSCSI sessions are activated only when the host
is being activated. So for example when you put host into maintenance, they
are logged off.
In this stage of upgrade what is expected I execute before running the
hosted-engine --deploy command?
Will the command log in into iSCSI sessions during itsmexecution (because
of the restore) or do I have to create / log in the connections before
running it?
b) FC
what should I have on host before running the command? Only the current
LUns (+ the new one intended for new hosted engine VM) visible, so that I
see all of them as output of "multipath -l" command and then the
hosted-engine --deploy command would mount the related filesystems and so
on..?
Possibly it would be nice to have an example output of the
hosted-engine --deploy --restore-from-file=backup.bck
command inside the doc page.... to see its expected workflow
Thanks for reading,
Gianluca
4 years
Upgrade from 4.3.5 to 4.3.10 HE Host issue
by Vrgotic, Marko
Hi oVirt,
Yesterday I upgraded one of my oVIrt platform from 4.3.5 to 4.3.10. Platform has 10Hosts with 3Hosts as Hosts for HA.
Before upgrade, all three Hosts were acknowledged Hosts for HE (little happy crown).
Pre upgrade:
Host1 – HE available host
Host2 – HE current host
Host3 – HE available host
Upgraded first HE to 4.3.10, than started with HE Hosts – post upgrade of Host2, the crown disappeared and did not show up. This happens sometimes, and I usually reinstall the host, than execute HE UNDEPLOY and DEPLOY, which fixes the state.
However, this time, after such actions, the server is still not an acknowledged HE host – no crown. However, if I try to remove it, oVIrt says that looks like HE is deploy on it and need to be removed, but command “hosted-engine –vm-status” says you must deploy first.
I removed the Host, reinstalled the OS on it, deployed it again as new, synced all networks, than deploy once more with HE Deploy, still same issue.
Any suggestions?
-----
kind regards/met vriendelijke groeten
Marko Vrgotic
Sr. System Engineer @ System Administration
ActiveVideo
o: +31 (35) 6774131
m: +31 (65) 5734174
e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com>
w: www.activevideo.com<http://www.activevideo.com>
ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.
4 years
Multipath flapping with SAS via FCP
by Benoit Chatelain
Hi,
I have some troubles with multipath.
When I add SAS disk over FCP as Storage Domain via oVirt WebUI,
The first link as active, but the second is stuck as failed.
Volum disk is provided from Dell Compellent via FCP, and disk is transported in SAS.
multipath is flapping in all hypervisor from the same domain disk:
[root@isildur-adm ~]# tail -f /var/log/messages
Feb 25 11:48:21 isildur-adm kernel: device-mapper: multipath: 253:3: Failing path 8:32.
Feb 25 11:48:24 isildur-adm multipathd[659460]: 36000d31003d5c2000000000000000010: sdc - tur checker reports path is up
Feb 25 11:48:24 isildur-adm multipathd[659460]: 8:32: reinstated
Feb 25 11:48:24 isildur-adm multipathd[659460]: 36000d31003d5c2000000000000000010: remaining active paths: 2
Feb 25 11:48:24 isildur-adm kernel: device-mapper: multipath: 253:3: Reinstating path 8:32.
Feb 25 11:48:24 isildur-adm kernel: sd 1:0:1:2: alua: port group f01c state S non-preferred supports toluSNA
Feb 25 11:48:24 isildur-adm kernel: sd 1:0:1:2: alua: port group f01c state S non-preferred supports toluSNA
Feb 25 11:48:24 isildur-adm kernel: device-mapper: multipath: 253:3: Failing path 8:32.
Feb 25 11:48:25 isildur-adm multipathd[659460]: sdc: mark as failed
Feb 25 11:48:25 isildur-adm multipathd[659460]: 36000d31003d5c2000000000000000010: remaining active paths: 1
---
[root@isildur-adm ~]# multipath -ll
36000d31003d5c2000000000000000010 dm-3 COMPELNT,Compellent Vol
size=1.5T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=25 status=active
|- 1:0:0:2 sdb 8:16 active ready running
`- 1:0:1:2 sdc 8:32 failed ready running
---
VDSM generate multipath.conf like this ( I have remove commented lines for read confort ) :
[root@isildur-adm ~]# cat /etc/multipath.conf
# VDSM REVISION 2.0
# This file is managed by vdsm.
defaults {
polling_interval 5
no_path_retry 16
user_friendly_names no
flush_on_last_del yes
fast_io_fail_tmo 5
dev_loss_tmo 30
max_fds 4096
}
blacklist {
protocol "(scsi:adt|scsi:sbp)"
}
no_path_retry 16
}
Have you some idea why this link is flapping on my two hypervisor?
Thanks a lot in advance.
- Benoit Chatelain
4 years
