- Users - oVirt List Archives

Re: Win Server 2k19 and BlueIris
by Robert Webb 08 Feb '20

08 Feb '20

Update: So after some digging, it seems this has been reported as an issue many times. As a workaround, I added the an entry to the kvm.conf in modprobe.d directory of ignore_msrs=1 and that allowed Blue Iris to boot. Would be nice to have an actual fix for this instead of a workaround. Would really like to know why kvm is thing that ia32_debugctl is not a valid cpu option. From: Robert Webb Sent: Saturday, February 8, 2020 10:33 AM To: users <users(a)ovirt.org> Subject: Win Server 2k19 and BlueIris So I have been running Windows Server 2019 on Proxmox VE with Blue Iris NVR. Tried to migrate the qcow disk over to oVirt, and like other Win2k19 servers, they all give a BSOD with a KMODE error. Last night I installed 2k19 from scratch, installed the latest drivers I could find, oVirt-toolsSetup-4.3-3.el7, and proceeded to load up Blue Iris. As soon as I go to start Blue Iris, the VM hangs, then gives a BSOD with SYSTEM SERVICE EXCEPTION. Looking at the debug log on the ovirt node the VM is running, there one entry that gets created on each crash with, "kvm [4723]: vcpu1 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop". Running on oVirt node 4.3.8, cluster cpu is Intel Westmere IBRS SSBD Family, pc-i440fx-rhel7.6.0, and the physical cpu is 24x Intel(R) Xeon(R) CPU X5670 @ 2.93GHz. Any ideas?

1 0

Win Server 2k19 and BlueIris
by Robert Webb 08 Feb '20

08 Feb '20

So I have been running Windows Server 2019 on Proxmox VE with Blue Iris NVR. Tried to migrate the qcow disk over to oVirt, and like other Win2k19 servers, they all give a BSOD with a KMODE error. Last night I installed 2k19 from scratch, installed the latest drivers I could find, oVirt-toolsSetup-4.3-3.el7, and proceeded to load up Blue Iris. As soon as I go to start Blue Iris, the VM hangs, then gives a BSOD with SYSTEM SERVICE EXCEPTION. Looking at the debug log on the ovirt node the VM is running, there one entry that gets created on each crash with, "kvm [4723]: vcpu1 kvm_set_msr_common: MSR_IA32_DEBUGCTLMSR 0x1, nop". Running on oVirt node 4.3.8, cluster cpu is Intel Westmere IBRS SSBD Family, pc-i440fx-rhel7.6.0, and the physical cpu is 24x Intel(R) Xeon(R) CPU X5670 @ 2.93GHz. Any ideas?

1 0

[ANN] oVirt 4.3.9 First Release Candidate is now available for testing
by Sandro Bonazzola 08 Feb '20

08 Feb '20

The oVirt Project is pleased to announce the availability of the oVirt 4.3.9 First Release Candidate for testing, as of January 30th, 2020. This update is a release candidate of the nineth in a series of stabilization updates to the 4.3 series. This is pre-release software. This pre-release should not to be used in production. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) * oVirt Node 4.3 (available for x86_64 only) has been built consuming CentOS 7.7 Release See the release notes [1] for known issues, new features and bugs fixed. Notes: - oVirt Appliance is already available - oVirt Node will be available soon Additional Resources: * Read more about the oVirt 4.3.9 release highlights: http://www.ovirt.org/release/4.3.9/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.9/ [2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

2 1

VM not getting up
by Crazy Ayansh 08 Feb '20

08 Feb '20

[image: image.png] disk is already attached, All vm on the same server giving same error. Help please.

4 8

3 Node Hyperconverged Install fails with - Error: the target storage domain contains only 46.0GiB of available space while a minimum of 61.0Gi
by Dirk Rydvan 07 Feb '20

07 Feb '20

Hello ovirts, I try to create a 3 Node hyperconverged cluster with the cockpit GUI, but ist fails at last step "5" in Hosted engine setup [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Error: the target storage domain contains only 46.0GiB of available space while a minimum of 61.0GiB is required If you wish to use the current target storage domain by extending it, make sure it contains nothing before adding it."} I checked the volumes at all 3 nodes - only node1 shows a mount - node2 and node3 not [root@node1 ~]# df -h Filesystem Size Used Avail Use% Mounted on <snip> node43.infra.qivicon.de:/engine 52G 3.0G 47G 7% /rhev/data-center/mnt/glusterSD/node43.infra.qivicon.de:_engine /dev/mapper/gluster_vg_nvme0n1-gluster_lv_data 500G 34M 500G 1% /gluster_bricks/data /dev/mapper/gluster_vg_nvme0n1-gluster_lv_engine 100G 33M 100G 1% /gluster_bricks/engine /dev/mapper/gluster_vg_nvme0n1-gluster_lv_vmstore 500G 33M 500G 1% /gluster_bricks/vmstore I created the gluster before with: Hosts: node1 node2 node3 Default Gluster values plus cache /gluster_bricks/engine 100GB /dev/nvme0n1 (Arbiter unchecked) /gluster_bricks/data 500GB /dev/nvme0n1 (Arbiter unchecked) /gluster_bricks/vmstore 500GB /dev/nvme0n1 (Arbiter unchecked) cache /dev/nvme1n1 10GB writeback The allredy existing thread does not help: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/W2CM74EU6KPPJ2… Any idea where the problem is? many Thanks in advance

2 1

Re: [Gluster-users] ACL issue v6.6, v6.7, v7.1, v7.2
by Paolo Margara 07 Feb '20

07 Feb '20

Hi, this is interesting, this happen always with gluster 6.6 or only in certain cases? I ask this because I have two ovirt clusters with gluster, both with gluster v6.6, in one case I've upgraded from 6.5 to 6.6 as Strahil, and I haven't hit this bug. When upgrading my clusters I follow exactly the steps reported by Strahil in the bug report, some other things should be different. Greetings, Paolo Il 06/02/20 23:30, Christian Reiss ha scritto: > Hey, > > I hit this bug, too. With disastrous results. > I second this post. > > > On 06.02.2020 19:59, Strahil Nikolov wrote: >> Hello List, >> >> Recently I had upgraded my oVirt + Gluster (v6.5 -> v6.6) and I hit >> an ACL bug , which forced me to upgrade to v7.0 >> >> Another oVirt user also hit the bug when upgrading to v6.7 and he >> had to rebuild his gluster cluster. >> >> Sadly, the fun doesn't stop here. Last week I have tried upgrading >> to v7.2 and again the ACL bug hit me. Downgrading to v7.1 doesn't >> help - so I downgraded to 7.0 and everything is operational. >> >> The bug report for the last issue: >> https://bugzilla.redhat.com/show_bug.cgi?id=1797099 >> >> I have 2 questions : >> 1. What is gluster's ACL evaluating/checking ? Is there an option to >> force gluster not to support ACL at all ? >> >> 2. Are you aware of the issue? That bug was supposed to be fixed a >> long time ago, yet the facts speak different. >> >> Thanks for reading this long post. >> >> Best Regards, >> Strahil Nikolov >> ________ >> >> Community Meeting Calendar: >> >> APAC Schedule - >> Every 2nd and 4th Tuesday at 11:30 AM IST >> Bridge: https://bluejeans.com/441850968 >> >> NA/EMEA Schedule - >> Every 1st and 3rd Tuesday at 01:00 PM EDT >> Bridge: https://bluejeans.com/441850968 >> >> Gluster-users mailing list >> Gluster-users(a)gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users

3 3

OVN issues: no connectivity between VMs on same host
by thkam＠hua.gr 07 Feb '20

07 Feb '20

Hi I am getting some problems setting up an internal networks for our oVirt VMs using the OVN external provider. The ovn-provider was setup during the ovirt engine installation (version 4.3.7.2-1.el7). I then just created a new network with the ovn external provider and tested the connectivity using the corresponding button on the oVirt UI. Two of my VMs are Ubuntu 18.04 servers with ips 10.0.0.101 and 10.0.0.102 with gateways 10.0.0.102 and 10.0.0.101 respectively. So I guess under normal circumstances they should be able to ping each other. I tried disabling the firewalld service on both the host and the ovirtengine but still nothing changed. However I noticed something odd on the firewalld status on the host. With firewalld enabled, I get: [root@ovirt7 ~]# systemctl status firewalld ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled) Active: active (running) since Πεμ 2020-02-06 06:10:38 UTC; 14s ago Docs: man:firewalld(1) Main PID: 4436 (firewalld) Tasks: 2 CGroup: /system.slice/firewalld.service └─4436 /usr/bin/python2 -Es /usr/sbin/firewalld --nofork --nopid Φεβ 06 06:10:39 ovirt7.hua.gr firewalld[4436]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet1 -j libvirt-O-vnet1' failed: Illegal target name 'libvirt-O-vnet1'. Φεβ 06 06:10:39 ovirt7.hua.gr firewalld[4436]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-I-vnet1' failed: Chain 'libvirt-I-vnet1' doesn't exist. Φεβ 06 06:10:39 ovirt7.hua.gr firewalld[4436]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-I-vnet1' failed: Chain 'libvirt-I-vnet1' doesn't exist. Φεβ 06 06:10:39 ovirt7.hua.gr firewalld[4436]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet1' failed: Chain 'libvirt-O-vnet1' doesn't exist. Φεβ 06 06:10:39 ovirt7.hua.gr firewalld[4436]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet1' failed: Chain 'libvirt-O-vnet1' doesn't exist. Φεβ 06 06:10:39 ovirt7.hua.gr firewalld[4436]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -E libvirt-P-vnet1 libvirt-O-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Φεβ 06 06:10:39 ovirt7.hua.gr firewalld[4436]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet1-mac' failed: Chain 'I-vnet1-mac' doesn't exist. Φεβ 06 06:10:39 ovirt7.hua.gr firewalld[4436]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet1-mac' failed: Chain 'I-vnet1-mac' doesn't exist. Φεβ 06 06:10:39 ovirt7.hua.gr firewalld[4436]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet1-arp-mac' failed: Chain 'I-vnet1-arp-mac' doesn't exist. Φεβ 06 06:10:39 ovirt7.hua.gr firewalld[4436]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet1-arp-mac' failed: Chain 'I-vnet1-arp-mac' doesn't exist. On the ovirt engine I have: [root@ovirtengine ~]# ovn-sbctl show Chassis "0e225912-6318-4bc3-9c94-8f2ab937876d" hostname: "ovirt3.hua.gr" Encap geneve ip: "10.100.59.53" options: {csum="true"} Chassis "47ff3058-effb-43c2-b9e0-9eaf6c72b1c2" hostname: "ovirt5.hua.gr" Encap geneve ip: "10.100.59.51" options: {csum="true"} Chassis "1fcca34f-6017-4882-8c13-4835dad03387" hostname: localhost Encap geneve ip: "10.100.59.55" options: {csum="true"} Chassis "25d35968-6c3c-4040-ab85-d881d3d524e4" hostname: "ovirt4.hua.gr" Encap geneve ip: "10.100.59.52" options: {csum="true"} Chassis "32697bcc-cc6a-4a59-8424-887d20df2d10" hostname: "ovirt7.hua.gr" Encap geneve ip: "10.100.59.49" options: {csum="true"} Port_Binding "ff31a88f-23f0-48fe-a657-3cd24d51f69e" Port_Binding "01da6ee3-abff-4423-954a-c4abf350e390" Port_Binding "caa870e1-8b6c-48bd-ac61-bfeda8befd10" Chassis "f60100e6-0ee5-4472-8095-cc48b5160f50" hostname: "ovirt6.hua.gr" Encap geneve ip: "10.100.59.50" options: {csum="true"} Chassis "fa7e6cbe-fa7f-46bc-9760-f581725f60a8" hostname: "ovirt2.hua.gr" Encap geneve ip: "10.100.59.54" options: {csum="true"} Chassis "58a9412f-bfad-4c98-9882-d7d006588e0b" hostname: "ovirt9.hua.gr" Encap geneve ip: "10.100.59.47" options: {csum="true"} Chassis "09dc1148-8ce5-425d-8f93-8dbf43fd7828" hostname: "ovirt8.hua.gr" Encap geneve ip: "10.100.59.48" options: {csum="true"} so I guess tunneling is correctly setup right. On the host I am seeing: [root@ovirt7 ~]# ovs-vsctl show 886fe6e5-13ea-4889-ba35-1ac0a422ca23 Bridge br-int fail_mode: secure Port "ovn-1fcca3-0" Interface "ovn-1fcca3-0" type: geneve options: {csum="true", key=flow, remote_ip="10.100.59.55"} Port "ovn-0e2259-0" Interface "ovn-0e2259-0" type: geneve options: {csum="true", key=flow, remote_ip="10.100.59.53"} Port "ovn-fa7e6c-0" Interface "ovn-fa7e6c-0" type: geneve options: {csum="true", key=flow, remote_ip="10.100.59.54"} Port "vnet3" Interface "vnet3" Port "vnet0" Interface "vnet0" Port "ovn-25d359-0" Interface "ovn-25d359-0" type: geneve options: {csum="true", key=flow, remote_ip="10.100.59.52"} Port br-int Interface br-int type: internal Port "ovn-f60100-0" Interface "ovn-f60100-0" type: geneve options: {csum="true", key=flow, remote_ip="10.100.59.50"} Port "ovn-47ff30-0" Interface "ovn-47ff30-0" type: geneve options: {csum="true", key=flow, remote_ip="10.100.59.51"} Port "ovn-58a941-0" Interface "ovn-58a941-0" type: geneve options: {csum="true", key=flow, remote_ip="10.100.59.47"} Port "vnet2" Interface "vnet2" Port "ovn-09dc11-0" Interface "ovn-09dc11-0" type: geneve options: {csum="true", key=flow, remote_ip="10.100.59.48"} ovs_version: "2.11.0" and: [root@ovirt7 ~]# ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master trunk state UP mode DEFAULT group default qlen 1000 link/ether 00:17:a4:77:00:30 brd ff:ff:ff:ff:ff:ff 3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 00:17:a4:77:00:32 brd ff:ff:ff:ff:ff:ff 19: eno1.59@eno1: <BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovirtmgmt state UP mode DEFAULT group default qlen 1000 link/ether 00:17:a4:77:00:30 brd ff:ff:ff:ff:ff:ff 20: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 2a:92:56:8b:36:b8 brd ff:ff:ff:ff:ff:ff 21: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether c6:70:c8:34:86:85 brd ff:ff:ff:ff:ff:ff 23: br-int: <BROADCAST,MULTICAST> mtu 1442 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether da:49:e1:b0:b2:4e brd ff:ff:ff:ff:ff:ff 25: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:17:a4:77:00:30 brd ff:ff:ff:ff:ff:ff 26: eno1.60@eno1: <BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue master vlan60 state UP mode DEFAULT group default qlen 1000 link/ether 00:17:a4:77:00:30 brd ff:ff:ff:ff:ff:ff 27: vlan60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:17:a4:77:00:30 brd ff:ff:ff:ff:ff:ff 28: eno1.61@eno1: <BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue master vlan61 state UP mode DEFAULT group default qlen 1000 link/ether 00:17:a4:77:00:30 brd ff:ff:ff:ff:ff:ff 29: vlan61: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:17:a4:77:00:30 brd ff:ff:ff:ff:ff:ff 30: trunk: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:17:a4:77:00:30 brd ff:ff:ff:ff:ff:ff 39: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc mq master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000 link/ether fe:6f:2b:f7:00:17 brd ff:ff:ff:ff:ff:ff 40: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc mq master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000 link/ether fe:6f:2b:f7:00:18 brd ff:ff:ff:ff:ff:ff 41: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovirtmgmt state UNKNOWN mode DEFAULT group default qlen 1000 link/ether fe:6f:2b:f7:00:0f brd ff:ff:ff:ff:ff:ff 42: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc mq master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000 link/ether fe:6f:2b:f7:00:16 brd ff:ff:ff:ff:ff:ff 44: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000 link/ether 06:6c:84:05:bf:6b brd ff:ff:ff:ff:ff:ff I have not found anything useful in the log files. Does anyone have any thoughts? Thanks! Thomas

1 1

Websocket-proxy not working after upgrade to 4.3
by nicolas＠devels.es 07 Feb '20

07 Feb '20

Hi, We recently upgraded to 4.3.8 and everything is working fine but the VNC Console (Browser). Once I click on "VNC Console (Browser)" on any machine from the VM Portal, I get a message like this: Disconnected from Console Cannot connect to websocket proxy server. Please check your websocket proxy certificate or ask your administrator for help. For further information please refer to the console manual. Press the 'Connect' button to reconnect the console. Thing is that everything seems ok to me, and I cannot find further error log about it. /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf content is: PROXY_PORT=6100 SSL_CERTIFICATE=/etc/ssl/certs/fqdn.combined.cert SSL_KEY=/etc/ssl/private/fqdn.key FORCE_DATA_VERIFICATION=False CERT_FOR_DATA_VERIFICATION=/etc/pki/ovirt-engine/certs/engine.cer SSL_ONLY=True On a "status" command on ovirt-websocket-proxy I just see: feb 05 12:23:22 fqdn systemd[1]: Starting oVirt Engine websockets proxy... feb 05 12:23:22 fqdn systemd[1]: Started oVirt Engine websockets proxy. feb 05 12:23:22 fqdn ovirt-websocket-proxy.py[3314]: ovirt-websocket-proxy[3314] INFO daemonContext:434 Using the following ciphers: HIGH:!aNULL feb 05 12:23:22 fqdn ovirt-websocket-proxy.py[3314]: ovirt-websocket-proxy[3314] INFO daemonContext:438 Minimum SSL version requested: TLSv1.2 feb 05 12:23:22 fqdn ovirt-websocket-proxy.py[3314]: ovirt-websocket-proxy[3314] INFO msg:887 WebSocket server settings: feb 05 12:23:22 fqdn ovirt-websocket-proxy.py[3314]: ovirt-websocket-proxy[3314] INFO msg:887 - Listen on *:6100 feb 05 12:23:22 fqdn ovirt-websocket-proxy.py[3314]: ovirt-websocket-proxy[3314] INFO msg:887 - Flash security policy server feb 05 12:23:22 fqdn ovirt-websocket-proxy.py[3314]: ovirt-websocket-proxy[3314] INFO msg:887 - SSL/TLS support feb 05 12:23:22 fqdn ovirt-websocket-proxy.py[3314]: ovirt-websocket-proxy[3314] INFO msg:887 - Deny non-SSL/TLS connections feb 05 12:23:22 fqdn ovirt-websocket-proxy.py[3314]: ovirt-websocket-proxy[3314] INFO msg:887 - proxying from *:6100 to targets generated by str On the ovirt-engine.log, I just see this information: 2020-02-05 12:29:10,085Z INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (default task-110) [68218d5b] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 5bf9a0bb-da18-4d07-87da-759c0b045e28 Type: VMAction group CONNECT_TO_VM with role type USER 2020-02-05 12:29:10,095Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-110) [68218d5b] START, SetVmTicketVDSCommand(HostName = kvmr01.fqdn, SetVmTicketVDSCommandParameters:{hostId='1828d0dc-e953-4d6a-8a95-528bb7aa849a', vmId='5bf9a0bb-da18-4d07-87da-759c0b045e28', protocol='VNC', ticket='oVoKEtgmDKnM', validTime='120', userName='user', userId='66a7a37f-d804-4192-9734-93f01a95dd98', disconnectAction='LOCK_SCREEN'}), log id: 596fbfb9 2020-02-05 12:29:10,167Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-110) [68218d5b] FINISH, SetVmTicketVDSCommand, return: , log id: 596fbfb9 2020-02-05 12:29:10,195Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-110) [68218d5b] EVENT_ID: VM_SET_TICKET(164), User user@domain-authz initiated console session for VM user.fqdn 2020-02-05 12:29:10,308Z INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (default task-110) [097f6518-5f87-4947-aee6-e76c9b740bcd] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 5bf9a0bb-da18-4d07-87da-759c0b045e28 Type: VMAction group CONNECT_TO_VM with role type USER 2020-02-05 12:29:10,316Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-110) [097f6518-5f87-4947-aee6-e76c9b740bcd] START, SetVmTicketVDSCommand(HostName = kvmr01.fqdn, SetVmTicketVDSCommandParameters:{hostId='1828d0dc-e953-4d6a-8a95-528bb7aa849a', vmId='5bf9a0bb-da18-4d07-87da-759c0b045e28', protocol='VNC', ticket='A7PQWaXupvbZ', validTime='7200', userName='user', userId='66a7a37f-d804-4192-9734-93f01a95dd98', disconnectAction='LOCK_SCREEN'}), log id: 71e5165c 2020-02-05 12:29:10,387Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-110) [097f6518-5f87-4947-aee6-e76c9b740bcd] FINISH, SetVmTicketVDSCommand, return: , log id: 71e5165c 2020-02-05 12:29:10,408Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-110) [097f6518-5f87-4947-aee6-e76c9b740bcd] EVENT_ID: VM_SET_TICKET(164), User user@domain-authz initiated console session for VM user.fqdn Please, any tip on how to debug this? I cannot seem to find the reason for this. Thanks.

4 5

Backup Solution
by Christian Reiss 07 Feb '20

07 Feb '20

Hey folks, Running a 3-way HCI (again (sigh)) on gluster. Now the _inside_ of the vms is backup'ed seperatly using bareos on an hourly basis, so files are present with worst case 59 minutes data loss. Now, on the outside I thought of doing gluster snapshots and then syncing those .snap dirs away to a remote 10gig connected machine on a weekly-or-so basis. As those contents of the snaps are the oVirt images (entire DC) I could re-setup gluster and copy those files back into gluster and be done with it. Now some questions, if I may: - If the hosts remain intact but gluster dies, I simply setup Gluster, stop the ovirt engine (seperate standalone hardware) copy everything back and start ovirt engine again. All disks are accessible again (tested). The bricks are marked as down (new bricks, same name). There is a "reset brick" button that made the bricks come back online again. What _exactly_ does it do? Does it reset the brick info in oVirt or copy all the data over from another node and really, really reset the brick? - If the hosts remain intact, but the engine dies: Can I re-attach the engine the the running cluster? - If hosts and engine dies and everything needs to be re-setup would it be possible to do the setup wizard(s) again up to a running point then copy the disk images to the new gluster-dc-data-dir? Would oVirt rescan the dir for newly found vms? - If _one_ host dies, but 2 and the engine remain online: Whats the oVirt way of resetting up the failed one? Reinstalling the node and then what? From all the cases above this is the most likely one. Having had to reinstall the entire Cluster three times already scares me. Always gluster related. Again thank you community for your great efforts! -- with kind regards, mit freundlichen Gruessen, Christian Reiss

4 5

Emergency :/ No VMs starting
by Christian Reiss 06 Feb '20

06 Feb '20

Hey folks, oh Jesus. 3-Way HCI. Gluster w/o any issues: [root@node01:/var/log/glusterfs] # gluster vol info ssd_storage Volume Name: ssd_storage Type: Replicate Volume ID: d84ec99a-5db9-49c6-aab4-c7481a1dc57b Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: node01.company.com:/gluster_bricks/ssd_storage/ssd_storage Brick2: node02.company.com:/gluster_bricks/ssd_storage/ssd_storage Brick3: node03.company.com:/gluster_bricks/ssd_storage/ssd_storage Options Reconfigured: performance.client-io-threads: on nfs.disable: on transport.address-family: inet performance.strict-o-direct: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.low-prio-threads: 32 network.remote-dio: off cluster.eager-lock: enable cluster.quorum-type: auto cluster.server-quorum-type: server cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 10000 features.shard: on user.cifs: off cluster.choose-local: off client.event-threads: 4 server.event-threads: 4 network.ping-timeout: 30 storage.owner-uid: 36 storage.owner-gid: 36 cluster.granular-entry-heal: enab [root@node01:/var/log/glusterfs] # gluster vol status ssd_storage Status of volume: ssd_storage Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick node01.company.com:/gluster_br icks/ssd_storage/ssd_storage 49152 0 Y 63488 Brick node02.company.com:/gluster_br icks/ssd_storage/ssd_storage 49152 0 Y 18860 Brick node03.company.com:/gluster_br icks/ssd_storage/ssd_storage 49152 0 Y 15262 Self-heal Daemon on localhost N/A N/A Y 63511 Self-heal Daemon on node03.dc-dus.dalason.n et N/A N/A Y 15285 Self-heal Daemon on 10.100.200.12 N/A N/A Y 18883 Task Status of Volume ssd_storage ------------------------------------------------------------------------------ There are no active volume tasks [root@node01:/var/log/glusterfs] # gluster vol heal ssd_storage info Brick node01.company.com:/gluster_bricks/ssd_storage/ssd_storage Status: Connected Number of entries: 0 Brick node02.company.com:/gluster_bricks/ssd_storage/ssd_storage Status: Connected Number of entries: 0 Brick node03.company.com:/gluster_bricks/ssd_storage/ssd_storage Status: Connected Number of entries: 0 And everything is mounted where its supposed to. But no VMs start due to IO Error. I checked a gluster-based file (CentOS iso) md5 against a local copy, it matches. One VM at one point managed to start, but failed subsequent starts. The data/disks seem okay, /var/log/glusterfs/"rhev-data-center-mnt-glusterSD-node01.company.com:_ssd__storage.log-20200202" has entries like: [2020-02-01 23:15:15.449902] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ssd_storage-client-1: remote operation failed. Path: /.shard/86da0289-f74f-4200-9284-678e7bd76195.1405 (00000000-0000-0000-0000-000000000000) [Permission denied] [2020-02-01 23:15:15.484363] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ssd_storage-client-1: remote operation failed. Path: /.shard/86da0289-f74f-4200-9284-678e7bd76195.1400 (00000000-0000-0000-0000-000000000000) [Permission denied] Before this happened we put one host into maintenance mode, it all started during migration. Any help? We're sweating blood here. -- with kind regards, mit freundlichen Gruessen, Christian Reiss

4 17