Well, now that I’ve gone and read through that bug again in detail, I’m not sure I’ve worked around it after all. I do seem to recall additional discussion on the original bug for HA engine ligfapi a mention that RR-DNS would work to resolve the issue, but can’t remember the bug ID at the moment. I will test thoroughly the next time I update my glusterfs servers. But I firmly believe that I’ve never encountered that issue in over 3 years of running gluster with libgfapi enabled.. I use round robin DNS, and in theory, QEMU retries until it gets a working server. I also have said DNS setup in host files on all my hosts and gluster servers, having discovered the hard way that when your DNS server runs on an ovirt managed VM, you have a bootstrap problem when thing break badly :) Somewhere around gluster 3.12, I added backup servers to the mount options for my gluster storage volumes as well, and have’t had any issues with that. And to be frank, the significant performance bonus from libgfapi is still absolutely worth it to me even if it means automatic HA won’t work if one particular server is down. I can always intervene in the DNS on my hosts if I have to, and it just hasn’t come up yet. -Darrell
On Feb 13, 2020, at 5:19 PM, Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
On February 13, 2020 11:51:41 PM GMT+02:00, Stephen Panicho <s.panicho@gmail.com <mailto:s.panicho@gmail.com>> wrote:
Darrell, would you care to elaborate on your HA workaround?
As far as I understand, only the primary Gluster host is visible to libvirt when using gfapi, so if that host goes down, all VMs break. I imagine you're using a round-robin DNS entry for the primary Gluster host, but I'd like to be sure.
On Wed, Feb 12, 2020 at 11:01 AM Darrell Budic <budic@onholyground.com> wrote:
Yes. I’m using libgfapi access on gluster 6.7 with overt 4.3.8 just fine, but I don’t use snapshots. You can work around the HA issue with DNS and backup server entries on the storage domain as well. Worth it to me for the performance, YMMV.
On Feb 12, 2020, at 8:04 AM, Jayme <jaymef@gmail.com> wrote:
From my understanding it's not a default option but many users are still using libgfapi successfully. I'm not sure about its status in the latest 4.3.8 release but I know it is/was working for people in previous versions. The libgfapi bugs affect HA and snapshots (on 3 way replica HCI) but it should still be working otherwise, unless like I said something changed in more recent releases of oVirt.
On Wed, Feb 12, 2020 at 9:43 AM Guillaume Pavese < guillaume.pavese@interactiv-group.com> wrote:
Libgfapi is not supported because of an old bug in qemu. That qemu bug is slowly getting fixed, but the bugs about Libgfapi support in ovirt have since been closed as WONTFIX and DEFERRED
See : https://bugzilla.redhat.com/show_bug.cgi?id=1465810 https://bugzilla.redhat.com/show_bug.cgi?id=1484660 <https://bugzilla.redhat.com/show_bug.cgi?id=1484227 <https://bugzilla.redhat.com/show_bug.cgi?id=1484227>> : "No plans to enable libgfapi in RHHI-V for now. Closing this bug" https://bugzilla.redhat.com/show_bug.cgi?id=1484227 <https://bugzilla.redhat.com/show_bug.cgi?id=1484227> : "No plans to enable libgfapi in RHHI-V for now. Closing this bug" https://bugzilla.redhat.com/show_bug.cgi?id=1633642 <https://bugzilla.redhat.com/show_bug.cgi?id=1633642> : "Closing this as no action taken from long back.Please reopen if required."
Would be nice if someone could reopen the closed bugs so this feature doesn't get forgotten
Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group
On Tue, Feb 11, 2020 at 9:58 AM Stephen Panicho <s.panicho@gmail.com <mailto:s.panicho@gmail.com>> wrote:
I used the cockpit-based hc setup and "option rpc-auth-allow-insecure" is absent from /etc/glusterfs/glusterd.vol.
I'm going to redo the cluster this week and report back. Thanks for the tip!
On Mon, Feb 10, 2020 at 6:01 PM Darrell Budic <budic@onholyground.com <mailto:budic@onholyground.com>> wrote:
The hosts will still mount the volume via FUSE, but you might double check you set the storage up as Gluster and not NFS.
Then gluster used to need some config in glusterd.vol to set
option rpc-auth-allow-insecure on
I’m not sure if that got added to a hyper converged setup or not, but I’d check it.
On Feb 10, 2020, at 4:41 PM, Stephen Panicho <s.panicho@gmail.com <mailto:s.panicho@gmail.com>> wrote:
No, this was a relatively new cluster-- only a couple days old. Just a handful of VMs including the engine.
On Mon, Feb 10, 2020 at 5:26 PM Jayme <jaymef@gmail.com <mailto:jaymef@gmail.com>> wrote:
> Curious do the vms have active snapshots? > > On Mon, Feb 10, 2020 at 5:59 PM <s.panicho@gmail.com <mailto:s.panicho@gmail.com>> wrote: > >> Hello, all. I have a 3-node Hyperconverged oVirt 4.3.8 cluster >> running on CentOS 7.7 hosts. I was investigating poor Gluster performance >> and heard about libgfapi, so I thought I'd give it a shot. Looking through >> the documentation, followed by lots of threads and BZ reports, I've done >> the following to enable it: >> >> First, I shut down all VMs except the engine. Then... >> >> On the hosts: >> 1. setsebool -P virt_use_glusterfs on >> 2. dynamic_ownership=0 in /etc/libvirt/qemu.conf >> >> On the engine VM: >> 1. engine-config -s LibgfApiSupported=true --cver=4.3 >> 2. systemctl restart ovirt-engine >> >> VMs now fail to launch. Am I doing this correctly? I should also note >> that the hosts still have the Gluster domain mounted via FUSE. >> >> Here's a relevant bit from engine.log: >> >> 2020-02-06T16:38:32.573511Z qemu-kvm: -drive file=gluster:// >> node1.fs.trashnet.xyz:24007/vmstore/781717e5-1cff-43a1-b586-9941503544e8/images/a1d56b14-6d72-4f46-a0aa-eb0870c36bc4/a2314816-7970-49ce-a80c-ab0d1cf17c78,file.debug=4,format=qcow2,if=none,id=drive-ua <http://node1.fs.trashnet.xyz:24007/vmstore/781717e5-1cff-43a1-b586-9941503544e8/images/a1d56b14-6d72-4f46-a0aa-eb0870c36bc4/a2314816-7970-49ce-a80c-ab0d1cf17c78,file.debug=4,format=qcow2,if=none,id=drive-ua>-a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,serial=a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,werror=stop,rerror=stop,cache=none,discard=unmap,aio=native: >> Could not read qcow2 header: Invalid argument. >> >> The full engine.log from one of the attempts: >> >> 2020-02-06 16:38:24,909Z INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (ForkJoinPool-1-worker-12) [] add VM >> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'(yumcache) to rerun treatment >> 2020-02-06 16:38:25,010Z ERROR >> [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] >> (ForkJoinPool-1-worker-12) [] Rerun VM >> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'. Called from VDS ' >> node2.ovirt.trashnet.xyz <http://node2.ovirt.trashnet.xyz/>' >> 2020-02-06 16:38:25,091Z WARN >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (EE-ManagedThreadFactory-engine-Thread-216) [] EVENT_ID: >> USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM yumcache on Host >> node2.ovirt.trashnet.xyz <http://node2.ovirt.trashnet.xyz/>. >> 2020-02-06 16:38:25,166Z INFO >> [org.ovirt.engine.core.bll.RunVmCommand] >> (EE-ManagedThreadFactory-engine-Thread-216) [] Lock Acquired to object >> 'EngineLock:{exclusiveLocks='[df9dbac4-35c0-40ee-acd4-a1cfc959aa8b=VM]', >> sharedLocks=''}' >> 2020-02-06 16:38:25,179Z INFO >> [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-216) [] START, >> IsVmDuringInitiatingVDSCommand( >> IsVmDuringInitiatingVDSCommandParameters:{vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'}), >> log id: 2107f52a >> 2020-02-06 16:38:25,181Z INFO >> [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-216) [] FINISH, >> IsVmDuringInitiatingVDSCommand, return: false, log id: 2107f52a >> 2020-02-06 16:38:25,298Z INFO >> [org.ovirt.engine.core.bll.RunVmCommand] >> (EE-ManagedThreadFactory-engine-Thread-216) [] Running command: >> RunVmCommand internal: false. Entities affected : ID: >> df9dbac4-35c0-40ee-acd4-a1cfc959aa8b Type: VMAction group RUN_VM with role >> type USER >> 2020-02-06 16:38:25,313Z INFO >> [org.ovirt.engine.core.bll.utils.EmulatedMachineUtils] >> (EE-ManagedThreadFactory-engine-Thread-216) [] Emulated machine >> 'pc-q35-rhel7.6.0' which is different than that of the cluster is set for >> 'yumcache'(df9dbac4-35c0-40ee-acd4-a1cfc959aa8b) >> 2020-02-06 16:38:25,382Z INFO >> [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-216) [] START, >> UpdateVmDynamicDataVDSCommand( >> UpdateVmDynamicDataVDSCommandParameters:{hostId='null', >> vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b', >> vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@9774a64'}), >> log id: 4a83911f >> 2020-02-06 16:38:25,417Z INFO >> [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-216) [] FINISH, >> UpdateVmDynamicDataVDSCommand, return: , log id: 4a83911f >> 2020-02-06 16:38:25,418Z INFO >> [org.ovirt.engine.core.vdsbroker.CreateVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-216) [] START, CreateVDSCommand( >> CreateVDSCommandParameters:{hostId='c3465ca2-395e-4c0c-b72e-b5b7153df452', >> vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b', vm='VM [yumcache]'}), log id: >> 5e07ba66 >> 2020-02-06 16:38:25,420Z INFO >> [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateBrokerVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-216) [] START, >> CreateBrokerVDSCommand(HostName = node1.ovirt.trashnet.xyz <http://node1.ovirt.trashnet.xyz/>, >> CreateVDSCommandParameters:{hostId='c3465ca2-395e-4c0c-b72e-b5b7153df452', >> vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b', vm='VM [yumcache]'}), log id: >> 1bfa03c4 >> 2020-02-06 16:38:25,424Z INFO >> [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuildUtils] >> (EE-ManagedThreadFactory-engine-Thread-216) [] Kernel FIPS - Guid: >> c3465ca2-395e-4c0c-b72e-b5b7153df452 fips: false >> 2020-02-06 16:38:25,435Z INFO >> [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateBrokerVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-216) [] VM <?xml version="1.0" >> encoding="UTF-8"?><domain type="kvm" xmlns:ovirt-tune=" >> http://ovirt.org/vm/tune/1.0 <http://ovirt.org/vm/tune/1.0>" xmlns:ovirt-vm="http://ovirt.org/vm/1.0 <http://ovirt.org/vm/1.0> >> "> >> <name>yumcache</name> >> <uuid>df9dbac4-35c0-40ee-acd4-a1cfc959aa8b</uuid> >> <memory>1048576</memory> >> <currentMemory>1048576</currentMemory> >> <iothreads>1</iothreads> >> <maxMemory slots="16">4194304</maxMemory> >> <vcpu current="1">16</vcpu> >> <sysinfo type="smbios"> >> <system> >> <entry name="manufacturer">oVirt</entry> >> <entry name="product">OS-NAME:</entry> >> <entry name="version">OS-VERSION:</entry> >> <entry name="serial">HOST-SERIAL:</entry> >> <entry name="uuid">df9dbac4-35c0-40ee-acd4-a1cfc959aa8b</entry> >> </system> >> </sysinfo> >> <clock offset="variable" adjustment="0"> >> <timer name="rtc" tickpolicy="catchup"/> >> <timer name="pit" tickpolicy="delay"/> >> <timer name="hpet" present="no"/> >> </clock> >> <features> >> <acpi/> >> </features> >> <cpu match="exact"> >> <model>EPYC</model> >> <feature name="ibpb" policy="require"/> >> <feature name="virt-ssbd" policy="require"/> >> <topology cores="1" threads="1" sockets="16"/> >> <numa> >> <cell id="0" cpus="0" memory="1048576"/> >> </numa> >> </cpu> >> <cputune/> >> <devices> >> <input type="tablet" bus="usb"/> >> <channel type="unix"> >> <target type="virtio" name="ovirt-guest-agent.0"/> >> <source mode="bind" >> path="/var/lib/libvirt/qemu/channels/df9dbac4-35c0-40ee-acd4-a1cfc959aa8b.ovirt-guest-agent.0"/> >> </channel> >> <channel type="unix"> >> <target type="virtio" name="org.qemu.guest_agent.0"/> >> <source mode="bind" >> path="/var/lib/libvirt/qemu/channels/df9dbac4-35c0-40ee-acd4-a1cfc959aa8b.org.qemu.guest_agent.0"/> >> </channel> >> <controller type="pci" model="pcie-root-port" index="1"> >> <address bus="0x00" domain="0x0000" function="0x0" slot="0x02" >> type="pci" multifunction="on"/> >> </controller> >> <memballoon model="virtio"> >> <stats period="5"/> >> <alias name="ua-27c77007-3a3c-4431-958d-90fd1c7257dd"/> >> <address bus="0x05" domain="0x0000" function="0x0" slot="0x00" >> type="pci"/> >> </memballoon> >> <controller type="pci" model="pcie-root-port" index="2"> >> <address bus="0x00" domain="0x0000" function="0x1" slot="0x02" >> type="pci"/> >> </controller> >> <controller type="pci" model="pcie-root-port" index="9"> >> <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" >> type="pci" multifunction="on"/> >> </controller> >> <controller type="sata" index="0"> >> <address bus="0x00" domain="0x0000" function="0x2" slot="0x1f" >> type="pci"/> >> </controller> >> <rng model="virtio"> >> <backend model="random">/dev/urandom</backend> >> <alias name="ua-51960005-6b95-47e9-82a7-67d5e0d6cf8a"/> >> </rng> >> <controller type="pci" model="pcie-root-port" index="6"> >> <address bus="0x00" domain="0x0000" function="0x5" slot="0x02" >> type="pci"/> >> </controller> >> <controller type="pci" model="pcie-root-port" index="15"> >> <address bus="0x00" domain="0x0000" function="0x6" slot="0x03" >> type="pci"/> >> </controller> >> <controller type="pci" model="pcie-root-port" index="13"> >> <address bus="0x00" domain="0x0000" function="0x4" slot="0x03" >> type="pci"/> >> </controller> >> <controller type="pci" model="pcie-root-port" index="7"> >> <address bus="0x00" domain="0x0000" function="0x6" slot="0x02" >> type="pci"/> >> </controller> >> <graphics type="vnc" port="-1" autoport="yes" passwd="*****" >> passwdValidTo="1970-01-01T00:00:01" keymap="en-us"> >> <listen type="network" network="vdsm-ovirtmgmt"/> >> </graphics> >> <controller type="pci" model="pcie-root-port" index="16"> >> <address bus="0x00" domain="0x0000" function="0x7" slot="0x03" >> type="pci"/> >> </controller> >> <controller type="pci" model="pcie-root-port" index="12"> >> <address bus="0x00" domain="0x0000" function="0x3" slot="0x03" >> type="pci"/> >> </controller> >> <video> >> <model type="qxl" vram="32768" heads="1" ram="65536" >> vgamem="16384"/> >> <alias name="ua-8a295e96-40c3-44de-a3b0-1c4a685a5473"/> >> <address bus="0x00" domain="0x0000" function="0x0" slot="0x01" >> type="pci"/> >> </video> >> <graphics type="spice" port="-1" autoport="yes" passwd="*****" >> passwdValidTo="1970-01-01T00:00:01" tlsPort="-1"> >> <channel name="main" mode="secure"/> >> <channel name="inputs" mode="secure"/> >> <channel name="cursor" mode="secure"/> >> <channel name="playback" mode="secure"/> >> <channel name="record" mode="secure"/> >> <channel name="display" mode="secure"/> >> <channel name="smartcard" mode="secure"/> >> <channel name="usbredir" mode="secure"/> >> <listen type="network" network="vdsm-ovirtmgmt"/> >> </graphics> >> <controller type="pci" model="pcie-root-port" index="5"> >> <address bus="0x00" domain="0x0000" function="0x4" slot="0x02" >> type="pci"/> >> </controller> >> <controller type="usb" model="qemu-xhci" index="0" ports="8"> >> <address bus="0x02" domain="0x0000" function="0x0" slot="0x00" >> type="pci"/> >> </controller> >> <controller type="pci" model="pcie-root-port" index="4"> >> <address bus="0x00" domain="0x0000" function="0x3" slot="0x02" >> type="pci"/> >> </controller> >> <controller type="pci" model="pcie-root-port" index="3"> >> <address bus="0x00" domain="0x0000" function="0x2" slot="0x02" >> type="pci"/> >> </controller> >> <controller type="pci" model="pcie-root-port" index="11"> >> <address bus="0x00" domain="0x0000" function="0x2" slot="0x03" >> type="pci"/> >> </controller> >> <controller type="scsi" model="virtio-scsi" index="0"> >> <driver iothread="1"/> >> <alias name="ua-d0bf6fcd-7aa2-4658-b7cc-3dac259b7ad2"/> >> <address bus="0x03" domain="0x0000" function="0x0" slot="0x00" >> type="pci"/> >> </controller> >> <controller type="pci" model="pcie-root-port" index="8"> >> <address bus="0x00" domain="0x0000" function="0x7" slot="0x02" >> type="pci"/> >> </controller> >> <controller type="pci" model="pcie-root-port" index="14"> >> <address bus="0x00" domain="0x0000" function="0x5" slot="0x03" >> type="pci"/> >> </controller> >> <controller type="pci" model="pcie-root-port" index="10"> >> <address bus="0x00" domain="0x0000" function="0x1" slot="0x03" >> type="pci"/> >> </controller> >> <controller type="virtio-serial" index="0" ports="16"> >> <address bus="0x04" domain="0x0000" function="0x0" slot="0x00" >> type="pci"/> >> </controller> >> <channel type="spicevmc"> >> <target type="virtio" name="com.redhat.spice.0"/> >> </channel> >> <controller type="pci" model="pcie-root"/> >> <interface type="bridge"> >> <model type="virtio"/> >> <link state="up"/> >> <source bridge="vmnet"/> >> <alias name="ua-ceda0ef6-9139-4e5c-8840-86fe344ecbd3"/> >> <address bus="0x01" domain="0x0000" function="0x0" slot="0x00" >> type="pci"/> >> <mac address="56:6f:91:b9:00:05"/> >> <mtu size="1500"/> >> <filterref filter="vdsm-no-mac-spoofing"/> >> <bandwidth/> >> </interface> >> <disk type="file" device="cdrom" snapshot="no"> >> <driver name="qemu" type="raw" error_policy="report"/> >> <source file="" startupPolicy="optional"> >> <seclabel model="dac" type="none" relabel="no"/> >> </source> >> <target dev="sdc" bus="sata"/> >> <readonly/> >> <alias name="ua-bdf99844-2d02-411b-90bb-671ee26764cb"/> >> <address bus="0" controller="0" unit="2" type="drive" >> target="0"/> >> </disk> >> <disk snapshot="no" type="network" device="disk"> >> <target dev="sda" bus="scsi"/> >> <source protocol="gluster" >> name="vmstore/781717e5-1cff-43a1-b586-9941503544e8/images/a1d56b14-6d72-4f46-a0aa-eb0870c36bc4/a2314816-7970-49ce-a80c-ab0d1cf17c78"> >> <host name="node1.fs.trashnet.xyz <http://node1.fs.trashnet.xyz/>" port="0"/> >> <seclabel model="dac" type="none" relabel="no"/> >> </source> >> <driver name="qemu" discard="unmap" io="native" type="qcow2" >> error_policy="stop" cache="none"/> >> <alias name="ua-a1d56b14-6d72-4f46-a0aa-eb0870c36bc4"/> >> <address bus="0" controller="0" unit="0" type="drive" >> target="0"/> >> <boot order="1"/> >> <serial>a1d56b14-6d72-4f46-a0aa-eb0870c36bc4</serial> >> </disk> >> <lease> >> <key>df9dbac4-35c0-40ee-acd4-a1cfc959aa8b</key> >> <lockspace>781717e5-1cff-43a1-b586-9941503544e8</lockspace> >> <target offset="6291456" path="/rhev/data-center/mnt/glusterSD/ >> node1.fs.trashnet.xyz <http://node1.fs.trashnet.xyz/> >> :_vmstore/781717e5-1cff-43a1-b586-9941503544e8/dom_md/xleases"/> >> </lease> >> </devices> >> <pm> >> <suspend-to-disk enabled="no"/> >> <suspend-to-mem enabled="no"/> >> </pm> >> <os> >> <type arch="x86_64" machine="pc-q35-rhel7.6.0">hvm</type> >> <smbios mode="sysinfo"/> >> </os> >> <metadata> >> <ovirt-tune:qos/> >> <ovirt-vm:vm> >> <ovirt-vm:minGuaranteedMemoryMb >> type="int">512</ovirt-vm:minGuaranteedMemoryMb> >> <ovirt-vm:clusterVersion>4.3</ovirt-vm:clusterVersion> >> <ovirt-vm:custom/> >> <ovirt-vm:device mac_address="56:6f:91:b9:00:05"> >> <ovirt-vm:custom/> >> </ovirt-vm:device> >> <ovirt-vm:device devtype="disk" name="sda"> >> >> <ovirt-vm:poolID>2ffaec76-462c-11ea-b155-00163e512202</ovirt-vm:poolID> >> >> <ovirt-vm:volumeID>a2314816-7970-49ce-a80c-ab0d1cf17c78</ovirt-vm:volumeID> >> >> <ovirt-vm:imageID>a1d56b14-6d72-4f46-a0aa-eb0870c36bc4</ovirt-vm:imageID> >> >> <ovirt-vm:domainID>781717e5-1cff-43a1-b586-9941503544e8</ovirt-vm:domainID> >> </ovirt-vm:device> >> <ovirt-vm:launchPaused>false</ovirt-vm:launchPaused> >> <ovirt-vm:resumeBehavior>kill</ovirt-vm:resumeBehavior> >> </ovirt-vm:vm> >> </metadata> >> </domain> >> >> 2020-02-06 16:38:25,455Z INFO >> [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateBrokerVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-216) [] FINISH, >> CreateBrokerVDSCommand, return: , log id: 1bfa03c4 >> 2020-02-06 16:38:25,494Z INFO >> [org.ovirt.engine.core.vdsbroker.CreateVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-216) [] FINISH, CreateVDSCommand, >> return: WaitForLaunch, log id: 5e07ba66 >> 2020-02-06 16:38:25,495Z INFO >> [org.ovirt.engine.core.bll.RunVmCommand] >> (EE-ManagedThreadFactory-engine-Thread-216) [] Lock freed to object >> 'EngineLock:{exclusiveLocks='[df9dbac4-35c0-40ee-acd4-a1cfc959aa8b=VM]', >> sharedLocks=''}' >> 2020-02-06 16:38:25,533Z INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (EE-ManagedThreadFactory-engine-Thread-216) [] EVENT_ID: >> USER_STARTED_VM(153), VM yumcache was started by admin@internal-authz >> (Host: node1.ovirt.trashnet.xyz <http://node1.ovirt.trashnet.xyz/>). >> 2020-02-06 16:38:33,300Z INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (ForkJoinPool-1-worker-5) [] VM 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b' was >> reported as Down on VDS 'c3465ca2-395e-4c0c-b72e-b5b7153df452'( >> node1.ovirt.trashnet.xyz <http://node1.ovirt.trashnet.xyz/>) >> 2020-02-06 16:38:33,301Z INFO >> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] >> (ForkJoinPool-1-worker-5) [] START, DestroyVDSCommand(HostName = >> node1.ovirt.trashnet.xyz <http://node1.ovirt.trashnet.xyz/>, >> DestroyVmVDSCommandParameters:{hostId='c3465ca2-395e-4c0c-b72e-b5b7153df452', >> vmId='df9dbac4-35c0-40ee-acd4-a1cfc959aa8b', secondsToWait='0', >> gracefully='false', reason='', ignoreNoVm='true'}), log id: 1f951ea9 >> 2020-02-06 16:38:33,478Z INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] >> (EE-ManagedThreadFactory-engineScheduled-Thread-8) [] Fetched 2 VMs from >> VDS 'c3465ca2-395e-4c0c-b72e-b5b7153df452' >> 2020-02-06 16:38:33,545Z INFO >> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] >> (ForkJoinPool-1-worker-5) [] FINISH, DestroyVDSCommand, return: , log id: >> 1f951ea9 >> 2020-02-06 16:38:33,546Z INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (ForkJoinPool-1-worker-5) [] VM >> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'(yumcache) moved from 'WaitForLaunch' >> --> 'Down' >> 2020-02-06 16:38:33,623Z ERROR >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (ForkJoinPool-1-worker-5) [] EVENT_ID: VM_DOWN_ERROR(119), VM yumcache is >> down with error. Exit message: internal error: qemu unexpectedly closed the >> monitor: [2020-02-06 16:38:31.723977] E [MSGID: 108006] >> [afr-common.c:5323:__afr_handle_child_down_event] 0-vmstore-replicate-0: >> All subvolumes are down. Going offline until at least one of them comes >> back up. >> [2020-02-06 16:38:31.724765] I [io-stats.c:4027:fini] 0-vmstore: >> io-stats translator unloaded >> 2020-02-06T16:38:32.573511Z qemu-kvm: -drive file=gluster:// >> node1.fs.trashnet.xyz:24007/vmstore/781717e5-1cff-43a1-b586-9941503544e8/images/a1d56b14-6d72-4f46-a0aa-eb0870c36bc4/a2314816-7970-49ce-a80c-ab0d1cf17c78,file.debug=4,format=qcow2,if=none,id=drive-ua <http://node1.fs.trashnet.xyz:24007/vmstore/781717e5-1cff-43a1-b586-9941503544e8/images/a1d56b14-6d72-4f46-a0aa-eb0870c36bc4/a2314816-7970-49ce-a80c-ab0d1cf17c78,file.debug=4,format=qcow2,if=none,id=drive-ua>-a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,serial=a1d56b14-6d72-4f46-a0aa-eb0870c36bc4,werror=stop,rerror=stop,cache=none,discard=unmap,aio=native: >> Could not read qcow2 header: Invalid argument. >> 2020-02-06 16:38:33,624Z INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (ForkJoinPool-1-worker-5) [] add VM >> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'(yumcache) to rerun treatment >> 2020-02-06 16:38:33,796Z ERROR >> [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] >> (ForkJoinPool-1-worker-5) [] Rerun VM >> 'df9dbac4-35c0-40ee-acd4-a1cfc959aa8b'. Called from VDS ' >> node1.ovirt.trashnet.xyz <http://node1.ovirt.trashnet.xyz/>' >> 2020-02-06 16:38:33,899Z WARN >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (EE-ManagedThreadFactory-engine-Thread-223) [] EVENT_ID: >> USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM yumcache on Host >> node1.ovirt.trashnet.xyz <http://node1.ovirt.trashnet.xyz/>. >> _______________________________________________ >> Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> >> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/6GTBANZ4R44HJE... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6GTBANZ4R44HJE2BU55GAEBLTETUXTKT/> >> > _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/5EZYM4OADD65RV...
_______________________________________________
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KKUVN3ZR5QOVKK...
Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK. <https://interactiv-group.com/disclaimer.html <https://interactiv-group.com/disclaimer.html>> _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S2RYKJLRE533BZ...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/23SAGPLPE5MHCG...
My guess is different - a real Load Balancer , like HAproxy, can take care when a backend server is down.
Still, it will be nice to learn Darrell's solution.
Best Regards, Strahil Nikolov