October 2020 - Users - oVirt List Archives

Last week for participating to oVirt Survey Autumn 2020
by Sandro Bonazzola 12 Oct '20

12 Oct '20

Hi, just a kind reminder this is last week for participating to oVirt Survey Autumn 2020! The survey is closing on October 18 2020. You can submit your response at https://docs.google.com/forms/u/1/d/e/1FAIpQLSdzzh_MSsSq-LSQLauJzuaHC0Va1ba… *Please note the answers to this survey will be publicly accessible.* This survey is under oVirt Privacy Policy available at https://www.ovirt.org/privacy-policy.html -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>* * <https://www.redhat.com/it/forums/emea/italy-track>*

1 0

oVirt 4.4 - HA Proxy Fencing
by Jeremey Wise 11 Oct '20

11 Oct '20

I have a pair of nodes which service DNS / NTP / FTP / AD /Kerberos / IPLB etc.. ns01, ns02 These two "infrastructure VMs have HA Proxy and pacemaker and I have set to have "HA" within ovirt and node affinity. But.. within HAProxy, the nodes use to be able to call the STONITH function of KVM to reset a node if / as needed. Fence Device: "stonith:fence_virsh" But with oVirt this function no longer works. For VMs and other applications which need STONITH functions, how would this be done with oVirt. I would assume the call would be to oVirt-Engine now, but not seeing documentation on HA-Proxy / Pacemaker to use oVirt. Can someone point me in the right direction? -- penguinpages

2 1

Fencing with oVirt 4.4 apc master switch
by Jeremey Wise 11 Oct '20

11 Oct '20

I use to have hand built CentOS+KVM + Gluster. Moved to HCI oVirt controlled system and so far, been very happy with stability and quality. One feature that was working in old "hand build" version was fencing. I have an old APC master switch AP9606. I use to have and use it for master switch functions of nodes but sense replacement with oVirt it has not worked. Here are setup notes and symptoms IP: 172.16.100.2 Username: apc Password: <blah> Port 8 (it is an eight port unit) Slot: <empty as I have no idea what is expects here I also input values here but no change> Error: Test failed: WARNING:root:Parse error: Ignoring option 'agent' because it does not have value ERROR:root:Unable to connect/login to fencing device 2020-10-11 11:28:50,885 ERROR: Unable to connect/login to fencing device Questions: 1) Has anyone setup this with oVirt such that it uses apc master switch 2) Where is oVirt putting these logs / running this service. I assumed oVirt engine running the services and so tried to find details and logs here. -- penguinpages

1 0

ovirt broker how to modify network_test dns ip address?
by wodel youchi 11 Oct '20

11 Oct '20

Hi, How to modify the ip address of the DNS server used by ovirt-broker for testing? Regards.

2 1

too many glitches while moving storage domain
by Jiří Sléžka 11 Oct '20

11 Oct '20

Hi, today I started one not so common operation and I would like to share my (not so good) experience. I have 4 old Opteron G3 hosts which creates 1 DC with 1 cluster with 4.2 compatibility. Then I got 2 newer Intel based hosts which creates separate DC and cluster. I use one shared FC storage with few LUNs for all the stuff. Intel cluster is selfhosted with oVirt 4.4.2 where I migrated original standalone oVirt 4.3.10 manager. So I have 2 DCs, one with 4.2 and one with 4.4 compatibility. Now the funny part... I had 2 LUNs connected to old 4.2 DC. My intention was detach one of them and attach it to the new DC. first pain was that some of vms had ther disks on both LUNs. It is indicated late in the process and without hint which vms they are. So I had to activate LUN in old DC and tried to find that vms (search string in vms tab "Storage = one and Storage = two" seems not working). Ok, it took two or three rounds... then, also late in the process there was problem that one vm had previewed snapshot so another round with activation of LUN in old DC... Then I was able detach and import LUN to new DC. Nice, but with warning that LUN has 4.2 compatibility which will be converted to 4.4 and there is no way back to connect it to old DC... It is logical but very scary if something went wrong... but it did not in my case :-) LUN is connected in new DC. Now I had to import vms. Most vms were imported ok but two of them were based on template wich resides on other LUN. It was not indicated during detaching! It looks like I cannot move template from storage to storage other way than through Export storage (which I don't have at this moment) or through OVA export for which I have not enough free storage space on hosts. Its a trap! :-) Btw. there is no check for free space while starting export to OVA (template uses preallocated disk). Exporting task still runs but there is no free space at the host... and probably no way to cancel it from manager :-( Ok, I had most of the vms imported. Last really strange thing is that I lost one vm during import. It is not listed in VirtualMachines nor in VM import tab on starage nor in Disk tab... that vm was an Aruba migration tool and was imported from OVN image. In fact there are two disks in Disk import tab, one of them has no Alias/description and was created today around time I started work on this migration. The second one has alias "vmdisk1" and is few months older but I have no idea if it is the lost vm... Sorry for long story, TL;DR version could be: There are glitches in some (not so common) workflows... Cheers, Jiri

2 1

Nodes with a version and others with anothers
by Julio Cesar Bustamante 09 Oct '20

09 Oct '20

Hi everyone is it recommendable to have nodes with ovirt host with a version and others nodes with another version? for example 5 nodes with this description Versión SO -: RHEL - 7 - 6.1810.2.el7.centos Descripción de SO: CentOS Linux 7 (Core) Versión del kernel: 3.10.0 - 957.10.1.el7.x86_64 Versión KVM: 2.12.0 - 18.el7_6.3.1 Versión LIBVIRT: libvirt-4.5.0-10.el7_6.6 Versión VDSM: vdsm-4.20.46-1.el7 Versión de SPICE: 0.14.0 - 6.el7_6.1 Versión GlusterFS: [N. D.] Versión CEPH: librbd1-10.2.5-4.el7 And a node with ths Versión SO -: RHEL - 7 - 5.1804.5.el7.centos Descripción de SO: oVirt Node 4.2.7.1 Versión del kernel: 3.10.0 - 862.14.4.el7.x86_64 Versión KVM: 2.10.0 - 21.el7_5.7.1 Versión LIBVIRT: libvirt-3.9.0-14.el7_5.8 Versión VDSM: vdsm-4.20.43-1.el7 Versión de SPICE: 0.14.0 - 2.el7_5.5 Versión GlusterFS: glusterfs-3.12.15-1.el7 Versión CEPH: librbd1-0.94.5-2.el7

1 0

Disconnected -- Server has closed the connection
by info＠worldhostess.com 09 Oct '20

09 Oct '20

A first part of the fresh install is without any problems and the second part of the fresh install the storage part "Disconnected - Server has closed the connection." Thereafter the OVirt is unstable and disconnect continuously. "Unexpected error - Cockpit had an unexpected internal error." "You can try restarting Cockpit by pressing refresh in your browser. The javascript console contains details about this error (Ctrl-Shift-J in most browsers)." Ctrl-Shift-J in most browsers --- RESULTS DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/base1/jquery.min.js.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/base1/cockpit.min.js.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/shell/index.min.js.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/base1/jquery.min.js.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/base1/cockpit.min.js.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/base1/patternfly.min.css.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE index.js:15 Refused to apply inline style because it violates the following Content Security Policy directive: "default-src 'self' https://XYZ.co.za:9090". Either the 'unsafe-inline' keyword, a hash ('sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU='), or a nonce ('nonce-...') is required to enable inline execution. Note also that 'style-src' was not explicitly set, so 'default-src' is used as a fallback. h @ index.js:15 t.inject @ index.js:15 t.a @ index.js:15 (anonymous) @ index.js:6 (anonymous) @ index.js:6 n @ index.js:1 (anonymous) @ index.js:72 n @ index.js:1 (anonymous) @ index.js:15 n @ index.js:1 (anonymous) @ index.js:1 (anonymous) @ index.js:1 DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/shell/index.css.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/shell/index.min.js.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca082ec115b4b7401aa8829/ov irt-dashboard/index.html#/:1 Refused to apply style from 'https://XYZ.co.za:9090/src/base1/patternfly.css' because its MIME type ('text/html') is not a supported stylesheet MIME type, and strict MIME checking is enabled. updates.js:11 Refused to apply inline style because it violates the following Content Security Policy directive: "default-src 'self' https://XYZ.co.za:9090". Either the 'unsafe-inline' keyword, a hash ('sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU='), or a nonce ('nonce-...') is required to enable inline execution. Note also that 'style-src' was not explicitly set, so 'default-src' is used as a fallback. m @ updates.js:11 t.inject @ updates.js:11 t.a @ updates.js:11 (anonymous) @ updates.js:6 (anonymous) @ updates.js:6 n @ updates.js:1 (anonymous) @ updates.js:43 n @ updates.js:1 (anonymous) @ updates.js:11 n @ updates.js:1 (anonymous) @ updates.js:1 (anonymous) @ updates.js:1 DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/base1/cockpit.min.js.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/base1/jquery.min.js.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE overview.js:20 Refused to apply inline style because it violates the following Content Security Policy directive: "default-src 'self' https://XYZ.co.za:9090". Either the 'unsafe-inline' keyword, a hash ('sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU='), or a nonce ('nonce-...') is required to enable inline execution. Note also that 'style-src' was not explicitly set, so 'default-src' is used as a fallback. p @ overview.js:20 t.inject @ overview.js:20 t.a @ overview.js:20 (anonymous) @ overview.js:6 (anonymous) @ overview.js:6 n @ overview.js:1 (anonymous) @ overview.js:20 n @ overview.js:1 (anonymous) @ overview.js:77 n @ overview.js:1 (anonymous) @ overview.js:82 n @ overview.js:1 (anonymous) @ overview.js:82 n @ overview.js:1 (anonymous) @ overview.js:1 (anonymous) @ overview.js:1 DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/base1/patternfly.min.css.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/system/services.css.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE services.js:19 Refused to apply inline style because it violates the following Content Security Policy directive: "default-src 'self' https://XYZ.co.za:9090". Either the 'unsafe-inline' keyword, a hash ('sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU='), or a nonce ('nonce-...') is required to enable inline execution. Note also that 'style-src' was not explicitly set, so 'default-src' is used as a fallback. m @ services.js:19 t.inject @ services.js:19 t.a @ services.js:19 (anonymous) @ services.js:6 (anonymous) @ services.js:6 n @ services.js:1 (anonymous) @ services.js:19 n @ services.js:1 (anonymous) @ services.js:19 n @ services.js:1 (anonymous) @ services.js:56 n @ services.js:1 (anonymous) @ services.js:56 n @ services.js:1 (anonymous) @ services.js:1 (anonymous) @ services.js:1 DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/base1/cockpit.min.css.map: Certificate error: net::ERR_CERT_AUTHORITY_INVALID DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/updates/updates.min.js.map: Certificate error: net::ERR_CERT_AUTHORITY_INVALID DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/updates/updates.css.map: Certificate error: net::ERR_CERT_AUTHORITY_INVALID DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/system/overview.min.js.map: Certificate error: net::ERR_CERT_AUTHORITY_INVALID DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/system/overview.css.map: Certificate error: net::ERR_CERT_AUTHORITY_INVALID DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/system/services.min.js.map: Certificate error: net::ERR_CERT_AUTHORITY_INVALID DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/performance/performance.min.js.map: Certificate error: net::ERR_CERT_AUTHORITY_INVALID DevTools failed to load SourceMap: Could not load content for https://XYZ.co.za:9090/cockpit/$1aa0545898ca0350c292c35fee90966b96bf77ad1ca0 82ec115b4b7401aa8829/base1/patternfly.min.css.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE 5[Intervention] Slow network is detected. See <URL> for more details. Fallback font will be used while loading: <URL> cockpit.js:606 grep: /sys/class/dmi/id/power/autosuspend_delay_ms: Input/output error p @ cockpit.js:606 m @ cockpit.js:616 v @ cockpit.js:511 a.onmessage.o.dispatch_data @ cockpit.js:441 e @ cockpit.js:297 postMessage (async) (anonymous) @ index.js:72 a.onmessage.o.dispatch_data @ cockpit.js:439 cockpit.js:606 which: no nodectl in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin) p @ cockpit.js:606 m @ cockpit.js:616 v @ cockpit.js:511 a.onmessage.o.dispatch_data @ cockpit.js:441 e @ cockpit.js:297 postMessage (async) (anonymous) @ index.js:72 a.onmessage.o.dispatch_data @ cockpit.js:439 app.js:43 nodectl is not installed. Disabling node functionality index.js:72 transport closed: disconnected (anonymous) @ index.js:72 dispatch @ jquery.js:3816 v.handle @ jquery.js:3679 se @ cockpit.js:76 e @ cockpit.js:223 p @ cockpit.js:607 m @ cockpit.js:616 e @ cockpit.js:456 a.onclose @ cockpit.js:421 patternfly.css:1 GET https://XYZ.co.za:9090/cockpit/static/fonts/RedHatDisplay-Medium.woff2 net::ERR_CERT_AUTHORITY_INVALID Yours Sincerely, Henni

2 2

Re: Gluster volume not responding
by Strahil Nikolov 09 Oct '20

09 Oct '20

Hi Simon, I doubt the system needs tuning from network perspective. I guess you can run some 'screen'-s which a pinging another system and logging everything to a file. Best Regards, Strahil Nikolov В петък, 9 октомври 2020 г., 01:05:22 Гринуич+3, Simon Scott <simon(a)justconnect.ie> написа: Thanks Strahil. I have found between 1 & 4 Gluster peer rpc-clnt-ping timer expired messages in the rhev-data-center-mnt-glusterSD-hostname-strg:_pltfm_data01.log on the storage network IP. Of the 6 Hosts only 1 does not have these timeouts. Fencing has been disabled but can you identify which logs are key to identifying the cause please. It's a bonded (bond1) 10GB ovirt-mgmt logical network and Prod VM VLAN interface AND a bonded (bond2) 10GB Gluster storage network. Dropped packets are seen incrementing in the vdsm.log but neither ethtool -S or kernel logs are showing dropped packets. I am wondering if they are being dropped due to the ring buffers being small. Kind Regards Shimme ________________________________ From: Strahil Nikolov <hunter86_bg(a)yahoo.com> Sent: Thursday 8 October 2020 20:40 To: users(a)ovirt.org <users(a)ovirt.org>; Simon Scott <simon(a)justconnect.ie> Subject: Re: [ovirt-users] Gluster volume not responding >Every Monday and Wednesday morning there are gluster connectivity timeouts >but all checks of the network and network configs are ok. Based on this one I make the following conclusions: 1. Issue is reoccuring 2. You most probably have a network issue Have you checked the following: - are there any ping timeouts between fuse clients and gluster nodes - Have you tried to disable fencing and check the logs after the issue reoccurs - Are you sharing Blackup and Prod networks ? Is it possible some backup/other production load in your environment to "black-out" your oVirt ? - Have you check the gluster cluster's logs for anything meaningful ? Best Regards, Strahil Nikolov

1 0

Moving VM disks from one storage domain to another. Automate?
by Green, Jacob Allen /C 09 Oct '20

09 Oct '20

I am looking for an automated way, via Ansible to move a VM disk from one storage domain to another. I found the following, https://docs.ansible.com/ansible/latest/modules/ovirt_disk_module.html and while it mentions copying a VM disk image from one domain to another it does not mention a live storage migration. Which is what I am looking to do. I want to take roughly 100 VMs and move their disk images from one domain to another that is available to the datacenter in some automated/scripted fashion. I am just curious if anyone out there has had to do this and how they tackled it. Or perhaps I am missing some easy obvious way, other than clicking all the disks and clicking move. However from the looks of it, if I did click all the disk and selected move, it appears RHV tries to do them all at once, which is probably not ideal, I would like it to move the disks in a serial One after another fashion, to conserve throughput and IO. I also did not see anything on Ansible galaxy or the ovirt github that would do this. Thank you.

3 2

Gluster volume not responding
by simon＠justconnect.ie 09 Oct '20

09 Oct '20

Hi, Every Monday and Wednesday morning there are gluster connectivity timeouts but all checks of the network and network configs are ok. Description of problem: The following entries were found in the engine.log following VMs becoming unresponsive and hosts fencing. This issue has been causing issues since the beginning of September and no amount of reading logs is helping. This issue occurs every Wednesday morning at exactly the same time. WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-974045) [] domain 'bc482086-598b-46b1-9189-0146fa03447c:pltfm_data03' in problem 'PROBLEMATIC'. vds: 'bdtpltfmovt02' WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-974069) [] domain 'bf807836-b64e-4913-ab41-cfe04ca9abab:pltfm_data01' in problem 'PROBLEMATIC'. vds: 'bdtpltfmovt02' WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-974121) [] domain 'bc482086-598b-46b1-9189-0146fa03447c:pltfm_data03' in problem 'PROBLEMATIC'. vds: 'bdtpltfmovt03' INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] VM '00a082de-c827-4c97-9846-ec32d1ddbfa6'(bdtfmnpproddb03) moved from 'Up' --> 'NotResponding' INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] VM 'f5457f04-054e-4684-9702-40ed4a3e4bdb'(bdtk8shaproxy02) moved from 'Up' --> 'NotResponding' INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] VM 'bea85b27-18e7-4936-9871-cdb987baebdd'(bdtdepjump) moved from 'Up' --> 'NotResponding' INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] VM 'ba1d4fe2-97e7-491a-9485-8319281e7784'(bdtcmgmtnfs01) moved from 'Up' --> 'NotResponding' INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] VM '9e253848-7153-43e8-8126-dba2d7f2d214'(bdtdepnfs01) moved from 'Up' --> 'NotResponding' INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] VM '1e58d106-4b65-4296-8d11-2142abb7808e'(bdtionjump) moved from 'Up' --> 'NotResponding' VDSM Log from one of the Gluster Peers: [2020-10-05 03:03:25.038883] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-pltfm_data02-client-0: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected] [2020-10-07 05:37:39.582138] C [rpc-clnt-ping.c:162:rpc_clnt_ping_timer_expired] 0-pltfm_data02-client-0: server x.x.x.x:49153 has not responded in the last 30 seconds, disconnecting. [2020-10-07 05:37:39.583217] I [MSGID: 114018] [client.c:2288:client_rpc_notify] 0-pltfm_data02-client-0: disconnected from pltfm_data02-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2020-10-07 05:37:39.584213] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fe83aa96fbb] (--> /lib64/libgfrpc.so.0(+0xce11)[0x7fe83a85fe11] (--> /lib64/libgfrpc.so.0(+0xcf2e)[0x7fe83a85ff2e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x7fe83a861521] (--> /lib64/libgfrpc.so.0(+0xf0c8)[0x7fe83a8620c8] ))))) 0-pltfm_data02-client-0: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-10-07 05:37:09.003907 (xid=0x7e6a8c) Current Version: 4.3.4.3-1.el7 - Although we are keen to upgrade, we need stability for this production environment before doing so. This Data Center has 2 x 3 Node clusters (Admin & Platform) which each have a 3 Replica Gluster configuration which is not managed by the self hosted ovirt engine. Any assistance is appreciated. Regards Shimme

3 3