Did you manage to fix the issue with the 0-xlator ? If yes, most probably the issue will be OK. Yet 'probably' doesn't meant that they 'will' be OK.If it was my lab - I would go ahead only if the 0-xlator issue is over.Yet, a lab is a different thing than prod - so it is your sole decision. Did you test the upgrade prior moving to Prod ? About the hooks - I had such issue before and I had to reinstall the gluster rpms to solve it. Best Regards,Strahil Nikolov В вторник, 4 февруари 2020 г., 16:35:27 ч. Гринуич+2, Goorkate, B.J. <b.j.goorkate@umcutrecht.nl> написа: Hi Strahil, Thanks for your time so far! The packages seem fine on all of the 3 nodes. Only /var/lib/glusterd/glusterd.info is modified and on the not yet upgraded nodes these files are missing: missing /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh missing /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh missing /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh missing /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh missing /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh But that doesn't seem relevant... I stopped a couple of virtual machines with image files with unsynched entries. When they were turned off, I couldn't find them anymore in the unsyced entries list. When I turned them on again, some re-appeared, some didn't. I really don't know where to look next. The big question is: will the problems be resolved when I upgrade the two remaining nodes, or will it get worse? Regards, Bertjan On Sun, Feb 02, 2020 at 08:06:58PM +0000, Strahil Nikolov wrote:
It seems that something went "off". That '0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol: xlator_api' is really worrisome. I think that you might have a bad package and based on the info it could be glusterfs-libs which should provide /usr/lib64/libgfrpc.so.0 . I'm currently on gluster v7.0 and I can't check it on my installation. Run a for loop to check the rpms: for i in $(rpm -qa | grep gluster) ; do echo "$i :" ; rpm -V $i'; echo;echo; done Most probably you can safely redo (on the upgraded node) the last yum transaction: yum history yum history info <ID> -> verify the gluster packages were installed here yum history redo <ID> If kernel, glibc, systemd were not in this transaction , you can stop gluster and start it again: Node in maintenance in oVirt systemctl stop ovirt-ha-agent ovirt-ha-broker vdsmd supervdsmd systemctl stop sanlock systemctl stop glusterd /usr/share/glusterfs/scripts/stop-all-gluster-processes.sh And then power up again: systemctl start glusterd gluster volume status -> check all connected systemctl start sanlock supervdsmd vdsmd systemctl start ovirt-ha-broker ovirt-ha-agent Check situation. Yet, you need to make gluster stop complaining , before you can take care of the heal. Usually 'rsync' is my best friend - but this is when gluster is working normally - and your case is far from normal. If redo doesn't work for you -> try the "yum history rollback" to recover to last good state. I think that 'BOOM Boot Manager' is best in such cases. Note: Never take any of my words for granted. I'm not running oVirt in production and some of my methods might not be OK for your environment. Best Regards, Strahil Nikolov В неделя, 2 февруари 2020 г., 08:56:01 ч. Гринуич+2, Mark Lamers <mark.r.lamers@gmail.com> написа:
Hi Strahil,
Bertjan is not in the office today, so I will reply if okay with you.
First I like to describe the status of our network
There are three bricks:
and 5 nodes:
Every host has a management and migrate vlan iface on a different bond iface.
The last octet of the ipaddress is similar
The output from 'gluster volume heal <volname> info' gives a long list of shards from all three nodes, the file is attached as 'gluster_volume_heal_info.txt'. The node without shards in the list is the updated node to gluster6, oVirt 4.3. That is curious i think.
Further more I find the following errors ' E ' in the glusterd.log of the upgrated host:
[2020-01-26 17:40:46.147730] E [MSGID: 106118] [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to acquire lock for vmstore1 [2020-01-26 22:47:16.655651] E [MSGID: 106118] [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to acquire lock for vmstore1 [2020-01-27 07:07:51.815490] E [MSGID: 106118] [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to acquire lock for vmstore1 [2020-01-27 18:28:14.953974] E [MSGID: 106118] [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to acquire lock for vmstore1 [2020-01-27 18:58:22.629457] E [MSGID: 106118] [glusterd-op-sm.c:4133:glusterd_op_ac_lock] 0-management: Unable to acquire lock for vmstore1 [2020-01-27 18:58:22.629595] E [MSGID: 106376] [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1 [2020-01-27 18:58:22.756430] E [MSGID: 106117] [glusterd-op-sm.c:4189:glusterd_op_ac_unlock] 0-management: Unable to release lock for vmstore1 [2020-01-27 18:58:22.756581] E [MSGID: 106376] [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1 [2020-01-28 05:31:52.427196] E [MSGID: 106118] [glusterd-op-sm.c:4133:glusterd_op_ac_lock] 0-management: Unable to acquire lock for vmstore1 [2020-01-28 05:31:52.427315] E [MSGID: 106376] [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1 [2020-01-28 05:31:52.537799] E [MSGID: 106115] [glusterd-mgmt.c:117:gd_mgmt_v3_collate_errors] 0-management: Locking failed on 10.13.250.14. Please check log file for details. [2020-01-28 05:31:52.537973] E [MSGID: 106150] [glusterd-syncop.c:1931:gd_sync_task_begin] 0-management: Locking Peers Failed. [2020-01-28 05:31:52.539620] E [MSGID: 106117] [glusterd-op-sm.c:4189:glusterd_op_ac_unlock] 0-management: Unable to release lock for vmstore1 [2020-01-28 05:31:52.539759] E [MSGID: 106376] [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1 [2020-01-28 05:31:52.539937] E [MSGID: 106115] [glusterd-mgmt.c:117:gd_mgmt_v3_collate_errors] 0-management: Unlocking failed on 10.13.250.14. Please check log file for details. [2020-01-28 05:31:52.540446] E [MSGID: 106151] [glusterd-syncop.c:1616:gd_unlock_op_phase] 0-management: Failed to unlock on some peer(s) [2020-01-28 08:51:45.638694] E [MSGID: 106118] [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to acquire lock for vmstore1 [2020-01-28 08:52:45.709950] E [MSGID: 106118] [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to acquire lock for vmstore1 [2020-01-28 08:54:12.455555] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol: xlator_api [2020-01-28 08:54:16.214779] E [MSGID: 106118] [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to acquire lock for vmstore1 [2020-01-28 08:57:59.496842] E [MSGID: 106118] [glusterd-op-sm.c:4133:glusterd_op_ac_lock] 0-management: Unable to acquire lock for vmstore1 [2020-01-28 08:57:59.496905] E [MSGID: 106376] [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1 [2020-01-28 08:57:59.505119] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli read-only, ProgVers: 2, Proc: 5) to rpc-transport (socket.management) [2020-01-28 08:57:59.505135] E [MSGID: 106430] [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed [2020-01-28 08:57:59.647456] E [MSGID: 106117] [glusterd-op-sm.c:4189:glusterd_op_ac_unlock] 0-management: Unable to release lock for vmstore1 [2020-01-28 08:57:59.647508] E [MSGID: 106376] [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: 1 [2020-01-28 08:57:59.654929] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, ProgVers: 2, Proc: 27) to rpc-transport (socket.management) [2020-01-28 08:57:59.654943] E [MSGID: 106430] [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed [2020-01-28 09:07:34.941350] E [MSGID: 106118] [glusterd-op-sm.c:4133:glusterd_op_ac_lock] 0-management: Unable to acquire lock for vmstore1 [2020-01-28 09:07:34.941391] E [MSGID: 106376] [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1 [2020-01-28 09:07:35.042466] E [MSGID: 106117] [glusterd-op-sm.c:4189:glusterd_op_ac_unlock] 0-management: Unable to release lock for vmstore1 [2020-01-28 09:07:35.042510] E [MSGID: 106376] [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1 [2020-01-28 09:13:32.329172] E [MSGID: 106244] [glusterd.c:1785:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2020-01-28 09:13:44.024431] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol: xlator_api [2020-01-28 09:45:46.347499] E [MSGID: 106118] [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to acquire lock for vmstore1 [2020-01-28 09:46:45.837466] E [MSGID: 106118] [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to acquire lock for vmstore1 [2020-01-28 09:47:45.976186] E [MSGID: 106118] [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to acquire lock for vmstore1 [2020-01-28 09:48:24.976568] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f61c7c07adb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f61c79ae7e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f61c79ae8fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f61c79af987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f61c79b0518] ))))) 0-management: forced unwinding frame type(glusterd mgmt v3) op(--(1)) called at 2020-01-28 09:44:45.743365 (xid=0x52) [2020-01-28 09:48:24.976640] E [MSGID: 106115] [glusterd-mgmt.c:117:gd_mgmt_v3_collate_errors] 0-management: Locking failed on 10.13.250.11. Please check log file for details. [2020-01-28 09:48:24.976743] E [MSGID: 106150] [glusterd-syncop.c:1931:gd_sync_task_begin] 0-management: Locking Peers Failed. [2020-01-28 09:48:24.977361] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f61c7c07adb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f61c79ae7e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f61c79ae8fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f61c79af987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f61c79b0518] ))))) 0-management: forced unwinding frame type(glusterd mgmt) op(--(3)) called at 2020-01-28 09:46:03.586874 (xid=0x53) [2020-01-28 09:48:24.977417] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Staging failed on 10.13.250.11. Please check log file for details. [2020-01-28 09:48:24.977631] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, ProgVers: 2, Proc: 22) to rpc-transport (socket.management) [2020-01-28 09:48:24.977664] E [MSGID: 106430] [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed [2020-01-28 09:48:24.977744] E [MSGID: 106115] [glusterd-mgmt.c:117:gd_mgmt_v3_collate_errors] 0-management: Unlocking failed on 10.13.250.13. Please check log file for details. [2020-01-28 09:48:24.977861] E [MSGID: 106115] [glusterd-mgmt.c:117:gd_mgmt_v3_collate_errors] 0-management: Unlocking failed on 10.13.250.15. Please check log file for details. [2020-01-28 09:48:25.012037] E [MSGID: 106115] [glusterd-mgmt.c:117:gd_mgmt_v3_collate_errors] 0-management: Unlocking failed on 10.13.250.14. Please check log file for details. [2020-01-28 09:48:25.012148] E [MSGID: 106151] [glusterd-syncop.c:1616:gd_unlock_op_phase] 0-management: Failed to unlock on some peer(s) [2020-01-28 09:48:25.012315] E [MSGID: 106117] [glusterd-syncop.c:1640:gd_unlock_op_phase] 0-management: Unable to release lock for vmstore1 [2020-01-28 09:48:25.012425] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, ProgVers: 2, Proc: 27) to rpc-transport (socket.management) [2020-01-28 09:48:25.012452] E [MSGID: 106430] [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed [2020-01-28 09:50:46.105492] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol: xlator_api [2020-01-28 10:13:16.199982] E [MSGID: 106118] [glusterd-op-sm.c:4133:glusterd_op_ac_lock] 0-management: Unable to acquire lock for vmstore1 [2020-01-28 10:13:16.200078] E [MSGID: 106376] [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1 [2020-01-28 10:13:16.345918] E [MSGID: 106117] [glusterd-op-sm.c:4189:glusterd_op_ac_unlock] 0-management: Unable to release lock for vmstore1 [2020-01-28 10:13:16.346012] E [MSGID: 106376] [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1 [2020-01-28 12:42:46.212783] E [MSGID: 106118] [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to acquire lock for vmstore1 [2020-01-28 13:25:49.857368] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol: xlator_api [2020-01-29 13:26:03.031179] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol: xlator_api [2020-01-30 12:33:18.590063] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol: xlator_api [2020-01-30 13:34:57.027468] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol: xlator_api [2020-01-30 15:08:10.814931] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol: xlator_api [2020-01-31 09:40:22.725825] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol: xlator_api
I 'm curious for your opinion of the data sofar.
Love to here from you (or ather members of the list)
Regards Mark Lamers
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VUBWMDUN3GMKGO...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PF2ONI7EHIXWXR...
------------------------------------------------------------------------------ De informatie opgenomen in dit bericht kan vertrouwelijk zijn en is uitsluitend bestemd voor de geadresseerde. Indien u dit bericht onterecht ontvangt, wordt u verzocht de inhoud niet te gebruiken en de afzender direct te informeren door het bericht te retourneren. Het Universitair Medisch Centrum Utrecht is een publiekrechtelijke rechtspersoon in de zin van de W.H.W. (Wet Hoger Onderwijs en Wetenschappelijk Onderzoek) en staat geregistreerd bij de Kamer van Koophandel voor Midden-Nederland onder nr. 30244197. Denk s.v.p aan het milieu voor u deze e-mail afdrukt. ------------------------------------------------------------------------------ This message may contain confidential information and is intended exclusively for the addressee. If you receive this message unintentionally, please do not use the contents but notify the sender immediately by return e-mail. University Medical Center Utrecht is a legal person by public law and is registered at the Chamber of Commerce for Midden-Nederland under no. 30244197. Please consider the environment before printing this e-mail.