Did you manage to fix the issue with the 0-xlator ? If yes, most probably the issue will be OK. Yet 'probably' doesn't meant that they 'will' be OK.
If it was my lab - I would go ahead only if the 0-xlator issue is over.Yet, a lab is a different thing than prod - so it is your sole decision. 

Did you test the upgrade prior moving to Prod ?

About the hooks - I had such issue before and I had to reinstall the gluster rpms to solve it.


Best Regards,
Strahil Nikolov

В вторник, 4 февруари 2020 г., 16:35:27 ч. Гринуич+2, Goorkate, B.J. <b.j.goorkate@umcutrecht.nl> написа:


Hi Strahil,

Thanks for your time so far!

The packages seem fine on all of the 3 nodes.
Only /var/lib/glusterd/glusterd.info is modified and on the not yet upgraded nodes these files are missing:

missing    /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh
missing    /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh
missing    /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh
missing    /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh
missing    /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh

But that doesn't seem relevant...

I stopped a couple of virtual machines with image files with unsynched entries. When they were turned off, I couldn't find them anymore in the unsyced entries list. When I turned them on again, some re-appeared, some didn't.

I really don't know where to look next.

The big question is: will the problems be resolved when I upgrade the two remaining nodes, or will it get worse?

Regards,

Bertjan


On Sun, Feb 02, 2020 at 08:06:58PM +0000, Strahil Nikolov wrote:
>    It seems that something went "off".
>    That '0-xlator: dlsym(xlator_api) missing:
>    /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol:
>    xlator_api' is really worrisome.
>    I think that you might have a bad package and based on the info it could
>    be glusterfs-libs which should provide /usr/lib64/libgfrpc.so.0 . I'm
>    currently on gluster v7.0 and I can't check it on my installation.
>    Run a for loop to check the rpms:
>    for i in $(rpm -qa | grep gluster) ; do echo "$i :" ; rpm -V $i';
>    echo;echo; done
>    Most probably you can safely redo (on the upgraded node) the last yum
>    transaction:
>    yum history
>    yum history info <ID> -> verify the gluster packages were installed here
>    yum history redo <ID>
>    If kernel, glibc, systemd were not in this transaction , you can stop
>    gluster and start it again:
>    Node in maintenance in oVirt
>    systemctl stop ovirt-ha-agent ovirt-ha-broker vdsmd supervdsmd
>    systemctl stop sanlock
>    systemctl stop glusterd
>    /usr/share/glusterfs/scripts/stop-all-gluster-processes.sh
>    And then power up again:
>    systemctl start glusterd
>    gluster volume status -> check all connected
>    systemctl start sanlock supervdsmd vdsmd
>    systemctl start ovirt-ha-broker ovirt-ha-agent
>    Check situation.
>    Yet, you need to make gluster stop complaining , before you can take care
>    of the heal. Usually 'rsync' is my best friend - but this is when gluster
>    is working normally - and your case is far from normal.
>    If redo doesn't work for you -> try the "yum history rollback" to recover
>    to last good state.
>    I think that 'BOOM Boot Manager' is best in such cases.
>    Note: Never take any of my words for granted. I'm not running oVirt in
>    production and some of my methods might not be OK for your environment.
>    Best Regards,
>    Strahil Nikolov
>    В неделя, 2 февруари 2020 г., 08:56:01 ч. Гринуич+2, Mark Lamers
>    <mark.r.lamers@gmail.com> написа:
>
>    Hi Strahil,
>
>    Bertjan is not in the office today, so I will reply if okay with you.
>
>    First I like to describe the status of our network
>
>    There are three bricks:
>
>    and 5 nodes:
>
>    Every host has a management and migrate vlan iface  on a different bond
>    iface.
>
>    The last octet of the ipaddress is similar
>
>    The output from 'gluster volume heal <volname> info' gives a long list of
>    shards from all three nodes, the file is attached as
>    'gluster_volume_heal_info.txt'. The node without shards in the list is the
>    updated node to gluster6, oVirt 4.3. That is curious i think.
>
>    Further more I find the following errors ' E '  in the glusterd.log of the
>    upgrated host:
>
>    [2020-01-26 17:40:46.147730] E [MSGID: 106118]
>    [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to
>    acquire lock for vmstore1
>    [2020-01-26 22:47:16.655651] E [MSGID: 106118]
>    [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to
>    acquire lock for vmstore1
>    [2020-01-27 07:07:51.815490] E [MSGID: 106118]
>    [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to
>    acquire lock for vmstore1
>    [2020-01-27 18:28:14.953974] E [MSGID: 106118]
>    [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to
>    acquire lock for vmstore1
>    [2020-01-27 18:58:22.629457] E [MSGID: 106118]
>    [glusterd-op-sm.c:4133:glusterd_op_ac_lock] 0-management: Unable to
>    acquire lock for vmstore1
>    [2020-01-27 18:58:22.629595] E [MSGID: 106376]
>    [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1
>    [2020-01-27 18:58:22.756430] E [MSGID: 106117]
>    [glusterd-op-sm.c:4189:glusterd_op_ac_unlock] 0-management: Unable to
>    release lock for vmstore1
>    [2020-01-27 18:58:22.756581] E [MSGID: 106376]
>    [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1
>    [2020-01-28 05:31:52.427196] E [MSGID: 106118]
>    [glusterd-op-sm.c:4133:glusterd_op_ac_lock] 0-management: Unable to
>    acquire lock for vmstore1
>    [2020-01-28 05:31:52.427315] E [MSGID: 106376]
>    [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1
>    [2020-01-28 05:31:52.537799] E [MSGID: 106115]
>    [glusterd-mgmt.c:117:gd_mgmt_v3_collate_errors] 0-management: Locking
>    failed on 10.13.250.14. Please check log file for details.
>    [2020-01-28 05:31:52.537973] E [MSGID: 106150]
>    [glusterd-syncop.c:1931:gd_sync_task_begin] 0-management: Locking Peers
>    Failed.
>    [2020-01-28 05:31:52.539620] E [MSGID: 106117]
>    [glusterd-op-sm.c:4189:glusterd_op_ac_unlock] 0-management: Unable to
>    release lock for vmstore1
>    [2020-01-28 05:31:52.539759] E [MSGID: 106376]
>    [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1
>    [2020-01-28 05:31:52.539937] E [MSGID: 106115]
>    [glusterd-mgmt.c:117:gd_mgmt_v3_collate_errors] 0-management: Unlocking
>    failed on 10.13.250.14. Please check log file for details.
>    [2020-01-28 05:31:52.540446] E [MSGID: 106151]
>    [glusterd-syncop.c:1616:gd_unlock_op_phase] 0-management: Failed to unlock
>    on some peer(s)
>    [2020-01-28 08:51:45.638694] E [MSGID: 106118]
>    [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to
>    acquire lock for vmstore1
>    [2020-01-28 08:52:45.709950] E [MSGID: 106118]
>    [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to
>    acquire lock for vmstore1
>    [2020-01-28 08:54:12.455555] E [MSGID: 101097]
>    [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing:
>    /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol:
>    xlator_api
>    [2020-01-28 08:54:16.214779] E [MSGID: 106118]
>    [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to
>    acquire lock for vmstore1
>    [2020-01-28 08:57:59.496842] E [MSGID: 106118]
>    [glusterd-op-sm.c:4133:glusterd_op_ac_lock] 0-management: Unable to
>    acquire lock for vmstore1
>    [2020-01-28 08:57:59.496905] E [MSGID: 106376]
>    [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1
>    [2020-01-28 08:57:59.505119] E [rpcsvc.c:1577:rpcsvc_submit_generic]
>    0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc
>    cli read-only, ProgVers: 2, Proc: 5) to rpc-transport (socket.management)
>    [2020-01-28 08:57:59.505135] E [MSGID: 106430]
>    [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission
>    failed
>    [2020-01-28 08:57:59.647456] E [MSGID: 106117]
>    [glusterd-op-sm.c:4189:glusterd_op_ac_unlock] 0-management: Unable to
>    release lock for vmstore1
>    [2020-01-28 08:57:59.647508] E [MSGID: 106376]
>    [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: 1
>    [2020-01-28 08:57:59.654929] E [rpcsvc.c:1577:rpcsvc_submit_generic]
>    0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc
>    cli, ProgVers: 2, Proc: 27) to rpc-transport (socket.management)
>    [2020-01-28 08:57:59.654943] E [MSGID: 106430]
>    [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission
>    failed
>    [2020-01-28 09:07:34.941350] E [MSGID: 106118]
>    [glusterd-op-sm.c:4133:glusterd_op_ac_lock] 0-management: Unable to
>    acquire lock for vmstore1
>    [2020-01-28 09:07:34.941391] E [MSGID: 106376]
>    [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1
>    [2020-01-28 09:07:35.042466] E [MSGID: 106117]
>    [glusterd-op-sm.c:4189:glusterd_op_ac_unlock] 0-management: Unable to
>    release lock for vmstore1
>    [2020-01-28 09:07:35.042510] E [MSGID: 106376]
>    [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1
>    [2020-01-28 09:13:32.329172] E [MSGID: 106244] [glusterd.c:1785:init]
>    0-management: creation of 1 listeners failed, continuing with succeeded
>    transport
>    [2020-01-28 09:13:44.024431] E [MSGID: 101097]
>    [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing:
>    /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol:
>    xlator_api
>    [2020-01-28 09:45:46.347499] E [MSGID: 106118]
>    [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to
>    acquire lock for vmstore1
>    [2020-01-28 09:46:45.837466] E [MSGID: 106118]
>    [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to
>    acquire lock for vmstore1
>    [2020-01-28 09:47:45.976186] E [MSGID: 106118]
>    [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to
>    acquire lock for vmstore1
>    [2020-01-28 09:48:24.976568] E [rpc-clnt.c:346:saved_frames_unwind] (-->
>    /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f61c7c07adb] (-->
>    /lib64/libgfrpc.so.0(+0xd7e4)[0x7f61c79ae7e4] (-->
>    /lib64/libgfrpc.so.0(+0xd8fe)[0x7f61c79ae8fe] (-->
>    /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f61c79af987]
>    (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f61c79b0518] ))))) 0-management:
>    forced unwinding frame type(glusterd mgmt v3) op(--(1)) called at
>    2020-01-28 09:44:45.743365 (xid=0x52)
>    [2020-01-28 09:48:24.976640] E [MSGID: 106115]
>    [glusterd-mgmt.c:117:gd_mgmt_v3_collate_errors] 0-management: Locking
>    failed on 10.13.250.11. Please check log file for details.
>    [2020-01-28 09:48:24.976743] E [MSGID: 106150]
>    [glusterd-syncop.c:1931:gd_sync_task_begin] 0-management: Locking Peers
>    Failed.
>    [2020-01-28 09:48:24.977361] E [rpc-clnt.c:346:saved_frames_unwind] (-->
>    /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f61c7c07adb] (-->
>    /lib64/libgfrpc.so.0(+0xd7e4)[0x7f61c79ae7e4] (-->
>    /lib64/libgfrpc.so.0(+0xd8fe)[0x7f61c79ae8fe] (-->
>    /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f61c79af987]
>    (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f61c79b0518] ))))) 0-management:
>    forced unwinding frame type(glusterd mgmt) op(--(3)) called at 2020-01-28
>    09:46:03.586874 (xid=0x53)
>    [2020-01-28 09:48:24.977417] E [MSGID: 106152]
>    [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Staging failed on
>    10.13.250.11. Please check log file for details.
>    [2020-01-28 09:48:24.977631] E [rpcsvc.c:1577:rpcsvc_submit_generic]
>    0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc
>    cli, ProgVers: 2, Proc: 22) to rpc-transport (socket.management)
>    [2020-01-28 09:48:24.977664] E [MSGID: 106430]
>    [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission
>    failed
>    [2020-01-28 09:48:24.977744] E [MSGID: 106115]
>    [glusterd-mgmt.c:117:gd_mgmt_v3_collate_errors] 0-management: Unlocking
>    failed on 10.13.250.13. Please check log file for details.
>    [2020-01-28 09:48:24.977861] E [MSGID: 106115]
>    [glusterd-mgmt.c:117:gd_mgmt_v3_collate_errors] 0-management: Unlocking
>    failed on 10.13.250.15. Please check log file for details.
>    [2020-01-28 09:48:25.012037] E [MSGID: 106115]
>    [glusterd-mgmt.c:117:gd_mgmt_v3_collate_errors] 0-management: Unlocking
>    failed on 10.13.250.14. Please check log file for details.
>    [2020-01-28 09:48:25.012148] E [MSGID: 106151]
>    [glusterd-syncop.c:1616:gd_unlock_op_phase] 0-management: Failed to unlock
>    on some peer(s)
>    [2020-01-28 09:48:25.012315] E [MSGID: 106117]
>    [glusterd-syncop.c:1640:gd_unlock_op_phase] 0-management: Unable to
>    release lock for vmstore1
>    [2020-01-28 09:48:25.012425] E [rpcsvc.c:1577:rpcsvc_submit_generic]
>    0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc
>    cli, ProgVers: 2, Proc: 27) to rpc-transport (socket.management)
>    [2020-01-28 09:48:25.012452] E [MSGID: 106430]
>    [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission
>    failed
>    [2020-01-28 09:50:46.105492] E [MSGID: 101097]
>    [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing:
>    /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol:
>    xlator_api
>    [2020-01-28 10:13:16.199982] E [MSGID: 106118]
>    [glusterd-op-sm.c:4133:glusterd_op_ac_lock] 0-management: Unable to
>    acquire lock for vmstore1
>    [2020-01-28 10:13:16.200078] E [MSGID: 106376]
>    [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1
>    [2020-01-28 10:13:16.345918] E [MSGID: 106117]
>    [glusterd-op-sm.c:4189:glusterd_op_ac_unlock] 0-management: Unable to
>    release lock for vmstore1
>    [2020-01-28 10:13:16.346012] E [MSGID: 106376]
>    [glusterd-op-sm.c:8150:glusterd_op_sm] 0-management: handler returned: -1
>    [2020-01-28 12:42:46.212783] E [MSGID: 106118]
>    [glusterd-syncop.c:1896:gd_sync_task_begin] 0-management: Unable to
>    acquire lock for vmstore1
>    [2020-01-28 13:25:49.857368] E [MSGID: 101097]
>    [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing:
>    /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol:
>    xlator_api
>    [2020-01-29 13:26:03.031179] E [MSGID: 101097]
>    [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing:
>    /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol:
>    xlator_api
>    [2020-01-30 12:33:18.590063] E [MSGID: 101097]
>    [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing:
>    /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol:
>    xlator_api
>    [2020-01-30 13:34:57.027468] E [MSGID: 101097]
>    [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing:
>    /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol:
>    xlator_api
>    [2020-01-30 15:08:10.814931] E [MSGID: 101097]
>    [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing:
>    /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol:
>    xlator_api
>    [2020-01-31 09:40:22.725825] E [MSGID: 101097]
>    [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing:
>    /usr/lib64/glusterfs/6.6/rpc-transport/socket.so: undefined symbol:
>    xlator_api
>
>    I 'm curious for your opinion of the data sofar.
>
>    Love to here from you (or ather members of the list)
>
>    Regards Mark Lamers
>
>    _______________________________________________
>    Users mailing list -- users@ovirt.org
>    To unsubscribe send an email to users-leave@ovirt.org
>    Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>    oVirt Code of Conduct:
>    https://www.ovirt.org/community/about/community-guidelines/
>    List Archives:
>    https://lists.ovirt.org/archives/list/users@ovirt.org/message/VUBWMDUN3GMKGOT63J2Y2A5X7VXXW3H3/




> _______________________________________________
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-leave@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PF2ONI7EHIXWXRBCJIZCVCFHYYKCYZSI/


------------------------------------------------------------------------------

De informatie opgenomen in dit bericht kan vertrouwelijk zijn en is
uitsluitend bestemd voor de geadresseerde. Indien u dit bericht onterecht
ontvangt, wordt u verzocht de inhoud niet te gebruiken en de afzender direct
te informeren door het bericht te retourneren. Het Universitair Medisch
Centrum Utrecht is een publiekrechtelijke rechtspersoon in de zin van de W.H.W.
(Wet Hoger Onderwijs en Wetenschappelijk Onderzoek) en staat geregistreerd bij
de Kamer van Koophandel voor Midden-Nederland onder nr. 30244197.

Denk s.v.p aan het milieu voor u deze e-mail afdrukt.

------------------------------------------------------------------------------

This message may contain confidential information and is intended exclusively
for the addressee. If you receive this message unintentionally, please do not
use the contents but notify the sender immediately by return e-mail. University
Medical Center Utrecht is a legal person by public law and is registered at
the Chamber of Commerce for Midden-Nederland under no. 30244197.

Please consider the environment before printing this e-mail.