3:26 a.m.
Hi,
I am currently upgrading my oVirt setup from 4.2.8 to 4.3.3.1.
The setup consists of:
Datacenter/Cluster Default: [fully upgraded to 4.3.3.1]
2 nodes (node04,node05)- NFS storage domain with self hosted engine
Datacenter Luise:
Cluster1: 3 nodes (node01,node02,node03) - Node NG with GlusterFS - Ceph Cinder storage
domain
[Node1 and Node3 are upgraded to 4.3.3.1, Node2 is on 4.2.8]
Cluster2: 1 node (node06) - only Ceph Cinder storage domain [fully upgraded to
4.3.3.1]
Problems started when upgrading Luise/Cluster1 with GlusterFS:
(I always waited for GlusterFS to be fully synced before proceeding to the next step)
- Upgrade node01 to 4.3.3 -> OK
- Upgrade node03 to 4.3.3.1 -> OK
- Upgrade node01 to 4.3.3.1 -> GlusterFS became unstable.
I now get the error message:
VDSM node03.infra.solutions.work command ConnectStoragePoolVDS failed: Cannot find master
domain: u'spUUID=f3218bf7-6158-4b2b-b272-51cdc3280376,
msdUUID=02a32017-cbe6-4407-b825-4e558b784157'
And on node03 there is a problem with Gluster:
node03#: ls -l /rhev/data-center/mnt/glusterSD/node01.infra.solutions.work:_vmstore
ls: cannot access /rhev/data-center/mnt/glusterSD/node01.infra.solutions.work:_vmstore:
Transport endpoint is not connected
The directory is available on node01 and node02.
The engine is reporting the brick on node03 as down. Node03 and Node06 are shown as
NonOperational, because they are not able to access the gluster storage domain.
A “gluster peer status” on node1, node2, and node3 shows all peers connected.
“gluster volume heal vmstore info” shows for all nodes:
gluster volume heal vmstore info
Brick node01.infra.solutions.work:/gluster_bricks/vmstore/vmstore
Status: Transport endpoint is not connected
Number of entries: -
Brick node02.infra.solutions.work:/gluster_bricks/vmstore/vmstore
<gfid:0bcb7825-e649-4178-a899-c5cc04c95286>
<gfid:71ec8035-f5a5-4e61-bb34-5ad9db28c0eb>
<gfid:16d5961e-c3bb-4493-a51d-bf83074c4cc7>
/02a32017-cbe6-4407-b825-4e558b784157/dom_md/ids
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.66
<gfid:5fe350e4-1eb5-4b6f-a3fb-42c98b7b2f8d>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.60
/02a32017-cbe6-4407-b825-4e558b784157/images/a3a10398-9698-4b73-84d9-9735448e3534/6161e310-4ad6-42d9-8117-5a89c5b2b4b6
<gfid:8eb9fd30-fdb9-442b-9c54-8ba256d7981b>
<gfid:c72001be-e7d3-4b34-bac5-9ab50b609eea>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.96
<gfid:447ec09b-336e-4d2b-8338-f31329ee7a55>
<gfid:9d7db516-d6fb-43d8-a069-dcbc1d72e62a>
/.shard/d66880de-3fa1-4362-8c43-574a173c5f7d.133
<gfid:2412d449-d3ed-40ef-b7eb-d81bdf7c5c05>
<gfid:0fae358b-2cdd-4064-b63c-7f31a35bc35a>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.38
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.67
/__DIRECT_IO_TEST__
<gfid:a7945526-9ff3-40fe-b1e2-3921117ef738>
<gfid:e78e9c1f-ce6b-4871-b5bf-9bde34685b99>
/02a32017-cbe6-4407-b825-4e558b784157/images/493188b2-c137-4440-99ee-43a753842a7d/9aa2d139-e3bd-406b-8fe0-b189123eaa73
<gfid:3aed3fb6-044a-4371-9302-e0bd54cbd794>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.64
/.shard/d66880de-3fa1-4362-8c43-574a173c5f7d.132
<gfid:f7631be7-2ab5-4985-904d-69174c0e1267>
<gfid:43001625-1aad-4032-a76e-4cc2a51de2b3>
<gfid:6ae3fe7f-15c9-4103-960c-faba0ba59cb3>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.44
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.9
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.69
<gfid:f20cc6f7-9391-4260-9238-3e1d0cabbfa3>
/02a32017-cbe6-4407-b825-4e558b784157/images/12e647fb-20aa-4957-b659-05fa75a9215e/f7e4b2a3-ab84-4eb5-a4e7-7208ddad8156
<gfid:c540368a-4431-4405-9a59-e11a217d0ea6>
<gfid:4e698a74-39dc-40a3-ac9c-14456420ab66>
<gfid:afd48e71-ff23-42d7-aef4-b2e2167b75e8>
<gfid:194589b3-0760-4150-80ef-d87376813835>
<gfid:6e17ead1-88cc-4e3e-84fa-7495c4fc3a0e>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.35
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.32
<gfid:982d071f-2081-4371-b007-bc48d8167e7c>
<gfid:e2285905-a8da-44f5-8c56-1f8a4d6326a8>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.39
<gfid:956c7d9c-2f96-42d1-bf6e-57bc9e534f84>
<gfid:16162bc7-201a-4842-a41a-af2cc4fb8a9e>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.34
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.68
Status: Connected
Number of entries: 47
Brick node03.infra.solutions.work:/gluster_bricks/vmstore/vmstore
/02a32017-cbe6-4407-b825-4e558b784157/images/12e647fb-20aa-4957-b659-05fa75a9215e/f7e4b2a3-ab84-4eb5-a4e7-7208ddad8156
<gfid:c540368a-4431-4405-9a59-e11a217d0ea6>
<gfid:4e698a74-39dc-40a3-ac9c-14456420ab66>
<gfid:afd48e71-ff23-42d7-aef4-b2e2167b75e8>
<gfid:194589b3-0760-4150-80ef-d87376813835>
<gfid:6e17ead1-88cc-4e3e-84fa-7495c4fc3a0e>
<gfid:099284a6-9538-4f9a-928a-d9b704fe0735>
<gfid:75d3c8f7-d67a-49a4-9cd4-7fff202df40d>
<gfid:982d071f-2081-4371-b007-bc48d8167e7c>
<gfid:e2285905-a8da-44f5-8c56-1f8a4d6326a8>
<gfid:447ec09b-336e-4d2b-8338-f31329ee7a55>
<gfid:956c7d9c-2f96-42d1-bf6e-57bc9e534f84>
/.shard/d66880de-3fa1-4362-8c43-574a173c5f7d.133
<gfid:43001625-1aad-4032-a76e-4cc2a51de2b3>
<gfid:6ae3fe7f-15c9-4103-960c-faba0ba59cb3>
<gfid:1a0b2737-9172-4c51-aa77-e93e9671840c>
<gfid:eb471e13-6749-4f62-b1f5-15a44f8990c2>
<gfid:a7945526-9ff3-40fe-b1e2-3921117ef738>
<gfid:e78e9c1f-ce6b-4871-b5bf-9bde34685b99>
/02a32017-cbe6-4407-b825-4e558b784157/images/493188b2-c137-4440-99ee-43a753842a7d/9aa2d139-e3bd-406b-8fe0-b189123eaa73
<gfid:6b418e80-9f61-4d6e-ba77-8a1969d9a99b>
<gfid:914c72d2-e45e-48f2-b7ef-5846b13f7a91>
<gfid:2bd28bdb-1dc6-41d5-96be-c696f452e3f2>
<gfid:9d7db516-d6fb-43d8-a069-dcbc1d72e62a>
<gfid:f7631be7-2ab5-4985-904d-69174c0e1267>
<gfid:16162bc7-201a-4842-a41a-af2cc4fb8a9e>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.44
<gfid:afc6d611-528d-441b-b74e-c5fae6672088>
<gfid:707308e9-e8e5-487a-b0f1-a816720c4243>
<gfid:5f1a81a7-7c42-4226-9142-1b5b35c2b1e9>
<gfid:f20cc6f7-9391-4260-9238-3e1d0cabbfa3>
<gfid:0bcb7825-e649-4178-a899-c5cc04c95286>
<gfid:71ec8035-f5a5-4e61-bb34-5ad9db28c0eb>
<gfid:16d5961e-c3bb-4493-a51d-bf83074c4cc7>
/02a32017-cbe6-4407-b825-4e558b784157/dom_md/ids
<gfid:7661180b-1917-4a7b-9749-5dfb826c4449>
<gfid:5fe350e4-1eb5-4b6f-a3fb-42c98b7b2f8d>
<gfid:a6197593-7e09-4d3f-b538-9cd1ebadd6c9>
/02a32017-cbe6-4407-b825-4e558b784157/images/a3a10398-9698-4b73-84d9-9735448e3534/6161e310-4ad6-42d9-8117-5a89c5b2b4b6
<gfid:8eb9fd30-fdb9-442b-9c54-8ba256d7981b>
<gfid:c72001be-e7d3-4b34-bac5-9ab50b609eea>
<gfid:3aed3fb6-044a-4371-9302-e0bd54cbd794>
/.shard/d66880de-3fa1-4362-8c43-574a173c5f7d.132
<gfid:2412d449-d3ed-40ef-b7eb-d81bdf7c5c05>
<gfid:0fae358b-2cdd-4064-b63c-7f31a35bc35a>
<gfid:c0ca2784-a8af-44b3-9091-a1eaf4c8676f>
/__DIRECT_IO_TEST__
Status: Connected
Number of entries: 47
On Node03 there are several self healing processes, that seem to be doing nothing.
Oh well.. What now?
Best regards,
- Andreas