Hi Pavel,
Thanks for the report. Can you begin with a more accurate description of your environment?Begin with host, oVirt and Gluster versions. Then continue with the exact setup (what are 'A', 'B', 'C' - domains? Volumes? What is the mapping between domains and volumes?).
Are there any logs you can share with us?
I'm sure with more information, we'd be happy to look at the issue.Y.
On Thu, Mar 31, 2016 at 3:09 PM, paf1(a= )email.cz <paf1(a)emai= l.cz> wrote:
Hello,
we tried the=C2=A0 following test - with unwanted results
input:
5 node gluster
A =3D replica 3 with arbiter 1 ( node1+node2+arbiter on node 5 )
B =3D replica 3 with arbiter 1 ( node3+node4+arbiter on node 5 )
C =3D distributed replica 3 arbiter 1=C2=A0 ( node1+node2, node3+node4, each arbiter on node 5)
node 5 has only arbiter replica ( 4x )
TEST:
1)=C2=A0 directly reboot one node - OK ( is not important whi= ch ( data node or arbiter node ))
2)=C2=A0 directly reboot two nodes - OK ( if=C2=A0 nodes are = not from the same replica )
3)=C2=A0 directly reboot three nodes - yes, this is the main problem and a questions ....
=C2=A0=C2=A0=C2=A0 - rebooted all three nodes from replica "B= "=C2=A0 ( not so possible, but who knows ... )
=C2=A0=C2=A0=C2=A0 - all VMs with data on this replica was pa= used ( no data access ) - OK
=C2=A0=C2=A0=C2=A0 - all VMs running on replica "B" nodes los= t (=C2=A0 started manually, later )( datas on other replicas ) - acceptable
BUT
=C2=A0=C2=A0=C2=A0 - !!! all oVIrt domains went down !! - mas= ter domain is on replica "A" which lost only one member from three !!!
=C2=A0=C2=A0=C2=A0 so we are not expecting that all domain wi= ll go down, especially master with 2 live members.
=C2=A0=C2=A0=C2=A0
Results:
=C2=A0=C2=A0=C2=A0 - the whole cluster unreachable until at a= ll domains up - depent of all nodes up !!!
=C2=A0=C2=A0=C2=A0 - all paused VMs started back - OK
=C2=A0=C2=A0=C2=A0 - rest of all VMs rebooted and runnig - OK=
Questions:
=C2=A0=C2=A0=C2=A0 1) why all domains down if master domain (= on replica "A" ) has two runnig members ( 2 of 3 )=C2=A0 ??
=C2=A0=C2=A0=C2=A0 2) how to fix that colaps without waiting = to all nodes up ? ( in worste case if node has HW error eg. ) ??
=C2=A0=C2=A0=C2=A0 3) which oVirt=C2=A0 cluster=C2=A0 policy= =C2=A0 can prevent that situation ?? ( if any )
regs.
Pavel
_______________________________________________
Users mailing list
U= sers(a)ovirt.org
http://lists.ovirt.org/m= ailman/listinfo/users
Hi Pavel,
Thanks for the report. Can you begin with a more accurate description of your environment?Begin with host, oVirt and Gluster versions. Then continue with the exact setup (what are 'A', 'B', 'C' - domains? Volumes? What is the mapping between domains and volumes?).
Are there any logs you can share with us?
I'm sure with more information, we'd be happy to look at the issue.Y.
On Thu, Mar 31, 2016 at 3:09 PM, paf1(a= )email.cz <paf1(a)emai= l.cz> wrote:
Hello,
we tried the=C2=A0 following test - with unwanted results
input:
5 node gluster
A =3D replica 3 with arbiter 1 ( node1+node2+arbiter on node 5 )
B =3D replica 3 with arbiter 1 ( node3+node4+arbiter on node 5 )
C =3D distributed replica 3 arbiter 1=C2=A0 ( node1+node2, node3+node4, each arbiter on node 5)
node 5 has only arbiter replica ( 4x )
TEST:
1)=C2=A0 directly reboot one node - OK ( is not important whi= ch ( data node or arbiter node ))
2)=C2=A0 directly reboot two nodes - OK ( if=C2=A0 nodes are = not from the same replica )
3)=C2=A0 directly reboot three nodes - yes, this is the main problem and a questions ....
=C2=A0=C2=A0=C2=A0 - rebooted all three nodes from replica "B= "=C2=A0 ( not so possible, but who knows ... )
=C2=A0=C2=A0=C2=A0 - all VMs with data on this replica was pa= used ( no data access ) - OK
=C2=A0=C2=A0=C2=A0 - all VMs running on replica "B" nodes los= t (=C2=A0 started manually, later )( datas on other replicas ) - acceptable
BUT
=C2=A0=C2=A0=C2=A0 - !!! all oVIrt domains went down !! - mas= ter domain is on replica "A" which lost only one member from three !!!
=C2=A0=C2=A0=C2=A0 so we are not expecting that all domain wi= ll go down, especially master with 2 live members.
=C2=A0=C2=A0=C2=A0
Results:
=C2=A0=C2=A0=C2=A0 - the whole cluster unreachable until at a= ll domains up - depent of all nodes up !!!
=C2=A0=C2=A0=C2=A0 - all paused VMs started back - OK
=C2=A0=C2=A0=C2=A0 - rest of all VMs rebooted and runnig - OK=
Questions:
=C2=A0=C2=A0=C2=A0 1) why all domains down if master domain (= on replica "A" ) has two runnig members ( 2 of 3 )=C2=A0 ??
=C2=A0=C2=A0=C2=A0 2) how to fix that colaps without waiting = to all nodes up ? ( in worste case if node has HW error eg. ) ??
=C2=A0=C2=A0=C2=A0 3) which oVirt=C2=A0 cluster=C2=A0 policy= =C2=A0 can prevent that situation ?? ( if any )
regs.
Pavel
_______________________________________________
Users mailing list
U= sers(a)ovirt.org
http://lists.ovirt.org/m= ailman/listinfo/users
Hi Pavel,
Thanks for the report. Can you begin with a more accurate description of your environment?Begin with host, oVirt and Gluster versions. Then continue with the exact setup (what are 'A', 'B', 'C' - domains? Volumes? What is the mapping between domains and volumes?).
Are there any logs you can share with us?
I'm sure with more information, we'd be happy to look at the issue.Y.
On Thu, Mar 31, 2016 at 3:09 PM, paf1(a= )email.cz <paf1(a)emai= l.cz> wrote:
Hello,
we tried the=C2=A0 following test - with unwanted results
input:
5 node gluster
A =3D replica 3 with arbiter 1 ( node1+node2+arbiter on node 5 )
B =3D replica 3 with arbiter 1 ( node3+node4+arbiter on node 5 )
C =3D distributed replica 3 arbiter 1=C2=A0 ( node1+node2, node3+node4, each arbiter on node 5)
node 5 has only arbiter replica ( 4x )
TEST:
1)=C2=A0 directly reboot one node - OK ( is not important whi= ch ( data node or arbiter node ))
2)=C2=A0 directly reboot two nodes - OK ( if=C2=A0 nodes are = not from the same replica )
3)=C2=A0 directly reboot three nodes - yes, this is the main problem and a questions ....
=C2=A0=C2=A0=C2=A0 - rebooted all three nodes from replica "B= "=C2=A0 ( not so possible, but who knows ... )
=C2=A0=C2=A0=C2=A0 - all VMs with data on this replica was pa= used ( no data access ) - OK
=C2=A0=C2=A0=C2=A0 - all VMs running on replica "B" nodes los= t (=C2=A0 started manually, later )( datas on other replicas ) - acceptable
BUT
=C2=A0=C2=A0=C2=A0 - !!! all oVIrt domains went down !! - mas= ter domain is on replica "A" which lost only one member from three !!!
=C2=A0=C2=A0=C2=A0 so we are not expecting that all domain wi= ll go down, especially master with 2 live members.
=C2=A0=C2=A0=C2=A0
Results:
=C2=A0=C2=A0=C2=A0 - the whole cluster unreachable until at a= ll domains up - depent of all nodes up !!!
=C2=A0=C2=A0=C2=A0 - all paused VMs started back - OK
=C2=A0=C2=A0=C2=A0 - rest of all VMs rebooted and runnig - OK=
Questions:
=C2=A0=C2=A0=C2=A0 1) why all domains down if master domain (= on replica "A" ) has two runnig members ( 2 of 3 )=C2=A0 ??
=C2=A0=C2=A0=C2=A0 2) how to fix that colaps without waiting = to all nodes up ? ( in worste case if node has HW error eg. ) ??
=C2=A0=C2=A0=C2=A0 3) which oVirt=C2=A0 cluster=C2=A0 policy= =C2=A0 can prevent that situation ?? ( if any )
regs.
Pavel
_______________________________________________
Users mailing list
U= sers(a)ovirt.org
http://lists.ovirt.org/m= ailman/listinfo/users
Hi,
rest of logs:
www.uschovna.cz/en/za= silka/HYGXR57CNHM3TP39-L3W
The TEST is the last big event in logs ....
TEST TIME : about 14:00-14:30=C2=A0 CET