mirror of
https://github.com/etcd-io/etcd.git
synced 2024-09-27 06:25:44 +00:00
raft: fix auto-transitioning out of joint config
The code doing so was undertested and buggy: it would launch multiple attempts to transition out when the conf change was not the last element in the log. This commit fixes the problem and adds a regression test. It also reworks the code to handle a former untested edge case, in which the auto-transition append is refused. This can't happen any more with the current version of the code because this proposal has size zero and is special cased in increaseUncommittedSize. Last but not least, the auto-leave proposal now also bumps pendingConfIndex, which was not done previously due to an oversight.
This commit is contained in:
Tobias Schottdorf
210
raft/testdata/confchange_v2_add_double_auto.txt
vendored
210
raft/testdata/confchange_v2_add_double_auto.txt
vendored
@@ -195,3 +195,213 @@ stabilize 1 3
|
||||
stabilize
|
||||
----
|
||||
ok
|
||||
|
||||
# Now remove two nodes. What's new here is that the leader will actually have
|
||||
# to go to a quorum to commit the transition into the joint config.
|
||||
|
||||
propose-conf-change 1
|
||||
r2 r3
|
||||
----
|
||||
ok
|
||||
|
||||
# n1 sends out MsgApps.
|
||||
stabilize 1
|
||||
----
|
||||
> 1 handling Ready
|
||||
Ready MustSync=true:
|
||||
Entries:
|
||||
1/6 EntryConfChangeV2 r2 r3
|
||||
Messages:
|
||||
1->2 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
|
||||
1->3 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
|
||||
|
||||
# n2, n3 ack them.
|
||||
stabilize 2 3
|
||||
----
|
||||
> 2 receiving messages
|
||||
1->2 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
|
||||
> 3 receiving messages
|
||||
1->3 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
|
||||
> 2 handling Ready
|
||||
Ready MustSync=true:
|
||||
Entries:
|
||||
1/6 EntryConfChangeV2 r2 r3
|
||||
Messages:
|
||||
2->1 MsgAppResp Term:1 Log:0/6
|
||||
> 3 handling Ready
|
||||
Ready MustSync=true:
|
||||
Entries:
|
||||
1/6 EntryConfChangeV2 r2 r3
|
||||
Messages:
|
||||
3->1 MsgAppResp Term:1 Log:0/6
|
||||
|
||||
# n1 gets some more proposals. This is part of a regression test: There used to
|
||||
# be a bug in which these proposals would prompt the leader to transition out of
|
||||
# the same joint state multiple times, which would cause a panic.
|
||||
propose 1 foo
|
||||
----
|
||||
ok
|
||||
|
||||
propose 1 bar
|
||||
----
|
||||
ok
|
||||
|
||||
# n1 switches to the joint config, then initiates a transition into the final
|
||||
# config.
|
||||
stabilize 1
|
||||
----
|
||||
> 1 handling Ready
|
||||
Ready MustSync=true:
|
||||
Entries:
|
||||
1/7 EntryNormal "foo"
|
||||
1/8 EntryNormal "bar"
|
||||
Messages:
|
||||
1->2 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
|
||||
1->3 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
|
||||
1->2 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
|
||||
1->3 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
|
||||
> 1 receiving messages
|
||||
2->1 MsgAppResp Term:1 Log:0/6
|
||||
3->1 MsgAppResp Term:1 Log:0/6
|
||||
> 1 handling Ready
|
||||
Ready MustSync=false:
|
||||
HardState Term:1 Vote:1 Commit:6
|
||||
CommittedEntries:
|
||||
1/6 EntryConfChangeV2 r2 r3
|
||||
Messages:
|
||||
1->2 MsgApp Term:1 Log:1/8 Commit:6
|
||||
1->3 MsgApp Term:1 Log:1/8 Commit:6
|
||||
INFO 1 switched to configuration voters=(1)&&(1 2 3) autoleave
|
||||
INFO initiating automatic transition out of joint configuration voters=(1)&&(1 2 3) autoleave
|
||||
> 1 handling Ready
|
||||
Ready MustSync=true:
|
||||
Entries:
|
||||
1/9 EntryConfChangeV2
|
||||
|
||||
# n2 and n3 also switch to the joint config, and ack the transition out of it.
|
||||
stabilize 2 3
|
||||
----
|
||||
> 2 receiving messages
|
||||
1->2 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
|
||||
1->2 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
|
||||
1->2 MsgApp Term:1 Log:1/8 Commit:6
|
||||
> 3 receiving messages
|
||||
1->3 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
|
||||
1->3 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
|
||||
1->3 MsgApp Term:1 Log:1/8 Commit:6
|
||||
> 2 handling Ready
|
||||
Ready MustSync=true:
|
||||
HardState Term:1 Commit:6
|
||||
Entries:
|
||||
1/7 EntryNormal "foo"
|
||||
1/8 EntryNormal "bar"
|
||||
CommittedEntries:
|
||||
1/6 EntryConfChangeV2 r2 r3
|
||||
Messages:
|
||||
2->1 MsgAppResp Term:1 Log:0/7
|
||||
2->1 MsgAppResp Term:1 Log:0/8
|
||||
2->1 MsgAppResp Term:1 Log:0/8
|
||||
INFO 2 switched to configuration voters=(1)&&(1 2 3) autoleave
|
||||
> 3 handling Ready
|
||||
Ready MustSync=true:
|
||||
HardState Term:1 Commit:6
|
||||
Entries:
|
||||
1/7 EntryNormal "foo"
|
||||
1/8 EntryNormal "bar"
|
||||
CommittedEntries:
|
||||
1/6 EntryConfChangeV2 r2 r3
|
||||
Messages:
|
||||
3->1 MsgAppResp Term:1 Log:0/7
|
||||
3->1 MsgAppResp Term:1 Log:0/8
|
||||
3->1 MsgAppResp Term:1 Log:0/8
|
||||
INFO 3 switched to configuration voters=(1)&&(1 2 3) autoleave
|
||||
|
||||
# n2 and n3 also leave the joint config and the dust settles. We see at the very
|
||||
# end that n1 receives some messages from them that it refuses because it does
|
||||
# not have them in its config any more.
|
||||
stabilize
|
||||
----
|
||||
> 1 receiving messages
|
||||
2->1 MsgAppResp Term:1 Log:0/7
|
||||
2->1 MsgAppResp Term:1 Log:0/8
|
||||
2->1 MsgAppResp Term:1 Log:0/8
|
||||
3->1 MsgAppResp Term:1 Log:0/7
|
||||
3->1 MsgAppResp Term:1 Log:0/8
|
||||
3->1 MsgAppResp Term:1 Log:0/8
|
||||
> 1 handling Ready
|
||||
Ready MustSync=false:
|
||||
HardState Term:1 Vote:1 Commit:8
|
||||
CommittedEntries:
|
||||
1/7 EntryNormal "foo"
|
||||
1/8 EntryNormal "bar"
|
||||
Messages:
|
||||
1->2 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
|
||||
1->3 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
|
||||
1->2 MsgApp Term:1 Log:1/9 Commit:8
|
||||
1->3 MsgApp Term:1 Log:1/9 Commit:8
|
||||
> 2 receiving messages
|
||||
1->2 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
|
||||
1->2 MsgApp Term:1 Log:1/9 Commit:8
|
||||
> 3 receiving messages
|
||||
1->3 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
|
||||
1->3 MsgApp Term:1 Log:1/9 Commit:8
|
||||
> 2 handling Ready
|
||||
Ready MustSync=true:
|
||||
HardState Term:1 Commit:8
|
||||
Entries:
|
||||
1/9 EntryConfChangeV2
|
||||
CommittedEntries:
|
||||
1/7 EntryNormal "foo"
|
||||
1/8 EntryNormal "bar"
|
||||
Messages:
|
||||
2->1 MsgAppResp Term:1 Log:0/9
|
||||
2->1 MsgAppResp Term:1 Log:0/9
|
||||
> 3 handling Ready
|
||||
Ready MustSync=true:
|
||||
HardState Term:1 Commit:8
|
||||
Entries:
|
||||
1/9 EntryConfChangeV2
|
||||
CommittedEntries:
|
||||
1/7 EntryNormal "foo"
|
||||
1/8 EntryNormal "bar"
|
||||
Messages:
|
||||
3->1 MsgAppResp Term:1 Log:0/9
|
||||
3->1 MsgAppResp Term:1 Log:0/9
|
||||
> 1 receiving messages
|
||||
2->1 MsgAppResp Term:1 Log:0/9
|
||||
2->1 MsgAppResp Term:1 Log:0/9
|
||||
3->1 MsgAppResp Term:1 Log:0/9
|
||||
3->1 MsgAppResp Term:1 Log:0/9
|
||||
> 1 handling Ready
|
||||
Ready MustSync=false:
|
||||
HardState Term:1 Vote:1 Commit:9
|
||||
CommittedEntries:
|
||||
1/9 EntryConfChangeV2
|
||||
Messages:
|
||||
1->2 MsgApp Term:1 Log:1/9 Commit:9
|
||||
1->3 MsgApp Term:1 Log:1/9 Commit:9
|
||||
INFO 1 switched to configuration voters=(1)
|
||||
> 2 receiving messages
|
||||
1->2 MsgApp Term:1 Log:1/9 Commit:9
|
||||
> 3 receiving messages
|
||||
1->3 MsgApp Term:1 Log:1/9 Commit:9
|
||||
> 2 handling Ready
|
||||
Ready MustSync=false:
|
||||
HardState Term:1 Commit:9
|
||||
CommittedEntries:
|
||||
1/9 EntryConfChangeV2
|
||||
Messages:
|
||||
2->1 MsgAppResp Term:1 Log:0/9
|
||||
INFO 2 switched to configuration voters=(1)
|
||||
> 3 handling Ready
|
||||
Ready MustSync=false:
|
||||
HardState Term:1 Commit:9
|
||||
CommittedEntries:
|
||||
1/9 EntryConfChangeV2
|
||||
Messages:
|
||||
3->1 MsgAppResp Term:1 Log:0/9
|
||||
INFO 3 switched to configuration voters=(1)
|
||||
> 1 receiving messages
|
||||
2->1 MsgAppResp Term:1 Log:0/9
|
||||
raft: cannot step as peer not found3->1 MsgAppResp Term:1 Log:0/9
|
||||
raft: cannot step as peer not found
|
||||
|
||||
Reference in New Issue
Block a user