mirror of
https://github.com/etcd-io/etcd.git
synced 2024-09-27 06:25:44 +00:00
0544f33248
Apps typically maintain the raft configuration as part of the state machine. As a result, they want to be able to reject configuration change entries at apply time based on the state on which the entry is supposed to be applied. When this happens, the app should not call ApplyConfChange, but the comments did not make this clear. As a result, it was tempting to pass an empty pb.ConfChange or it's V2 version instead of not calling ApplyConfChange. However, an empty V1 or V2 proto aren't noops when the configuration is joint: an empty V1 change is treated internally as a single configuration change for NodeID zero and will cause a panic when applied in a joint state. An empty V2 proto is treated as a signal to leave a joint state, which means that the app's config and raft's would diverge. The comments updated in this commit now ask users to not call ApplyConfState when they reject a conf change. Apps that never use joint consensus can keep their old behavior since the distinction only matters when in a joint state, but we don't want to encourage that.
409 lines
14 KiB
Plaintext
409 lines
14 KiB
Plaintext
# Run a V2 membership change that adds two voters at once and auto-leaves the
|
|
# joint configuration. (This is the same as specifying an explicit transition
|
|
# since more than one change is being made atomically).
|
|
|
|
# Bootstrap n1.
|
|
add-nodes 1 voters=(1) index=2
|
|
----
|
|
INFO 1 switched to configuration voters=(1)
|
|
INFO 1 became follower at term 0
|
|
INFO newRaft 1 [peers: [1], term: 0, commit: 2, applied: 2, lastindex: 2, lastterm: 1]
|
|
|
|
campaign 1
|
|
----
|
|
INFO 1 is starting a new election at term 0
|
|
INFO 1 became candidate at term 1
|
|
INFO 1 received MsgVoteResp from 1 at term 1
|
|
INFO 1 became leader at term 1
|
|
|
|
propose-conf-change 1 transition=auto
|
|
v2 v3
|
|
----
|
|
ok
|
|
|
|
# Add two "empty" nodes to the cluster, n2 and n3.
|
|
add-nodes 2
|
|
----
|
|
INFO 2 switched to configuration voters=()
|
|
INFO 2 became follower at term 0
|
|
INFO newRaft 2 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
|
|
INFO 3 switched to configuration voters=()
|
|
INFO 3 became follower at term 0
|
|
INFO newRaft 3 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
|
|
|
|
# n1 immediately gets to commit & apply the conf change using only itself. We see that
|
|
# it starts transitioning out of that joint configuration (though we will only see that
|
|
# proposal in the next ready handling loop, when it is emitted). We also see that this
|
|
# is using joint consensus, which it has to since we're carrying out two additions at
|
|
# once.
|
|
process-ready 1
|
|
----
|
|
Ready MustSync=true:
|
|
Lead:1 State:StateLeader
|
|
HardState Term:1 Vote:1 Commit:4
|
|
Entries:
|
|
1/3 EntryNormal ""
|
|
1/4 EntryConfChangeV2 v2 v3
|
|
CommittedEntries:
|
|
1/3 EntryNormal ""
|
|
1/4 EntryConfChangeV2 v2 v3
|
|
INFO 1 switched to configuration voters=(1 2 3)&&(1) autoleave
|
|
INFO initiating automatic transition out of joint configuration voters=(1 2 3)&&(1) autoleave
|
|
|
|
# n1 immediately probes n2 and n3.
|
|
stabilize 1
|
|
----
|
|
> 1 handling Ready
|
|
Ready MustSync=true:
|
|
Entries:
|
|
1/5 EntryConfChangeV2
|
|
Messages:
|
|
1->2 MsgApp Term:1 Log:1/3 Commit:4 Entries:[1/4 EntryConfChangeV2 v2 v3]
|
|
1->3 MsgApp Term:1 Log:1/3 Commit:4 Entries:[1/4 EntryConfChangeV2 v2 v3]
|
|
|
|
# First, play out the whole interaction between n1 and n2. We see n1's probe to
|
|
# n2 get rejected (since n2 needs a snapshot); the snapshot is delivered at which
|
|
# point n2 switches to the correct config, and n1 catches it up. This notably
|
|
# includes the empty conf change which gets committed and applied by both and
|
|
# which transitions them out of their joint configuration into the final one (1 2 3).
|
|
stabilize 1 2
|
|
----
|
|
> 2 receiving messages
|
|
1->2 MsgApp Term:1 Log:1/3 Commit:4 Entries:[1/4 EntryConfChangeV2 v2 v3]
|
|
INFO 2 [term: 0] received a MsgApp message with higher term from 1 [term: 1]
|
|
INFO 2 became follower at term 1
|
|
DEBUG 2 [logterm: 0, index: 3] rejected MsgApp [logterm: 1, index: 3] from 1
|
|
> 2 handling Ready
|
|
Ready MustSync=true:
|
|
Lead:1 State:StateFollower
|
|
HardState Term:1 Commit:0
|
|
Messages:
|
|
2->1 MsgAppResp Term:1 Log:0/3 Rejected (Hint: 0)
|
|
> 1 receiving messages
|
|
2->1 MsgAppResp Term:1 Log:0/3 Rejected (Hint: 0)
|
|
DEBUG 1 received MsgAppResp(MsgApp was rejected, lastindex: 0) from 2 for index 3
|
|
DEBUG 1 decreased progress of 2 to [StateProbe match=0 next=1]
|
|
DEBUG 1 [firstindex: 3, commit: 4] sent snapshot[index: 4, term: 1] to 2 [StateProbe match=0 next=1]
|
|
DEBUG 1 paused sending replication messages to 2 [StateSnapshot match=0 next=1 paused pendingSnap=4]
|
|
> 1 handling Ready
|
|
Ready MustSync=false:
|
|
Messages:
|
|
1->2 MsgSnap Term:1 Log:0/0 Snapshot: Index:4 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[1] Learners:[] LearnersNext:[] AutoLeave:true
|
|
> 2 receiving messages
|
|
1->2 MsgSnap Term:1 Log:0/0 Snapshot: Index:4 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[1] Learners:[] LearnersNext:[] AutoLeave:true
|
|
INFO log [committed=0, applied=0, unstable.offset=1, len(unstable.Entries)=0] starts to restore snapshot [index: 4, term: 1]
|
|
INFO 2 switched to configuration voters=(1 2 3)&&(1) autoleave
|
|
INFO 2 [commit: 4, lastindex: 4, lastterm: 1] restored snapshot [index: 4, term: 1]
|
|
INFO 2 [commit: 4] restored snapshot [index: 4, term: 1]
|
|
> 2 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Commit:4
|
|
Snapshot Index:4 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[1] Learners:[] LearnersNext:[] AutoLeave:true
|
|
Messages:
|
|
2->1 MsgAppResp Term:1 Log:0/4
|
|
> 1 receiving messages
|
|
2->1 MsgAppResp Term:1 Log:0/4
|
|
DEBUG 1 recovered from needing snapshot, resumed sending replication messages to 2 [StateSnapshot match=4 next=5 paused pendingSnap=4]
|
|
> 1 handling Ready
|
|
Ready MustSync=false:
|
|
Messages:
|
|
1->2 MsgApp Term:1 Log:1/4 Commit:4 Entries:[1/5 EntryConfChangeV2]
|
|
> 2 receiving messages
|
|
1->2 MsgApp Term:1 Log:1/4 Commit:4 Entries:[1/5 EntryConfChangeV2]
|
|
> 2 handling Ready
|
|
Ready MustSync=true:
|
|
Entries:
|
|
1/5 EntryConfChangeV2
|
|
Messages:
|
|
2->1 MsgAppResp Term:1 Log:0/5
|
|
> 1 receiving messages
|
|
2->1 MsgAppResp Term:1 Log:0/5
|
|
> 1 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Vote:1 Commit:5
|
|
CommittedEntries:
|
|
1/5 EntryConfChangeV2
|
|
Messages:
|
|
1->2 MsgApp Term:1 Log:1/5 Commit:5
|
|
INFO 1 switched to configuration voters=(1 2 3)
|
|
> 2 receiving messages
|
|
1->2 MsgApp Term:1 Log:1/5 Commit:5
|
|
> 2 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Commit:5
|
|
CommittedEntries:
|
|
1/5 EntryConfChangeV2
|
|
Messages:
|
|
2->1 MsgAppResp Term:1 Log:0/5
|
|
INFO 2 switched to configuration voters=(1 2 3)
|
|
> 1 receiving messages
|
|
2->1 MsgAppResp Term:1 Log:0/5
|
|
|
|
# n3 immediately receives a snapshot in the final configuration.
|
|
stabilize 1 3
|
|
----
|
|
> 3 receiving messages
|
|
1->3 MsgApp Term:1 Log:1/3 Commit:4 Entries:[1/4 EntryConfChangeV2 v2 v3]
|
|
INFO 3 [term: 0] received a MsgApp message with higher term from 1 [term: 1]
|
|
INFO 3 became follower at term 1
|
|
DEBUG 3 [logterm: 0, index: 3] rejected MsgApp [logterm: 1, index: 3] from 1
|
|
> 3 handling Ready
|
|
Ready MustSync=true:
|
|
Lead:1 State:StateFollower
|
|
HardState Term:1 Commit:0
|
|
Messages:
|
|
3->1 MsgAppResp Term:1 Log:0/3 Rejected (Hint: 0)
|
|
> 1 receiving messages
|
|
3->1 MsgAppResp Term:1 Log:0/3 Rejected (Hint: 0)
|
|
DEBUG 1 received MsgAppResp(MsgApp was rejected, lastindex: 0) from 3 for index 3
|
|
DEBUG 1 decreased progress of 3 to [StateProbe match=0 next=1]
|
|
DEBUG 1 [firstindex: 3, commit: 5] sent snapshot[index: 5, term: 1] to 3 [StateProbe match=0 next=1]
|
|
DEBUG 1 paused sending replication messages to 3 [StateSnapshot match=0 next=1 paused pendingSnap=5]
|
|
> 1 handling Ready
|
|
Ready MustSync=false:
|
|
Messages:
|
|
1->3 MsgSnap Term:1 Log:0/0 Snapshot: Index:5 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[] Learners:[] LearnersNext:[] AutoLeave:false
|
|
> 3 receiving messages
|
|
1->3 MsgSnap Term:1 Log:0/0 Snapshot: Index:5 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[] Learners:[] LearnersNext:[] AutoLeave:false
|
|
INFO log [committed=0, applied=0, unstable.offset=1, len(unstable.Entries)=0] starts to restore snapshot [index: 5, term: 1]
|
|
INFO 3 switched to configuration voters=(1 2 3)
|
|
INFO 3 [commit: 5, lastindex: 5, lastterm: 1] restored snapshot [index: 5, term: 1]
|
|
INFO 3 [commit: 5] restored snapshot [index: 5, term: 1]
|
|
> 3 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Commit:5
|
|
Snapshot Index:5 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[] Learners:[] LearnersNext:[] AutoLeave:false
|
|
Messages:
|
|
3->1 MsgAppResp Term:1 Log:0/5
|
|
> 1 receiving messages
|
|
3->1 MsgAppResp Term:1 Log:0/5
|
|
DEBUG 1 recovered from needing snapshot, resumed sending replication messages to 3 [StateSnapshot match=5 next=6 paused pendingSnap=5]
|
|
> 1 handling Ready
|
|
Ready MustSync=false:
|
|
Messages:
|
|
1->3 MsgApp Term:1 Log:1/5 Commit:5
|
|
> 3 receiving messages
|
|
1->3 MsgApp Term:1 Log:1/5 Commit:5
|
|
> 3 handling Ready
|
|
Ready MustSync=false:
|
|
Messages:
|
|
3->1 MsgAppResp Term:1 Log:0/5
|
|
> 1 receiving messages
|
|
3->1 MsgAppResp Term:1 Log:0/5
|
|
|
|
# Nothing else happens.
|
|
stabilize
|
|
----
|
|
ok
|
|
|
|
# Now remove two nodes. What's new here is that the leader will actually have
|
|
# to go to a quorum to commit the transition into the joint config.
|
|
|
|
propose-conf-change 1
|
|
r2 r3
|
|
----
|
|
ok
|
|
|
|
# n1 sends out MsgApps.
|
|
stabilize 1
|
|
----
|
|
> 1 handling Ready
|
|
Ready MustSync=true:
|
|
Entries:
|
|
1/6 EntryConfChangeV2 r2 r3
|
|
Messages:
|
|
1->2 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
|
|
1->3 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
|
|
|
|
# n2, n3 ack them.
|
|
stabilize 2 3
|
|
----
|
|
> 2 receiving messages
|
|
1->2 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
|
|
> 3 receiving messages
|
|
1->3 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
|
|
> 2 handling Ready
|
|
Ready MustSync=true:
|
|
Entries:
|
|
1/6 EntryConfChangeV2 r2 r3
|
|
Messages:
|
|
2->1 MsgAppResp Term:1 Log:0/6
|
|
> 3 handling Ready
|
|
Ready MustSync=true:
|
|
Entries:
|
|
1/6 EntryConfChangeV2 r2 r3
|
|
Messages:
|
|
3->1 MsgAppResp Term:1 Log:0/6
|
|
|
|
# n1 gets some more proposals. This is part of a regression test: There used to
|
|
# be a bug in which these proposals would prompt the leader to transition out of
|
|
# the same joint state multiple times, which would cause a panic.
|
|
propose 1 foo
|
|
----
|
|
ok
|
|
|
|
propose 1 bar
|
|
----
|
|
ok
|
|
|
|
# n1 switches to the joint config, then initiates a transition into the final
|
|
# config.
|
|
stabilize 1
|
|
----
|
|
> 1 handling Ready
|
|
Ready MustSync=true:
|
|
Entries:
|
|
1/7 EntryNormal "foo"
|
|
1/8 EntryNormal "bar"
|
|
Messages:
|
|
1->2 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
|
|
1->3 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
|
|
1->2 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
|
|
1->3 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
|
|
> 1 receiving messages
|
|
2->1 MsgAppResp Term:1 Log:0/6
|
|
3->1 MsgAppResp Term:1 Log:0/6
|
|
> 1 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Vote:1 Commit:6
|
|
CommittedEntries:
|
|
1/6 EntryConfChangeV2 r2 r3
|
|
Messages:
|
|
1->2 MsgApp Term:1 Log:1/8 Commit:6
|
|
1->3 MsgApp Term:1 Log:1/8 Commit:6
|
|
INFO 1 switched to configuration voters=(1)&&(1 2 3) autoleave
|
|
INFO initiating automatic transition out of joint configuration voters=(1)&&(1 2 3) autoleave
|
|
> 1 handling Ready
|
|
Ready MustSync=true:
|
|
Entries:
|
|
1/9 EntryConfChangeV2
|
|
|
|
# n2 and n3 also switch to the joint config, and ack the transition out of it.
|
|
stabilize 2 3
|
|
----
|
|
> 2 receiving messages
|
|
1->2 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
|
|
1->2 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
|
|
1->2 MsgApp Term:1 Log:1/8 Commit:6
|
|
> 3 receiving messages
|
|
1->3 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
|
|
1->3 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
|
|
1->3 MsgApp Term:1 Log:1/8 Commit:6
|
|
> 2 handling Ready
|
|
Ready MustSync=true:
|
|
HardState Term:1 Commit:6
|
|
Entries:
|
|
1/7 EntryNormal "foo"
|
|
1/8 EntryNormal "bar"
|
|
CommittedEntries:
|
|
1/6 EntryConfChangeV2 r2 r3
|
|
Messages:
|
|
2->1 MsgAppResp Term:1 Log:0/7
|
|
2->1 MsgAppResp Term:1 Log:0/8
|
|
2->1 MsgAppResp Term:1 Log:0/8
|
|
INFO 2 switched to configuration voters=(1)&&(1 2 3) autoleave
|
|
> 3 handling Ready
|
|
Ready MustSync=true:
|
|
HardState Term:1 Commit:6
|
|
Entries:
|
|
1/7 EntryNormal "foo"
|
|
1/8 EntryNormal "bar"
|
|
CommittedEntries:
|
|
1/6 EntryConfChangeV2 r2 r3
|
|
Messages:
|
|
3->1 MsgAppResp Term:1 Log:0/7
|
|
3->1 MsgAppResp Term:1 Log:0/8
|
|
3->1 MsgAppResp Term:1 Log:0/8
|
|
INFO 3 switched to configuration voters=(1)&&(1 2 3) autoleave
|
|
|
|
# n2 and n3 also leave the joint config and the dust settles. We see at the very
|
|
# end that n1 receives some messages from them that it refuses because it does
|
|
# not have them in its config any more.
|
|
stabilize
|
|
----
|
|
> 1 receiving messages
|
|
2->1 MsgAppResp Term:1 Log:0/7
|
|
2->1 MsgAppResp Term:1 Log:0/8
|
|
2->1 MsgAppResp Term:1 Log:0/8
|
|
3->1 MsgAppResp Term:1 Log:0/7
|
|
3->1 MsgAppResp Term:1 Log:0/8
|
|
3->1 MsgAppResp Term:1 Log:0/8
|
|
> 1 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Vote:1 Commit:8
|
|
CommittedEntries:
|
|
1/7 EntryNormal "foo"
|
|
1/8 EntryNormal "bar"
|
|
Messages:
|
|
1->2 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
|
|
1->3 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
|
|
1->2 MsgApp Term:1 Log:1/9 Commit:8
|
|
1->3 MsgApp Term:1 Log:1/9 Commit:8
|
|
> 2 receiving messages
|
|
1->2 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
|
|
1->2 MsgApp Term:1 Log:1/9 Commit:8
|
|
> 3 receiving messages
|
|
1->3 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
|
|
1->3 MsgApp Term:1 Log:1/9 Commit:8
|
|
> 2 handling Ready
|
|
Ready MustSync=true:
|
|
HardState Term:1 Commit:8
|
|
Entries:
|
|
1/9 EntryConfChangeV2
|
|
CommittedEntries:
|
|
1/7 EntryNormal "foo"
|
|
1/8 EntryNormal "bar"
|
|
Messages:
|
|
2->1 MsgAppResp Term:1 Log:0/9
|
|
2->1 MsgAppResp Term:1 Log:0/9
|
|
> 3 handling Ready
|
|
Ready MustSync=true:
|
|
HardState Term:1 Commit:8
|
|
Entries:
|
|
1/9 EntryConfChangeV2
|
|
CommittedEntries:
|
|
1/7 EntryNormal "foo"
|
|
1/8 EntryNormal "bar"
|
|
Messages:
|
|
3->1 MsgAppResp Term:1 Log:0/9
|
|
3->1 MsgAppResp Term:1 Log:0/9
|
|
> 1 receiving messages
|
|
2->1 MsgAppResp Term:1 Log:0/9
|
|
2->1 MsgAppResp Term:1 Log:0/9
|
|
3->1 MsgAppResp Term:1 Log:0/9
|
|
3->1 MsgAppResp Term:1 Log:0/9
|
|
> 1 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Vote:1 Commit:9
|
|
CommittedEntries:
|
|
1/9 EntryConfChangeV2
|
|
Messages:
|
|
1->2 MsgApp Term:1 Log:1/9 Commit:9
|
|
1->3 MsgApp Term:1 Log:1/9 Commit:9
|
|
INFO 1 switched to configuration voters=(1)
|
|
> 2 receiving messages
|
|
1->2 MsgApp Term:1 Log:1/9 Commit:9
|
|
> 3 receiving messages
|
|
1->3 MsgApp Term:1 Log:1/9 Commit:9
|
|
> 2 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Commit:9
|
|
CommittedEntries:
|
|
1/9 EntryConfChangeV2
|
|
Messages:
|
|
2->1 MsgAppResp Term:1 Log:0/9
|
|
INFO 2 switched to configuration voters=(1)
|
|
> 3 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Commit:9
|
|
CommittedEntries:
|
|
1/9 EntryConfChangeV2
|
|
Messages:
|
|
3->1 MsgAppResp Term:1 Log:0/9
|
|
INFO 3 switched to configuration voters=(1)
|
|
> 1 receiving messages
|
|
2->1 MsgAppResp Term:1 Log:0/9
|
|
raft: cannot step as peer not found
|
|
3->1 MsgAppResp Term:1 Log:0/9
|
|
raft: cannot step as peer not found
|