While it appears that etcd is not vulnerable to CVE-2021-3121,
it is a good idea to update to the new generator so that new
vulnerable code isn't generated in any future APIs. Also, this
lays the issue to rest of whether there is any issue with
etcd and CVE-2021-3121.
This commit removes the `XXX_NoUnkeyedLiteral`, `XXX_unrecognized`, and
`XXX_sizecache` auto-generated fields from generated protobuf structs in
the raft package. This was done for all of the same reasons CockroachDB
removed the generation of these fields in https://github.com/cockroachdb/cockroach/pull/38404.
They come with very limited advantages but moderate disadvantages.
`XXX_NoUnkeyedLiteral` and `XXX_sizecache` were only enabled recently in
cc7b4fa, and this appears to have been unintentional. Meanwhile,
`XXX_unrecognized` has been around for longer and has arguably more
reason to stay because it can assist with forwards compatibility.
However, any real mixed-version upgrade story for this package is mostly
untold at this point, and keeping this field seems just as likely to
cause unexpected bugs (e.g. a field was propagated but not updated
correctly when passed through an old version) as it seems to fix real
issues, so it also doesn't warrant its cost.
This reduces the in-memory representation size of all Raft protos.
Notably, it reduces the memory size of an `Entry` proto from *80 bytes*
to *48 bytes* and the memory size of a `Message` proto from *392 bytes*
to *264 bytes*. Both of these structs are used frequently, and often in
slices, where this wasted space really starts to add up.
This was motivated by a regression in microbenchmarks in CockroachDB due
to cc7b4fa, which was caught in https://github.com/cockroachdb/cockroach/issues/62212.
Thanks to this the logs:
- are automatically printed if the test fails.
- are in pretty consistent format.
- are annotated by 'member' information of the cluster emitting them.
Side changes:
- Set propert default got DefaultWarningApplyDuration (used to be '0')
- Name the members based on their 'place' on the list (as opposed to
'random')
The flag protects etcd memory from being swapped out to disk.
This can happen in memory constrained systems where mmaped bbolt
area is natural condidate for swapping out.
This flag should provide better tail latency on the cost of higher RSS
ram usage. If the experiment is successful, the logic should get moved
into bbolt layer, where we can protect specific bbolt instances
(e.g. avoid protecting both during defragmentation).
Seems old versions of https://github.com/certifi/gocertifi where
categorized as "Do What The F*ck You Want To Public License".
Update to newer version that is explicit `Mozilla Public License` 2.0 (MPL 2.0).
- Add a large detailed comment about the use and necessity of
both the follower and leader probing optimization
- fix the log message in stepLeader that previously mixed up the
log term for the rejection and the index of the append
- improve the test via subtests
- add some verbiage in findConflictByTerm around first index
This commit creates a new probe_and_replicate.txt interactive test. The
test creates a complete Raft log configuration and demonstrates how a
leader probes and replicates to each of its followers. The log
configuration constructed is identical to the one present in Figure 7 of
the raft paper (https://raft.github.io/raft.pdf), which looks like:
```
1 2 3 4 5 6 7 8 9 10 11 12
n1: [1][1][1][4][4][5][5][6][6][6]
n2: [1][1][1][4][4][5][5][6][6]
n3: [1][1][1][4]
n4: [1][1][1][4][4][5][5][6][6][6][6]
n5: [1][1][1][4][4][5][5][6][7][7][7][7]
n6: [1][1][1][4][4][4][4]
n7: [1][1][1][2][2][2][3][3][3][3][3]
```
Once in this state, we then elect node 1 as the leader and stabilize the
entire raft group. This demonstrates how a newly elected leader probes
for matching indexes, overwrites conflicting entries, and catches up all
followers.
This will be useful to demonstrate the impact of more efficient probing
behavior.
