999 Commits

Author SHA1 Message Date
Piotr Tabor
2c8de24afc
Merge pull request #12495 from VertexC/fix/testErrorMsg
raft/raft_test: minor fix on inconsistent format of test case's error message
2021-01-31 22:25:23 +01:00
Piotr Tabor
5472b3336b
Merge pull request #12525 from sakateka/remove_raft.peers
raft tests: Remove Config.peers and Config.learners
2021-01-19 16:00:54 +01:00
Sergey Kacheev
ccfd00f687
raft: specify voters and learners via snapshot 2021-01-16 13:03:47 +07:00
Piotr Tabor
bf6f173d5e Document Raft.send method.
The change makes it explicit that sending messages does not happen
immidietely and is subject to proper persist & then send protocol
on the application side. See:

https://github.com/etcd-io/etcd/issues/12589#issuecomment-752867024

for more context.
2021-01-15 12:35:58 +01:00
Piotr Tabor
58f78df1de Raft: Expand raft documentation, in particular point on godocs. 2021-01-15 12:34:02 +01:00
VertexC
082b9a815d minor fix on inconsistent format of test case's error message 2020-11-30 12:18:12 +08:00
Piotr Tabor
eeafcef0d2 Use "v3.5.0-pre" to reference within-etcd modules
instead of v3.0.0-000101010000000-00000000000,
that might be misleading as we don't develop etcd v3.0.0 any longer.

This version is a virtual version and is not supposed to be tagged
within the repository. We should tag real versions like: 3.5.0-alpha.0.

Please notice that go.etcd.io/etcd/client/v2 will be versioned as `v2.305.0-pre`.
The reason is that client v2 must have v2 version. I propose a
convention to envode the major version as 100x in minor version to make
the association to the underlying repository clear, staying within v2
version family.

The change was generated using:
```
DRY_RUN=false TARGET_VERSION="v3.5.0-pre" ./scripts/release_mod.sh update_versions
```
2020-11-04 18:28:43 +01:00
Piotr Tabor
e33c6dd9df client/v3: Rename of imports 2020-10-20 10:13:06 +02:00
Piotr Tabor
e62417297d *: Rename of imports of raft (as its now a module)
% find -name '*.go' -o -name '*.md' -o -name '*.sh' | xargs sed -i --follow-symlinks 's|etcd/v3/raft|etcd/raft/v3|g'
2020-10-16 13:58:18 +02:00
Piotr Tabor
bea5f379bc raft: Make raft a module
Establishing a module: Creating a go.mod files
2020-10-16 09:15:48 +02:00
Piotr Tabor
b7f0f52a16
*: Refresh of dependencies (#12399)
This PR focuses on dependencies that have new stable versions.
2020-10-15 15:32:00 -07:00
Piotr Tabor
371ddf0b69 tests: Update diagnostic update in tests after change of proto version. 2020-10-14 18:46:38 +02:00
Piotr Tabor
cc7b4fab9f *: Execution of ./scripts/genproto.sh after versions change 2020-10-14 18:46:38 +02:00
Piotr Tabor
de55bb6331 pkg: Rename imports after making 'pkg' a module
find -name '*.go' | xargs sed --follow-symlinks -i 's|go.etcd.io/etcd/v3/pkg/|go.etcd.io/etcd/pkg/v3/|g'
go fmt ./...
2020-10-13 00:09:27 +02:00
vivian
ab4cc3caef
raft/log: remove redundant code logic (#12346)
Remove redundant code logic

Co-authored-by: yangweiwei <yangweiwei@cmss.chinamobile.com>
2020-09-29 19:48:32 -07:00
Piotr Tabor
3f449a8548 *: 'go test --short ./...' runs all UNIT tests.
Marked all 'integrational, e2e' as skipped in the --short mode.

Thanks to this we will be able to significantly simplify ./test script.

The run currently takes ~23s.
With (follow up) move of ~clientv3/snapshot to integration tests (as
part of modularization), we can expect this to fall to 5-10s.

```
% time go test --short ./... --count=1
ok  	go.etcd.io/etcd/v3	0.098s
?   	go.etcd.io/etcd/v3/Documentation/learning/lock/client	[no test files]
?   	go.etcd.io/etcd/v3/Documentation/learning/lock/storage	[no test files]
ok  	go.etcd.io/etcd/v3/auth	0.724s
?   	go.etcd.io/etcd/v3/auth/authpb	[no test files]
ok  	go.etcd.io/etcd/v3/client	0.166s
ok  	go.etcd.io/etcd/v3/client/integration	0.166s
ok  	go.etcd.io/etcd/v3/clientv3	3.219s
ok  	go.etcd.io/etcd/v3/clientv3/balancer	1.102s
?   	go.etcd.io/etcd/v3/clientv3/balancer/connectivity	[no test files]
?   	go.etcd.io/etcd/v3/clientv3/balancer/picker	[no test files]
?   	go.etcd.io/etcd/v3/clientv3/balancer/resolver/endpoint	[no test files]
ok  	go.etcd.io/etcd/v3/clientv3/clientv3util	0.096s [no tests to run]
ok  	go.etcd.io/etcd/v3/clientv3/concurrency	3.323s
?   	go.etcd.io/etcd/v3/clientv3/credentials	[no test files]
ok  	go.etcd.io/etcd/v3/clientv3/integration	0.131s
?   	go.etcd.io/etcd/v3/clientv3/leasing	[no test files]
?   	go.etcd.io/etcd/v3/clientv3/mirror	[no test files]
ok  	go.etcd.io/etcd/v3/clientv3/namespace	0.041s
ok  	go.etcd.io/etcd/v3/clientv3/naming	0.115s
ok  	go.etcd.io/etcd/v3/clientv3/ordering	0.121s
ok  	go.etcd.io/etcd/v3/clientv3/snapshot	19.325s
ok  	go.etcd.io/etcd/v3/clientv3/yaml	0.090s
ok  	go.etcd.io/etcd/v3/contrib/raftexample	7.572s
?   	go.etcd.io/etcd/v3/contrib/recipes	[no test files]
ok  	go.etcd.io/etcd/v3/embed	0.282s
ok  	go.etcd.io/etcd/v3/etcdctl	0.054s
?   	go.etcd.io/etcd/v3/etcdctl/ctlv2	[no test files]
ok  	go.etcd.io/etcd/v3/etcdctl/ctlv2/command	0.117s
?   	go.etcd.io/etcd/v3/etcdctl/ctlv3	[no test files]
ok  	go.etcd.io/etcd/v3/etcdctl/ctlv3/command	0.070s
ok  	go.etcd.io/etcd/v3/etcdmain	0.172s
ok  	go.etcd.io/etcd/v3/etcdserver	1.698s
?   	go.etcd.io/etcd/v3/etcdserver/api	[no test files]
ok  	go.etcd.io/etcd/v3/etcdserver/api/etcdhttp	0.075s
ok  	go.etcd.io/etcd/v3/etcdserver/api/membership	0.104s
?   	go.etcd.io/etcd/v3/etcdserver/api/membership/membershippb	[no test files]
ok  	go.etcd.io/etcd/v3/etcdserver/api/rafthttp	0.181s
ok  	go.etcd.io/etcd/v3/etcdserver/api/snap	0.078s
?   	go.etcd.io/etcd/v3/etcdserver/api/snap/snappb	[no test files]
ok  	go.etcd.io/etcd/v3/etcdserver/api/v2auth	0.142s
ok  	go.etcd.io/etcd/v3/etcdserver/api/v2discovery	0.035s
ok  	go.etcd.io/etcd/v3/etcdserver/api/v2error	0.043s
ok  	go.etcd.io/etcd/v3/etcdserver/api/v2http	0.070s
ok  	go.etcd.io/etcd/v3/etcdserver/api/v2http/httptypes	0.031s
?   	go.etcd.io/etcd/v3/etcdserver/api/v2stats	[no test files]
ok  	go.etcd.io/etcd/v3/etcdserver/api/v2store	0.645s
ok  	go.etcd.io/etcd/v3/etcdserver/api/v2v3	0.218s
?   	go.etcd.io/etcd/v3/etcdserver/api/v3alarm	[no test files]
?   	go.etcd.io/etcd/v3/etcdserver/api/v3client	[no test files]
ok  	go.etcd.io/etcd/v3/etcdserver/api/v3compactor	1.765s
?   	go.etcd.io/etcd/v3/etcdserver/api/v3election	[no test files]
?   	go.etcd.io/etcd/v3/etcdserver/api/v3election/v3electionpb	[no test files]
?   	go.etcd.io/etcd/v3/etcdserver/api/v3election/v3electionpb/gw	[no test files]
?   	go.etcd.io/etcd/v3/etcdserver/api/v3lock	[no test files]
?   	go.etcd.io/etcd/v3/etcdserver/api/v3lock/v3lockpb	[no test files]
?   	go.etcd.io/etcd/v3/etcdserver/api/v3lock/v3lockpb/gw	[no test files]
ok  	go.etcd.io/etcd/v3/etcdserver/api/v3rpc	0.091s
ok  	go.etcd.io/etcd/v3/etcdserver/api/v3rpc/rpctypes	0.012s
ok  	go.etcd.io/etcd/v3/etcdserver/cindex	0.054s
ok  	go.etcd.io/etcd/v3/etcdserver/etcdserverpb	0.039s
?   	go.etcd.io/etcd/v3/etcdserver/etcdserverpb/gw	[no test files]
ok  	go.etcd.io/etcd/v3/functional/agent	0.094s
?   	go.etcd.io/etcd/v3/functional/cmd/etcd-agent	[no test files]
?   	go.etcd.io/etcd/v3/functional/cmd/etcd-proxy	[no test files]
?   	go.etcd.io/etcd/v3/functional/cmd/etcd-runner	[no test files]
?   	go.etcd.io/etcd/v3/functional/cmd/etcd-tester	[no test files]
ok  	go.etcd.io/etcd/v3/functional/rpcpb	0.060s
?   	go.etcd.io/etcd/v3/functional/runner	[no test files]
ok  	go.etcd.io/etcd/v3/functional/tester	0.079s
ok  	go.etcd.io/etcd/v3/integration	0.684s
ok  	go.etcd.io/etcd/v3/integration/embed	0.101s
ok  	go.etcd.io/etcd/v3/lease	3.455s
ok  	go.etcd.io/etcd/v3/lease/leasehttp	2.185s
?   	go.etcd.io/etcd/v3/lease/leasepb	[no test files]
ok  	go.etcd.io/etcd/v3/mvcc	7.246s
ok  	go.etcd.io/etcd/v3/mvcc/backend	0.354s
?   	go.etcd.io/etcd/v3/mvcc/mvccpb	[no test files]
ok  	go.etcd.io/etcd/v3/pkg/adt	0.025s
?   	go.etcd.io/etcd/v3/pkg/contention	[no test files]
?   	go.etcd.io/etcd/v3/pkg/cpuutil	[no test files]
ok  	go.etcd.io/etcd/v3/pkg/crc	0.008s
?   	go.etcd.io/etcd/v3/pkg/debugutil	[no test files]
ok  	go.etcd.io/etcd/v3/pkg/expect	0.015s
ok  	go.etcd.io/etcd/v3/pkg/fileutil	0.268s
ok  	go.etcd.io/etcd/v3/pkg/flags	0.021s
ok  	go.etcd.io/etcd/v3/pkg/httputil	0.020s
ok  	go.etcd.io/etcd/v3/pkg/idutil	0.008s
ok  	go.etcd.io/etcd/v3/pkg/ioutil	0.025s
ok  	go.etcd.io/etcd/v3/pkg/logutil	0.047s
?   	go.etcd.io/etcd/v3/pkg/mock/mockserver	[no test files]
?   	go.etcd.io/etcd/v3/pkg/mock/mockstorage	[no test files]
?   	go.etcd.io/etcd/v3/pkg/mock/mockstore	[no test files]
?   	go.etcd.io/etcd/v3/pkg/mock/mockwait	[no test files]
ok  	go.etcd.io/etcd/v3/pkg/netutil	1.024s
ok  	go.etcd.io/etcd/v3/pkg/osutil	0.021s
ok  	go.etcd.io/etcd/v3/pkg/pathutil	0.008s
ok  	go.etcd.io/etcd/v3/pkg/pbutil	0.008s
ok  	go.etcd.io/etcd/v3/pkg/proxy	4.081s
ok  	go.etcd.io/etcd/v3/pkg/report	0.008s
?   	go.etcd.io/etcd/v3/pkg/runtime	[no test files]
ok  	go.etcd.io/etcd/v3/pkg/schedule	0.009s
ok  	go.etcd.io/etcd/v3/pkg/srv	0.019s
ok  	go.etcd.io/etcd/v3/pkg/stringutil	0.008s
?   	go.etcd.io/etcd/v3/pkg/systemd	[no test files]
ok  	go.etcd.io/etcd/v3/pkg/testutil	0.023s
ok  	go.etcd.io/etcd/v3/pkg/tlsutil	3.965s
ok  	go.etcd.io/etcd/v3/pkg/traceutil	0.034s
ok  	go.etcd.io/etcd/v3/pkg/transport	0.532s
ok  	go.etcd.io/etcd/v3/pkg/types	0.028s
ok  	go.etcd.io/etcd/v3/pkg/wait	0.023s
ok  	go.etcd.io/etcd/v3/proxy/grpcproxy	0.101s
?   	go.etcd.io/etcd/v3/proxy/grpcproxy/adapter	[no test files]
?   	go.etcd.io/etcd/v3/proxy/grpcproxy/cache	[no test files]
ok  	go.etcd.io/etcd/v3/proxy/httpproxy	0.044s
ok  	go.etcd.io/etcd/v3/proxy/tcpproxy	0.047s
ok  	go.etcd.io/etcd/v3/raft	0.312s
ok  	go.etcd.io/etcd/v3/raft/confchange	0.183s
ok  	go.etcd.io/etcd/v3/raft/quorum	0.316s
ok  	go.etcd.io/etcd/v3/raft/raftpb	0.024s
ok  	go.etcd.io/etcd/v3/raft/rafttest	0.640s
ok  	go.etcd.io/etcd/v3/raft/tracker	0.026s
ok  	go.etcd.io/etcd/v3/tests/e2e	0.077s
?   	go.etcd.io/etcd/v3/tools/benchmark	[no test files]
?   	go.etcd.io/etcd/v3/tools/benchmark/cmd	[no test files]
?   	go.etcd.io/etcd/v3/tools/etcd-dump-db	[no test files]
ok  	go.etcd.io/etcd/v3/tools/etcd-dump-logs	0.088s
?   	go.etcd.io/etcd/v3/tools/etcd-dump-metrics	[no test files]
?   	go.etcd.io/etcd/v3/tools/local-tester/bridge	[no test files]
?   	go.etcd.io/etcd/v3/version	[no test files]
ok  	go.etcd.io/etcd/v3/wal	1.517s
?   	go.etcd.io/etcd/v3/wal/walpb	[no test files]
go test --short ./... --count=1  76.12s user 12.57s system 375% cpu 23.635 total
```
2020-09-16 13:21:46 +02:00
Jay
26b89fd418
raft: don't campaign with pending snapshot (#12163)
Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
2020-07-26 00:04:46 -07:00
Jay
d0e4fe56a5
raft: check pending conf change before campaign (#12134)
* raft: check conf change before campaign

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* raft: extract hup function

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* raft: check pending conf change for transferleader

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
2020-07-22 17:04:48 -07:00
Jay
cc656718fa
raft: correct pendingConfIndex check for AutoLeave (#12137)
Close #12136

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
2020-07-20 16:49:22 -07:00
Zhijie Zhan
93637b1779
raft: bug fix (#12123)
we need to test the case when configuration set is changed, but there is typo.

None

Signed-off-by: accelsao <bebe8277@gmail.com>
2020-07-20 16:30:17 -07:00
Binacs
89da79188f
raft: simplify the code in progress.go (#12119) 2020-07-20 00:13:05 -07:00
gavwu
8f72368070 raft: fix typo 2020-06-14 13:26:37 +08:00
Ted Yu
9d20ba4822 raft: avoid data race by not reading raft.lead
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
2020-05-16 20:40:25 -07:00
Zhihong Yu
7cc2f8a411
raft: break out of nested loop when id is found (#11870)
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
2020-05-12 16:59:22 -07:00
Ted Yu
43794efb1f raft: break out of nested loop when raft id is found
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
2020-05-09 17:56:44 -07:00
Brandon Philips
c9cdefaeac *: run gofmt and make fmt make test pass
gofmt tests were failing, fix.
2020-04-28 00:57:52 +00:00
Brandon Philips
7a5fefedcd *: fix imports missed by gomove 2020-04-28 00:57:52 +00:00
Brandon Philips
96cce208c2 go.mod: use go.etcd.io/etcd/v3 versioning
This change makes the etcd package compatible with the existing Go
ecosystem for module versioning.

Used this tool to update package imports:
  https://github.com/KSubedi/gomove
2020-04-28 00:57:35 +00:00
Andrei Matei
0db04a38a5 raft: minor comment fix in confchange 2020-04-12 22:44:37 -04:00
Fullstop000
7eae024ead
raft: only redirect msg produced by own node (#11466)
Signed-off-by: Fullstop000 <fullstop1005@gmail.com>
2020-04-06 20:27:46 -07:00
Lars Lehtonen
dbcf540c88
raft/rafttest: Prune Unused Functions (#11625)
* raft/rafttest: prune unused network interface

* raft/rafttest: prune unused raftNetwork.heal()

* raft/rafttest: prune unused logLevels.strToLev()

* raft/rafttest: prune unused InteractionEnv.writeErr()
2020-04-04 14:38:14 -07:00
qupeng
6f850a65a1
raft: cleanup read index code (#11528)
Signed-off-by: qupeng <qupeng@pingcap.com>
2020-03-03 09:20:25 -08:00
Tobias Schottdorf
0544f33248 raft: clarify ApplyConfChange contract for rejected conf changes
Apps typically maintain the raft configuration as part of the state
machine. As a result, they want to be able to reject configuration change
entries at apply time based on the state on which the entry is supposed
to be applied. When this happens, the app should not call
ApplyConfChange, but the comments did not make this clear.

As a result, it was tempting to pass an empty pb.ConfChange or it's V2
version instead of not calling ApplyConfChange.

However, an empty V1 or V2 proto aren't noops when the configuration is
joint: an empty V1 change is treated internally as a single
configuration change for NodeID zero and will cause a panic when applied
in a joint state. An empty V2 proto is treated as a signal to leave a
joint state, which means that the app's config and raft's would diverge.

The comments updated in this commit now ask users to not call
ApplyConfState when they reject a conf change. Apps that never use joint
consensus can keep their old behavior since the distinction only matters
when in a joint state, but we don't want to encourage that.
2020-02-25 12:45:45 +01:00
Tobias Schottdorf
37c7e4d1d8 raft: fix auto-transitioning out of joint config
The code doing so was undertested and buggy: it would launch multiple
attempts to transition out when the conf change was not the last element
in the log.

This commit fixes the problem and adds a regression test. It also
reworks the code to handle a former untested edge case, in which the
auto-transition append is refused. This can't happen any more with the
current version of the code because this proposal has size zero and is
special cased in increaseUncommittedSize. Last but not least, the
auto-leave proposal now also bumps pendingConfIndex, which was not done
previously due to an oversight.
2020-02-25 12:35:51 +01:00
Tobias Schottdorf
2332705f10 raft: remove bogus tail end of membership change interaction test
The test was supposed to end earlier, but some old copy pasta
survived.
2020-02-25 12:35:51 +01:00
po3rin
53f15caf73
raft: Fixed missing package name in README.md (#11566) 2020-01-29 10:15:00 -08:00
qupeng
eaa0612e02 raft: abort leader transferring if the target is demoted (#11417)
Signed-off-by: qupeng <qupeng@pingcap.com>
2019-12-20 12:07:52 +08:00
Baohua Yang
4b755e8935 docs: Update the raft usage by adding Hyperledger project 2019-11-19 09:54:16 -08:00
Ilya Sevostyanov
d487b16de1
confchange: removed duplicate check in confchange.Simple. 2019-10-01 11:32:12 +03:00
Wine93
b0534c1b44 raft/log_test: fixed wrong index 2019-08-25 04:47:11 +00:00
Wine93
5f42161750 raft: fixed some typos and simplify minor logic 2019-08-25 04:46:29 +00:00
Xiang Li
f4dfd1976d
Merge pull request #10971 from nilsocket/codeReformat
raft : write compact if statements
2019-08-23 09:10:26 +08:00
nilsocket
702c69c906
raft : Write compact if statements 2019-08-23 04:31:40 +05:30
Tobias Schottdorf
47ae53d25d rafttest: print Ready before processing it
It was confusing to see the effects of the Ready (i.e. log messages)
printed before the Ready itself.
2019-08-16 09:41:35 +02:00
Tobias Schottdorf
99f8046fd1 raft: fix a test file name 2019-08-16 09:38:44 +02:00
Tobias Schottdorf
8d1946d16a raft: document problem with leader self-removal
When a leader removes itself, it will retain its leadership but not
accept new proposals, making the range effectively stuck until manual
intervention triggers a campaign event.

This commit documents the behavior. It does not correct it yet.
2019-08-16 09:38:44 +02:00
Tobias Schottdorf
306e75a96f raft: add a batch of interaction-driven conf change tests
Verifiy the behavior in various v1 and v2 conf change operations.
This also includes various fixups, notably it adds protection
against transitioning in and out of new configs when this is not
permissible.

There are more threads to pull, but those are left for future commits.
2019-08-16 09:38:44 +02:00
Tobias Schottdorf
4e19150676 raft: proactively probe newly added followers
When the leader applied a new configuration that added voters, it would
not immediately probe these voters, delaying when they would be caught
up.

I noticed this while writing an interaction-driven test, which has now
been cleaned up and completed.
2019-08-14 20:53:34 +02:00
Tobias Schottdorf
3d6721f751 rafttest: add _breakpoint directive
It is a helper case to attach a debugger to when a problem needs
to be investigated in a longer test file. In such a case, add the
following stanza immediately before the interesting behavior starts:

_breakpoint:
----
ok

and set a breakpoint on the _breakpoint case.
2019-08-14 20:53:34 +02:00
Tobias Schottdorf
fdaed88f14 raft: initialize new Progress at LastIndex, not LastIndex+1
Initializing at LastIndex+1 meant that new peers would not be probed
immediately when they appeared in the leader's config, which delays
their getting caught up.
2019-08-14 20:53:34 +02:00