Benjamin Wang
cd9764a99f
Merge pull request #14534 from demoManito/feat/golangci
...
golangci: add import order check
2022-09-29 15:26:00 +08:00
demoManito
7020e2fd2b
fix lint
...
Signed-off-by: demoManito <1430482733@qq.com>
2022-09-29 13:57:55 +08:00
demoManito
ae118a17f0
fix goimorts local-prefixes
...
Signed-off-by: demoManito <1430482733@qq.com>
2022-09-29 13:55:47 +08:00
demoManito
0a08fa2b43
golangci: add import order check
...
Signed-off-by: demoManito <1430482733@qq.com>
2022-09-29 11:30:04 +08:00
Marek Siarkowicz
d379e6221e
Merge pull request #14535 from demoManito/feat/golangci-unconvert
...
golangci: add unconvert check
2022-09-28 14:04:04 +02:00
demoManito
93b9dc6583
golangci: add unconvert check
...
Signed-off-by: demoManito <1430482733@qq.com>
2022-09-28 19:13:58 +08:00
Marek Siarkowicz
6828e53b0a
Merge pull request #14531 from ahrtr/changelog_3.4_memberId_zero
...
Update changelog-3.4 to cover the issue: memberID equals zero in corruption alarm
2022-09-28 09:30:45 +02:00
Benjamin Wang
ea17d9d6bf
update changelog-3.4 to cover the issue: memberID equals zero in corruption alarm
...
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-09-28 11:05:53 +08:00
Benjamin Wang
abef537a90
Merge pull request #14515 from spongecaptain/btree-generics
...
upate:use google/btree in the genric way
2022-09-27 16:44:13 +08:00
wathenjiang
319db38b0a
update: add benchmark test
...
benchmark result:
(1) master branch
$ go test -bench='BenchmarkIndexPut$' -count=5
goos: darwin
goarch: amd64
pkg: go.etcd.io/etcd/server/v3/storage/mvcc
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkIndexPut-12 1000000 2591 ns/op
BenchmarkIndexPut-12 1000000 2531 ns/op
BenchmarkIndexPut-12 1000000 2536 ns/op
BenchmarkIndexPut-12 1000000 2546 ns/op
BenchmarkIndexPut-12 1000000 2538 ns/op
PASS
ok go.etcd.io/etcd/server/v3/storage/mvcc 167.439s
$ go test -bench='BenchmarkIndexGet$' -count=5
goos: darwin
goarch: amd64
pkg: go.etcd.io/etcd/server/v3/storage/mvcc
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkIndexGet-12 1000000 2021 ns/op
BenchmarkIndexGet-12 1000000 2029 ns/op
BenchmarkIndexGet-12 1000000 2044 ns/op
BenchmarkIndexGet-12 1000000 1973 ns/op
BenchmarkIndexGet-12 1000000 2027 ns/op
PASS
ok go.etcd.io/etcd/server/v3/storage/mvcc 177.815s
(2) google/btree in the generic way
$ go test -bench='BenchmarkIndexPut$' -count=5
goos: darwin
goarch: amd64
pkg: go.etcd.io/etcd/server/v3/storage/mvcc
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkIndexPut-12 1000000 2477 ns/op
BenchmarkIndexPut-12 1000000 2380 ns/op
BenchmarkIndexPut-12 1000000 2360 ns/op
BenchmarkIndexPut-12 1000000 2396 ns/op
BenchmarkIndexPut-12 1000000 2382 ns/op
PASS
ok go.etcd.io/etcd/server/v3/storage/mvcc 165.841s
$ go test -bench='BenchmarkIndexGet$' -count=5
goos: darwin
goarch: amd64
pkg: go.etcd.io/etcd/server/v3/storage/mvcc
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkIndexGet-12 1000000 1985 ns/op
BenchmarkIndexGet-12 1000000 1914 ns/op
BenchmarkIndexGet-12 1000000 1900 ns/op
BenchmarkIndexGet-12 1000000 1905 ns/op
BenchmarkIndexGet-12 1000000 1894 ns/op
PASS
ok go.etcd.io/etcd/server/v3/storage/mvcc 177.573s
Signed-off-by: wathenjiang <wathenjiang@tencent.com>
2022-09-27 14:33:02 +08:00
Spongecaptain
c53dfc7c5b
upate:use google/btree in the genric way
...
Signed-off-by: wathenjiang <wathenjiang@tencent.com>
2022-09-27 10:16:15 +08:00
Benjamin Wang
1c20ed2cc5
Merge pull request #14521 from lovehhf/remove_pick_peer_url
...
membership: Remove PickPeerURL Method
2022-09-27 02:10:35 +08:00
Benjamin Wang
9cb5764859
Merge pull request #14296 from kafuu-chino/fix
...
Fixed unexpected cancelled watch with WatchID=0.
2022-09-27 02:08:23 +08:00
Hongfei Huang
f6d808736c
membership: Remove PickPeerURL Method
...
PickPeerURL only used by unit test
Signed-off-by: Hongfei Huang <853885165@qq.com>
2022-09-26 23:21:10 +08:00
Kafuu Chino
f1d4935e91
*: avoid closing a watch with ID 0 incorrectly
...
Signed-off-by: Kafuu Chino <KafuuChinoQ@gmail.com>
add test
2022-09-26 20:30:33 +08:00
Benjamin Wang
434c7c4309
Merge pull request #14506 from SimFG/main
...
wal: Fix the `walWriteBytes` metric
2022-09-26 09:36:07 +08:00
Jeremy Leach
cc1e245368
etcdserver: fix corruption check when server has just been compacted
...
When a key-value store corruption check happens immediately after a
compaction, the revision at which the key-value store hash is computed,
is the compacted revision itself.
In that case, the hash computation logic was incorrect because it
returned an ErrCompacted error; this error should instead be returned when
the revision at which the key-value store is hashed, is strictly lower
than the compacted revision.
Fixes #14325
Signed-off-by: Jeremy Leach <44558776+jbml@users.noreply.github.com>
2022-09-24 22:20:26 +10:00
Marek Siarkowicz
4513671b69
Merge pull request #14512 from joycebrum/main
...
Enable the OpenSSF Scorecard Github Action
2022-09-23 19:02:17 +02:00
Joyce Brum
c6dd01f337
add scorecard badge to README file
...
Signed-off-by: Joyce Brum <joycebrumu.u@gmail.com>
2022-09-23 13:59:17 -03:00
Joyce Brum
336bae2d4e
github: enable the scorecard github action
...
Signed-off-by: Joyce Brum <joycebrumu.u@gmail.com>
2022-09-23 13:57:22 -03:00
SimFG
5702765729
wal: Fix the walWriteBytes
metric
...
Signed-off-by: SimFG <1142838399@qq.com>
2022-09-22 19:23:06 +08:00
Benjamin Wang
54f9483e72
Merge pull request #14504 from ahrtr/update_changelog_go1.19
...
Update changelog-3.6 to reflect the bumping of go 1.19
2022-09-22 17:06:46 +08:00
Benjamin Wang
ea3d78faae
Merge pull request #14505 from ahrtr/revert_14400
...
etcdserve: revert the etcdserver side change for the data loss in one node cluster
2022-09-22 17:06:10 +08:00
Benjamin Wang
9097e61b40
etcdserve: revert the etcdserver side change for the data loss on one node cluster
...
Since the raft side change has been merged, so we need to revert the etcdserver
side change.
Refer to
https://github.com/etcd-io/etcd/pull/14413
https://github.com/etcd-io/etcd/pull/14400
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-09-22 15:19:20 +08:00
Benjamin Wang
da7e640473
Update changelog-3.6 to reflect the bumping of go 1.19
...
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-09-22 15:07:52 +08:00
Benjamin Wang
997260a832
Merge pull request #14463 from ahrtr/bump_go_1.19
...
Bump golang version to 1.19.1
2022-09-22 09:25:58 +08:00
Benjamin Wang
dd7d30017c
Bump go 1.19: revert the change to pkg/adt/interval_tree.go
...
Some comments in the file are formatted automatically into ugly style,
because the hierarchical structure is missing. After removing the
leading numbers in the comments, `go fmt` will not format the comments
anymore.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-09-22 08:47:46 +08:00
Benjamin Wang
7f10dccbaf
Bump go 1.19: update all the dependencies and go.sum files
...
1. run ./scripts/fix.sh;
2. cd tools/mod; gofmt -w . & go mod tidy;
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-09-22 08:47:46 +08:00
Benjamin Wang
cb5f7276c3
Bump go 1.19: upgrade go version to 1.19.1 in the pipeline
...
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-09-22 08:47:46 +08:00
Benjamin Wang
cd0b1d0c66
Bump go 1.19: upgrade go version to 1.19 in all go.mod files
...
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-09-22 08:47:46 +08:00
Benjamin Wang
31d9664cb5
Merge pull request #14413 from tbg/raft-single-voter
...
raft: don't emit unstable CommittedEntries
2022-09-22 08:43:37 +08:00
Marek Siarkowicz
026794495f
Merge pull request #14494 from demoManito/remove/redundant-type-conversion
...
etcd: remove redundant type conversion
2022-09-21 11:34:19 +02:00
Benjamin Wang
6333f375a7
Merge pull request #14488 from serathius/update-fix
...
Improve static analysis fixing scripts
2022-09-21 06:20:08 +08:00
Benjamin Wang
2441a24cee
Merge pull request #14493 from demoManito/style/format-import-order
...
etcd: format import order
2022-09-21 06:03:31 +08:00
Marek Siarkowicz
bea478266e
makefile: Raname targets update* to fix* to distinquish from update_dep
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-09-20 13:58:17 +02:00
Marek Siarkowicz
5bfda80836
makefile: test the update target
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-09-20 13:57:59 +02:00
Marek Siarkowicz
bb139b15f8
makefile: Don't run update_dep.sh as it's not a check
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-09-20 13:55:51 +02:00
Marek Siarkowicz
05104ee9a7
makefile: Remove verify-revive as it is already run by golangci
...
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-09-20 13:55:51 +02:00
demoManito
f67ec10779
etcd: format import order
...
golang CodeReviewComments:
https://github.com/golang/go/wiki/CodeReviewComments#imports
Signed-off-by: demoManito <1430482733@qq.com>
2022-09-20 18:41:39 +08:00
Tobias Grieger
d56676c9b3
raft: benchmark results for ./benchmark put
...
I ran this PR against its main merge-base twice (on my 2021 Mac M1 pro),
and in both cases this PR was slightly faster, using the benchmark
invocation from [^1].
2819.6 vs 2808.4
2873.1 vs 2835
Full output below.
----
Script:
```
killall etcd
rm -rf default.etcd
scripts/build.sh
nohup ./bin/etcd --quota-backend-bytes=4300000000 &
sleep 10
f=bench-$(git log -1 --pretty=%s | sed -E 's/[^A-Za-z0-9]+/_/g').txt
go run ./tools/benchmark txn-put --endpoints="http://127.0.0.1:2379 " --clients=200 --conns=200 --key-space-size=4000000000 --key-size=128 --val-size=10240 --total=200000 --rate=40000 | tee "${f}"
```
PR:
```
Summary:
Total: 70.9320 secs.
Slowest: 0.3003 secs.
Fastest: 0.0044 secs.
Average: 0.0707 secs.
Stddev: 0.0437 secs.
Requests/sec: 2819.6030 (second run: 2873.0935)
Response time histogram:
0.0044 [1] |
0.0340 [2877] |
0.0636 [119485] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
0.0932 [17436] |∎∎∎∎∎
0.1228 [27364] |∎∎∎∎∎∎∎∎∎
0.1524 [20349] |∎∎∎∎∎∎
0.1820 [10214] |∎∎∎
0.2116 [1248] |
0.2412 [564] |
0.2707 [318] |
0.3003 [144] |
Latency distribution:
10% in 0.0368 secs.
25% in 0.0381 secs.
50% in 0.0416 secs.
75% in 0.0998 secs.
90% in 0.1375 secs.
95% in 0.1571 secs.
99% in 0.1850 secs.
99.9% in 0.2650 secs.
```
main:
```
Summary:
Total: 71.2152 secs.
Slowest: 0.6926 secs.
Fastest: 0.0040 secs.
Average: 0.0710 secs.
Stddev: 0.0461 secs.
Requests/sec: 2808.3903 (second run: 2834.98)
Response time histogram:
0.0040 [1] |
0.0728 [125816] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
0.1417 [59127] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
0.2105 [13476] |∎∎∎∎
0.2794 [1125] |
0.3483 [137] |
0.4171 [93] |
0.4860 [193] |
0.5549 [4] |
0.6237 [16] |
0.6926 [12] |
Latency distribution:
10% in 0.0367 secs.
25% in 0.0379 secs.
50% in 0.0417 secs.
75% in 0.0993 secs.
90% in 0.1367 secs.
95% in 0.1567 secs.
99% in 0.1957 secs.
99.9% in 0.4361 secs.
```
[^1]: https://github.com/etcd-io/etcd/pull/14394#issuecomment-1229606410
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
2022-09-20 09:01:42 +02:00
Tobias Grieger
9ad36eecab
fixup! address comments
...
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
2022-09-20 09:01:42 +02:00
Tobias Grieger
304e260038
raft: benchmark results
...
```
for sha in :/^Revert :/BenchmarkRawNode :/^raft:.directly; do git checkout raft-single-voter && git checkout $(git log -n 1 '--pretty=format:%H' $sha) && f=$(git log -1 --pretty=%s | sed -E 's/[^A-Za-z0-9]+/_/g').txt && go test -run - -count 10 -bench BenchmarkRawNode -benchmem -benchtime=100000x . > $f; done; git checkout raft-single-voter
```
The two possible solutions (directly updating progress and calling
maybeCommit in `(*raft).advance` vs calling `r.Step`) are identical. In
fact, we've gotten a tiny bit better with the `.Step` solution in terms
of not calling `firstIndex` as much, in the common case of not being a
single voter.
```
$ benchstat raft_directly_update_leader_in_advance.txt Revert_raft_directly_update_leader_in_advance_.txt
name old time/op new time/op delta
RawNode/single-voter-10 482ns ± 2% 742ns ± 1% +54.02% (p=0.000 n=9+9)
RawNode/two-voters-10 1.29µs ± 1% 1.31µs ± 2% +1.70% (p=0.000 n=9+10)
name old firstIndex/op new firstIndex/op delta
RawNode/single-voter-10 4.00 ± 0% 5.00 ± 0% +25.00% (p=0.000 n=10+10)
RawNode/two-voters-10 10.0 ± 0% 9.0 ± 0% -10.00% (p=0.000 n=10+10)
name old lastIndex/op new lastIndex/op delta
RawNode/single-voter-10 1.00 ± 0% 2.00 ± 0% +100.00% (p=0.000 n=10+10)
RawNode/two-voters-10 2.00 ± 0% 2.00 ± 0% ~ (all equal)
name old ready/op new ready/op delta
RawNode/single-voter-10 1.00 ± 0% 2.00 ± 0% +100.00% (p=0.000 n=10+10)
RawNode/two-voters-10 2.00 ± 0% 2.00 ± 0% ~ (all equal)
name old term/op new term/op delta
RawNode/single-voter-10 0.00 ± 0% 0.00 ± 0% ~ (all equal)
RawNode/two-voters-10 1.00 ± 0% 1.00 ± 0% ~ (all equal)
name old alloc/op new alloc/op delta
RawNode/single-voter-10 372B ± 0% 388B ± 0% +4.30% (p=0.000 n=10+10)
RawNode/two-voters-10 964B ± 0% 964B ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
RawNode/single-voter-10 4.00 ± 0% 5.00 ± 0% +25.00% (p=0.000 n=10+10)
RawNode/two-voters-10 7.00 ± 0% 7.00 ± 0% ~ (all equal)
```
We then compare the `.Step` solution against the previous "status quo"
that prematurely emitted uncommitted entries for command application
below.
Importantly, we don't regress in the case of multiple peers. We actually
gain slightly in terms of `lastIndex` calls, but run a bit more code;
acceptable.
In the single-voter case, since we now need two Ready handling cycles
per op instead of one, we see additional calls to lastIndex and
firstIndex as well as slightly increased allocations. These are expected
and trade-offs we're willing to make to avoid correctness problems. Note
that the benchmark intentionally forces full processing of each
individual entries, so some of the new overhead would likely amortize on
a singleton voter seeing high throughput as multiple proposals could
share the Ready cycles.
```
$ benchstat raft_add_BenchmarkRawNode.txt Revert_raft_directly_update_leader_in_advance_.txt
name old time/op new time/op delta
RawNode/single-voter-10 482ns ± 2% 742ns ± 1% +54.02% (p=0.000 n=9+9)
RawNode/two-voters-10 1.29µs ± 1% 1.31µs ± 2% +1.70% (p=0.000 n=9+10)
name old firstIndex/op new firstIndex/op delta
RawNode/single-voter-10 4.00 ± 0% 5.00 ± 0% +25.00% (p=0.000 n=10+10)
RawNode/two-voters-10 10.0 ± 0% 9.0 ± 0% -10.00% (p=0.000 n=10+10)
name old lastIndex/op new lastIndex/op delta
RawNode/single-voter-10 1.00 ± 0% 2.00 ± 0% +100.00% (p=0.000 n=10+10)
RawNode/two-voters-10 2.00 ± 0% 2.00 ± 0% ~ (all equal)
name old ready/op new ready/op delta
RawNode/single-voter-10 1.00 ± 0% 2.00 ± 0% +100.00% (p=0.000 n=10+10)
RawNode/two-voters-10 2.00 ± 0% 2.00 ± 0% ~ (all equal)
name old term/op new term/op delta
RawNode/single-voter-10 0.00 ± 0% 0.00 ± 0% ~ (all equal)
RawNode/two-voters-10 1.00 ± 0% 1.00 ± 0% ~ (all equal)
name old alloc/op new alloc/op delta
RawNode/single-voter-10 372B ± 0% 388B ± 0% +4.30% (p=0.000 n=10+10)
RawNode/two-voters-10 964B ± 0% 964B ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
RawNode/single-voter-10 4.00 ± 0% 5.00 ± 0% +25.00% (p=0.000 n=10+10)
RawNode/two-voters-10 7.00 ± 0% 7.00 ± 0% ~ (all equal)
```
`tools/benchmark put`:
```
Summary[main]: | Summary[this PR]:
Total: 284.4443 secs. | Total: 288.1100 secs.
Slowest: 0.1626 secs. | Slowest: 0.1456 secs.
Fastest: 0.0027 secs. | Fastest: 0.0018 secs.
Average: 0.0284 secs. | Average: 0.0288 secs.
Stddev: 0.0178 secs. | Stddev: 0.0182 secs.
Requests/sec: 35.1563 | Requests/sec: 34.7090 [=0.98727681809x main]
Response time histogram: | Response time histogram:
0.0027 [1] | | 0.0018 [1] |
0.0187 [137] | | 0.0162 [34] |
0.0347 [7895] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ | 0.0305 [7938] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
0.0507 [86] | | 0.0449 [103] |
0.0667 [1328] |∎∎∎∎∎∎ | 0.0593 [1056] |∎∎∎∎∎
0.0827 [480] |∎∎ | 0.0737 [420] |∎∎
0.0987 [45] | | 0.0881 [370] |∎
0.1147 [18] | | 0.1025 [48] |
0.1306 [7] | | 0.1168 [19] |
0.1466 [2] | | 0.1312 [6] |
0.1626 [1] | | 0.1456 [5] |
Latency distribution: | Latency distribution:
10% in 0.0195 secs. | 10% in 0.0194 secs.
25% in 0.0198 secs. | 25% in 0.0198 secs.
50% in 0.0201 secs. | 50% in 0.0201 secs.
75% in 0.0210 secs. | 75% in 0.0214 secs.
90% in 0.0585 secs. | 90% in 0.0589 secs.
95% in 0.0727 secs. | 95% in 0.0731 secs.
99% in 0.0762 secs. | 99% in 0.0788 secs.
99.9% in 0.1244 secs. | 99.9% in 0.1240 secs.
```
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
2022-09-20 09:01:42 +02:00
Tobias Grieger
3c3e30a30e
Revert "raft: directly update leader in advance"
...
This reverts commit d73a986e4edb15ef9dbfc994f1cbf5e96694d877, which
was added only for benchmarking purposes.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
2022-09-20 09:01:42 +02:00
Tobias Grieger
67c3522893
raft: directly update leader in advance
...
This makes the alternative option of implementing the leader's self-ack
of entry append the default.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
2022-09-20 09:01:42 +02:00
Tobias Grieger
894e5cb685
move ctx param to the front
...
to appease linter
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
2022-09-20 09:01:42 +02:00
Tobias Grieger
f62b9d5e19
remove TestNodeReadIndex
...
This is tested directly at the level of `RawNode` in
`TestRawNodeReadIndex`. `*node` is a thin wrapper around `RawNode` so
this is sufficient.
The reason to remove the test is that it now incurs data races
since it's not possible to adjust the `readStates` and `step`
fields while the node is running, and there is no primitive
to synchronize with its goroutine. This could all be fixed
but it's not worth it.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
2022-09-20 09:01:40 +02:00
Tobias Grieger
f7dcb9ec2a
TestInteraction
...
Reviewed the diff in detail.
The changes here were benign, just the extra raft cycle.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
2022-09-20 08:59:37 +02:00
Tobias Grieger
f7b0a6ad33
TestRawNodeBoundedLogGrowthWithPartition
...
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
2022-09-20 08:59:37 +02:00
Tobias Grieger
02efe5135d
TestRawNodeStart
...
Now also sees the extra Ready cycle.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
2022-09-20 08:59:37 +02:00
Tobias Grieger
79bf3b0df4
TestRawNodeJointAutoLeave
...
This now needs an additional Ready cycle to apply the previous conf change,
so the finalizing conf change does too.
Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
2022-09-20 08:59:37 +02:00