Mirroristas/etcd
2
0
Fork 0
mirror of https://github.com/etcd-io/etcd.git synced 2024-09-27 06:25:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
19,257 Commits 15 Branches 491 Tags 148 MiB
Commit Graph

162 Commits

Author SHA1 Message Date
Benjamin Wang
1fe7b9793a etcdserver: add failpoints walBeforeSync and walAfterSync 
Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2023-02-08 09:08:09 +08:00
guozhao
fab8474ef8 test: add CompactionSleepInterval in FakeStore's config 
After setting the ComparionSleepInterval, we can use time.Ticker
instead of time.After to optimize the scheduleComparison(),
otherwise it will fail in the 'TestStoreCompact(t)' test.

Signed-off-by: guozhao <guozhao@360.cn>
 2023-01-20 06:25:09 +08:00
guozhao
de8d6b3792 etcdserver: use time.Ticker instead of time.After 
Using time.After will create a new Timer in each cycle, In these cases
, it is better to use time.Ticker.

Signed-off-by: guozhao <guozhao@360.cn>
 2023-01-17 16:58:13 +08:00
Benjamin Wang
bd9f1584d4 process the scenaro of the last WAL record being partially synced to disk 
We need to return io.ErrUnexpectedEOF in the error chain, so that
etcdserver can repair it automatically.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2023-01-08 04:46:51 +08:00
Piotr Tabor
8ec3cbc551 fixup! Add --raw mode to ./etcd-dump-log 
Signed-off-by: Piotr Tabor <ptab@google.com>
 2022-12-30 09:22:39 +01:00
Piotr Tabor
e571fb7baa Add --raw mode to ./etcd-dump-log 
This mode allows to look at RAW protos for all entries in WAL logs in the given directory.

Signed-off-by: Piotr Tabor <ptab@google.com>
 2022-12-30 09:22:34 +01:00
Piotr Tabor
58681d3feb Expose types of entries within the WAL log for access from the tools. 
Signed-off-by: Piotr Tabor <ptab@google.com>
 2022-12-30 09:19:23 +01:00
Piotr Tabor
0d8aad54ba wal: Expose Decoder as package visible interface. 
Such that can be used by tools.

Signed-off-by: Piotr Tabor <ptab@google.com>
 2022-12-30 09:19:23 +01:00
Piotr Tabor
bee2a08968 wal decoding: Add optional mode to ignore CRC errors. 
Signed-off-by: Piotr Tabor <ptab@google.com>
 2022-12-30 09:19:23 +01:00
Piotr Tabor
498619bdda wal decoder: report file & offset in case of CRC mismatch. 
Signed-off-by: Piotr Tabor <ptab@google.com>
 2022-12-30 09:19:23 +01:00
Piotr Tabor
9abc895122 Goimports: Apply automated fixing to test files as well. 
Signed-off-by: Piotr Tabor <ptab@google.com>
 2022-12-29 13:04:45 +01:00
Piotr Tabor
6f899a7b40
Merge pull request #15052 from ptabor/20221228-goimports-fix 
./scripts/fix.sh: Takes care of goimports across the whole project.
 2022-12-29 11:31:22 +01:00
Piotr Tabor
9e1abbab6e Fix goimports in all existing files. Execution of ./scripts/fix.sh 
Signed-off-by: Piotr Tabor <ptab@google.com>
 2022-12-29 09:41:31 +01:00
Wei Fu
4d0b91947e chore: delete // +build buildtag by go fix 
Signed-off-by: Wei Fu <fuweid89@gmail.com>
 2022-12-29 14:17:05 +08:00
Benjamin Wang
cc58edecf2 raft: add raft into the tools/mod 
Previously etcdservers depends on raft/raftpb/raft.proto directly.
After moving raft to a separate repo, we need to add raft to the
tools/mod, and get raft included in the -I protc flags.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2022-12-02 15:36:41 +08:00
Benjamin Wang
faff80a2b3 etcdserve: format the source code 
gofmt -w ./server

Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2022-12-02 13:00:59 +08:00
Benjamin Wang
e9aa275b36 etcdserver: update etcdserver to use the new raft module go.etcd.io/raft/v3 
Just replaced all go.etcd.io/etcd/raft/v3 with go.etcd.io/raft/v3
under directory server.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2022-12-02 09:33:45 +08:00
Benjamin Wang
686846ec65 add more debug info for opening WAL files failure 
Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2022-11-28 18:58:01 +08:00
Bhargav Ravuri
2feec4fe68 comments: fix comments as per goword in go test files 
Comments fixed as per goword in go test files that shell
function go_srcs_in_module lists as per changes on #14827

Helps in #14827

Signed-off-by: Bhargav Ravuri <bhargav.ravuri@infracloud.io>
 2022-11-23 23:05:42 +05:30
Andrew Sims
f656fa0f49 add missing copyright headers 
Signed-off-by: Andrew Sims <andrew.cameron.sims@gmail.com>
 2022-11-23 19:13:43 +11:00
Sasha Melentyev
c3b6cbdb73 all: goimports -w . 
Signed-off-by: Sasha Melentyev <sasha@melentyev.io>
 2022-11-17 19:07:04 +03:00
Marek Siarkowicz
f96957adba tests: Add compact failpoints 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2022-11-15 14:59:03 +01:00
Benjamin Wang
4f824336ad etcdserver: add two failpoints for backend 
1. before and after create boltDB transaction;
2. before and after writebuf back to read buffer;

Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2022-11-15 08:09:05 +08:00
Benjamin Wang
3f18816e7d etcdserver: add gofail points before and after OnPreCommitUnsafe 
Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2022-11-14 11:02:18 +08:00
Benjamin Wang
5a3ef953eb etcdserver: call the OnPreCommitUnsafe in unsafeCommit 
`unsafeCommit` is called by both `(*batchTxBuffered) commit` and
`(*backend) defrag`. When users perform the defragmentation
operation, etcd doesn't update the consistent index. If etcd
crashes(e.g. panicking) in the process for whatever reason, then
etcd replays the WAL entries starting from the latest snapshot,
accordingly it may re-apply entries which might have already been
applied, eventually the revision isn't consistent with other members.

Refer to discussion in https://github.com/etcd-io/etcd/pull/14685

Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2022-11-11 10:57:15 +08:00
Marek Siarkowicz
2a1055c7f3 raft: Remove dependency on etcd api 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2022-11-08 13:56:46 +01:00
Cenk Alti
580a86ebe5
server: add more context to panic message 
Signed-off-by: Cenk Alti <cenkalti@gmail.com>
 2022-10-31 20:29:15 -04:00
Benjamin Wang
d15a9d0edc
Merge pull request #14457 from jbml/hashbyrev_compact_main 
etcdserver: fix corruption check when server has just been compacted
 2022-10-13 15:17:38 +08:00
Benjamin Wang
abef537a90
Merge pull request #14515 from spongecaptain/btree-generics 
upate:use google/btree in the genric way
 2022-09-27 16:44:13 +08:00
wathenjiang
319db38b0a update: add benchmark test 
benchmark result:
(1) master branch
$ go test -bench='BenchmarkIndexPut$' -count=5
goos: darwin
goarch: amd64
pkg: go.etcd.io/etcd/server/v3/storage/mvcc
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkIndexPut-12             1000000              2591 ns/op
BenchmarkIndexPut-12             1000000              2531 ns/op
BenchmarkIndexPut-12             1000000              2536 ns/op
BenchmarkIndexPut-12             1000000              2546 ns/op
BenchmarkIndexPut-12             1000000              2538 ns/op
PASS
ok      go.etcd.io/etcd/server/v3/storage/mvcc  167.439s

$ go test -bench='BenchmarkIndexGet$' -count=5
goos: darwin
goarch: amd64
pkg: go.etcd.io/etcd/server/v3/storage/mvcc
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkIndexGet-12             1000000              2021 ns/op
BenchmarkIndexGet-12             1000000              2029 ns/op
BenchmarkIndexGet-12             1000000              2044 ns/op
BenchmarkIndexGet-12             1000000              1973 ns/op
BenchmarkIndexGet-12             1000000              2027 ns/op
PASS
ok      go.etcd.io/etcd/server/v3/storage/mvcc  177.815s

(2) google/btree in the generic way
$ go test -bench='BenchmarkIndexPut$' -count=5
goos: darwin
goarch: amd64
pkg: go.etcd.io/etcd/server/v3/storage/mvcc
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkIndexPut-12             1000000              2477 ns/op
BenchmarkIndexPut-12             1000000              2380 ns/op
BenchmarkIndexPut-12             1000000              2360 ns/op
BenchmarkIndexPut-12             1000000              2396 ns/op
BenchmarkIndexPut-12             1000000              2382 ns/op
PASS
ok      go.etcd.io/etcd/server/v3/storage/mvcc  165.841s

$ go test -bench='BenchmarkIndexGet$' -count=5
goos: darwin
goarch: amd64
pkg: go.etcd.io/etcd/server/v3/storage/mvcc
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkIndexGet-12             1000000              1985 ns/op
BenchmarkIndexGet-12             1000000              1914 ns/op
BenchmarkIndexGet-12             1000000              1900 ns/op
BenchmarkIndexGet-12             1000000              1905 ns/op
BenchmarkIndexGet-12             1000000              1894 ns/op
PASS
ok      go.etcd.io/etcd/server/v3/storage/mvcc  177.573s

Signed-off-by: wathenjiang <wathenjiang@tencent.com>
 2022-09-27 14:33:02 +08:00
Spongecaptain
c53dfc7c5b upate:use google/btree in the genric way 
Signed-off-by: wathenjiang <wathenjiang@tencent.com>
 2022-09-27 10:16:15 +08:00
Kafuu Chino
f1d4935e91 *: avoid closing a watch with ID 0 incorrectly 
Signed-off-by: Kafuu Chino <KafuuChinoQ@gmail.com>

add test
 2022-09-26 20:30:33 +08:00
Jeremy Leach
cc1e245368 etcdserver: fix corruption check when server has just been compacted 
When a key-value store corruption check happens immediately after a
compaction, the revision at which the key-value store hash is computed,
is the compacted revision itself.
In that case, the hash computation logic was incorrect because it
returned an ErrCompacted error; this error should instead be returned when
the revision at which the key-value store is hashed, is strictly lower
than the compacted revision.

Fixes #14325

Signed-off-by: Jeremy Leach <44558776+jbml@users.noreply.github.com>
 2022-09-24 22:20:26 +10:00
SimFG
5702765729 wal: Fix the walWriteBytes metric 
Signed-off-by: SimFG <1142838399@qq.com>
 2022-09-22 19:23:06 +08:00
Benjamin Wang
7f10dccbaf Bump go 1.19: update all the dependencies and go.sum files 
1. run ./scripts/fix.sh;
2. cd tools/mod; gofmt -w . & go mod tidy;

Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2022-09-22 08:47:46 +08:00
demoManito
a9c3d56508 etcd: remove redundant type conversion 
Signed-off-by: demoManito <1430482733@qq.com>
 2022-09-20 11:26:02 +08:00
demoManito
72cf0cc04a etcd: modify declaring empty slices 
declare an empty slice to var s []int replace  s :=[]int{}, https://github.com/golang/go/wiki/CodeReviewComments#declaring-empty-slices

Signed-off-by: demoManito <1430482733@qq.com>
 2022-09-16 14:41:14 +08:00
lovehhf
3b585e94fc mvcc: Remove unused revisions and change comment rev to modified 
Signed-off-by: Hongfei Huang <853885165@qq.com>
 2022-09-14 23:36:54 +08:00
Vladimir Sokolov
a4f140c9fa testing: fix TestOpenWithMaxIndex cleanup 
A WAL object was closed by defer, however the WAL was rewritten afterwards,
so defer closed already closed WAL but not the new one. It caused a data
race between writing file and cleaning up a temporary test directory,
which led to a non-deterministic bug.

Fixes #14332

Signed-off-by: Vladimir Sokolov <vsvastey@gmail.com>
 2022-09-07 21:30:16 +03:00
Abirdcfly
08a9d1da07
chore: remove duplicate word in comments 
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
 2022-08-27 13:39:48 +08:00
Benjamin Wang
fff5d00ccf
Merge pull request #14149 from lavacat/main-txn-panic 
server: don't panic in readonly serializable txn
 2022-08-14 05:41:57 +08:00
Bogdan Kanivets
43bb9d5c22 server: don't panic in readonly serializable txn 
Problem: We pass grpc context down to applier in readonly serializable txn.
This context can be cancelled for example due to timeout.
This will trigger panic inside applyTxn

Solution: Only panic for transactions with write operations

fixes https://github.com/etcd-io/etcd/issues/14110

Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
 2022-08-09 00:46:50 -07:00
Benjamin Wang
3dd7d3f9af enhance the WAL file related error 
The `ErrFileNotFound` was used for for three cases:
1. There is no any WAL files (probably due to no read permission);
2. There is no WAL files matching the snapshot index;
3. The WAL file seqs do not increase continuously.

It's not good for debug when users see the `ErrFileNotFound` error,
so in this PR, a different error is returned for each case above.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2022-08-03 05:37:30 +08:00
Marek Siarkowicz
6697fca97d server: Implement compaction hash checking 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2022-07-26 09:31:14 +02:00
Marek Siarkowicz
264498258b tests: Move CorruptBBolt to testutil 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2022-07-25 13:59:30 +02:00
Benjamin Wang
33347b6845
Merge pull request #13954 from mitake/perm-cache-lock 
server/auth: protect rangePermCache with a mutex
 2022-07-03 07:01:46 +08:00
Hitoshi Mitake
de09174a3f server/auth: protect rangePermCache with a RW lock 
Signed-off-by: Hitoshi Mitake <h.mitake@gmail.com>
 2022-07-02 23:23:13 +09:00
Benjamin Wang
e0998f49e1
Merge pull request #14124 from wayblink/index-op 
mvcc:add ut for Revisions/CountRevisions and remove RangeSince as it …
 2022-06-21 12:41:25 +08:00
wayblink
428d21f5eb mvcc:add ut for Revisions/CountRevisions and remove RangeSince as it is not used 
Signed-off-by: wayblink <wayasxxx@gmail.com>
 2022-06-21 11:41:44 +08:00
Benjamin Wang
8038e876d3 replace ioutil with os package 
Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2022-06-17 10:11:30 +08:00