69 Commits

Author SHA1 Message Date
Wei Fu
32ee8b877a etcdserver: drain leaky goroutines before test completed
If pending changes aren't committed before test completed, it might cause
data race when we don't drain all the background goroutines.

```bash
$ cd server
$ go test -race -v -run TestApplyRepeat ./etcdserver
...
panic: Log in goroutine after TestApplyRepeat has completed: 2024-02-03T17:06:13.262+0800       DEBUG   bbolt   Committing transaction 2

goroutine 81 [running]:
testing.(*common).logDepth(0xc000502820, {0xc0001b0460, 0x41}, 0x3)
        /usr/local/go/src/testing/testing.go:1022 +0x6d4
testing.(*common).log(...)
        /usr/local/go/src/testing/testing.go:1004
testing.(*common).Logf(0xc000502820, {0x1421ad7, 0x2}, {0xc000603520, 0x1, 0x1})
        /usr/local/go/src/testing/testing.go:1055 +0xa5
go.uber.org/zap/zaptest.testingWriter.Write({{0x15f1f90?, 0xc000502820?}, 0xda?}, {0xc000119800, 0x42, 0x400})
        /home/fuwei/go/pkg/mod/go.uber.org/zap@v1.26.0/zaptest/logger.go:130 +0x11e
go.uber.org/zap/zapcore.(*ioCore).Write(0xc0000b55c0, {0xff, {0xc1679e614f9fd7a4, 0x73a3657, 0x1cc2400}, {0x1422b2d, 0x5}, {0xc0001a0330, 0x18}, {0x0, ...}, ...}, ...)
        /home/fuwei/go/pkg/mod/go.uber.org/zap@v1.26.0/zapcore/core.go:99 +0x193
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000115930, {0x0, 0x0, 0x0})
        /home/fuwei/go/pkg/mod/go.uber.org/zap@v1.26.0/zapcore/entry.go:253 +0x2f0
go.uber.org/zap.(*SugaredLogger).log(0xc0001960f8, 0xff, {0x1437885, 0x19}, {0xc0006034e0, 0x1, 0x1}, {0x0, 0x0, 0x0})
        /home/fuwei/go/pkg/mod/go.uber.org/zap@v1.26.0/sugar.go:316 +0x130
go.uber.org/zap.(*SugaredLogger).Debugf(...)
        /home/fuwei/go/pkg/mod/go.uber.org/zap@v1.26.0/sugar.go:171
go.etcd.io/bbolt.(*Tx).Commit(0xc0001aa9a0)
        /home/fuwei/go/pkg/mod/go.etcd.io/bbolt@v1.4.0-alpha.0/tx.go:173 +0x206
go.etcd.io/etcd/server/v3/storage/backend.(*batchTx).commit(0xc00019b180, 0x0)
        /home/fuwei/go/src/go.etcd.io/etcd/server/storage/backend/batch_tx.go:269 +0xdf
go.etcd.io/etcd/server/v3/storage/backend.(*batchTxBuffered).unsafeCommit(0xc00019b180, 0x0)
        /home/fuwei/go/src/go.etcd.io/etcd/server/storage/backend/batch_tx.go:378 +0x425
go.etcd.io/etcd/server/v3/storage/backend.(*batchTxBuffered).commit(0xc00019b180, 0x80?)
        /home/fuwei/go/src/go.etcd.io/etcd/server/storage/backend/batch_tx.go:355 +0x78
go.etcd.io/etcd/server/v3/storage/backend.(*batchTxBuffered).Commit(0xc00019b180)
        /home/fuwei/go/src/go.etcd.io/etcd/server/storage/backend/batch_tx.go:342 +0x35
go.etcd.io/etcd/server/v3/storage/backend.(*backend).run(0xc000478180)
        /home/fuwei/go/src/go.etcd.io/etcd/server/storage/backend/backend.go:426 +0x2c7
created by go.etcd.io/etcd/server/v3/storage/backend.newBackend in goroutine 80
        /home/fuwei/go/src/go.etcd.io/etcd/server/storage/backend/backend.go:227 +0xbfd
FAIL    go.etcd.io/etcd/server/v3/etcdserver    0.129s
FAIL
```

This patch also drains goroutines related to raftNode and watch store.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2024-02-03 18:58:17 +08:00
Neil Shen
9f82390ae9 server: refine TestProcessIgnoreMismatchMessage
Signed-off-by: Neil Shen <overvenus@gmail.com>
2023-12-07 20:32:43 +08:00
Neil Shen
fb769c4306 server: ignore raft messages if member id mismatch
Ignore Raft messages when the `To` field mismatches the local member ID.
In cases where incorrect Raft messages are dispatched, potentially due
to a malfunctioning switch, this proactive check prevents panics,
such as "tocommit is out of range".

Signed-off-by: Neil Shen <overvenus@gmail.com>
2023-12-07 11:57:45 +08:00
Marek Siarkowicz
03d551243b
Merge pull request #17015 from serathius/extract-membership-applier
Extract membership applier
2023-11-27 19:59:21 +01:00
Marek Siarkowicz
e192a05193 Test ApplyConfState after restart
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-11-26 17:40:28 +01:00
Marek Siarkowicz
2ad21558ac Remove shouldApplyV3 from the v3 applier
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-11-24 16:13:25 +01:00
Marek Siarkowicz
093666f450 Cleanup v2 applier
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-11-23 15:41:13 +01:00
Marek Siarkowicz
c72ff1e69c Remove syncing the v2 store TTLs
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-11-23 14:55:01 +01:00
Marek Siarkowicz
ed3375e076 Remove v2 apply logic
v2 store is no longer available in v3.6.
We can remove apply logic for it as they will never be used.

Only v2 PUT is neeeded as it applies to v3 storage and etcd v3.5 uses it for setting member
attributes and cluster version.

Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-11-23 14:13:07 +01:00
Marek Siarkowicz
28d9564962
Merge pull request #16994 from serathius/tests-v2-cluster-version
Add tests for setting cluster version using v2 request
2023-11-22 21:11:18 +01:00
Marek Siarkowicz
2f30760b37 Add tests for setting cluster version using v2 request
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-11-22 18:05:19 +01:00
Marek Siarkowicz
29dd025b84 Stop using v2 requests in server tests
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-11-22 14:41:53 +01:00
Marek Siarkowicz
dd7a4d28a8 Remove code used to make v2 proposals
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-11-19 22:39:33 +01:00
Marek Siarkowicz
b4fd31f254 Remove code for setting cluster version via V2 API
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-11-19 15:28:52 +01:00
Benjamin Wang
34382006db test: implement method ForgetLeader for struct nodeRecorder
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2023-10-24 10:19:13 +01:00
Wei Fu
aea1cd0077 feat: enable unparam lint
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-10-17 21:24:13 +08:00
Geeta Gharpure
70a3205506 fix broken unit test in server_test.go
Signed-off-by: Geeta Gharpure <geetagh@amazon.com>
2023-09-28 20:07:06 +01:00
Wei Fu
aa97484166 *: enable goimports in verify-lint
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-09-21 21:14:09 +08:00
chenyahui
c0aa3b613b Use any instead of interface{}
Signed-off-by: chenyahui <cyhone@qq.com>
2023-09-17 17:41:58 +08:00
Geeta Gharpure
59332dc194 Update to generate v2 snapshot from v3 state
Signed-off-by: Geeta Gharpure <geetagh@amazon.com>
2023-08-21 19:18:11 +00:00
Benjamin Wang
e887e5291a
Merge pull request #16067 from geetasg/pr1
Adding test for updateClusterVersionV3
2023-07-06 08:36:19 +01:00
Chao Chen
6cdc9ae4fe server/etcdserver/raft.go:
1. rename confChangeCh to raftAdvancedC
2. rename waitApply to confChanged
3. add comments and test assertion

Signed-off-by: Chao Chen <chaochn@amazon.com>
2023-06-26 22:42:44 -07:00
Benjamin Wang
ad3b6ee4c6 etcdserver: wait for raft is notified on confChange before responding to client
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2023-06-26 13:40:51 -07:00
Geeta Gharpure
550aa152a7 Verify consistent index is latest at the time of snapshot
Signed-off-by: Geeta Gharpure <geetagh@amazon.com>
2023-06-19 16:00:04 +00:00
Geeta Gharpure
2b81483103 Adding test for version update function used in 3.6
Signed-off-by: Geeta Gharpure <geetagh@amazon.com>
2023-06-13 17:38:04 +00:00
Piotr Tabor
9abc895122 Goimports: Apply automated fixing to test files as well.
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-29 13:04:45 +01:00
Benjamin Wang
faff80a2b3 etcdserve: format the source code
gofmt -w ./server

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-02 13:00:59 +08:00
Benjamin Wang
e9aa275b36 etcdserver: update etcdserver to use the new raft module go.etcd.io/raft/v3
Just replaced all go.etcd.io/etcd/raft/v3 with go.etcd.io/raft/v3
under directory server.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-02 09:33:45 +08:00
Bhargav Ravuri
2feec4fe68 comments: fix comments as per goword in go test files
Comments fixed as per goword in go test files that shell
function go_srcs_in_module lists as per changes on #14827

Helps in #14827

Signed-off-by: Bhargav Ravuri <bhargav.ravuri@infracloud.io>
2022-11-23 23:05:42 +05:30
Nathan VanBenschoten
0f9d7a4f95 raft: make Message.Snapshot nullable, halve struct size
This commit makes the rarely used `raftpb.Message.Snapshot` field nullable.
In doing so, it reduces the memory size of a `raftpb.Message` message from
264 bytes to 128 bytes — a 52% reduction in size.

While this commit does not change the protobuf encoding, it does change
how that encoding is used. `(gogoproto.nullable) = false` instruct the
generated proto marshaling logic to always encode a value for the field,
even if that value is empty. `(gogoproto.nullable) = true` instructs the
generated proto marshaling logic to omit an encoded value for the field
if the field is nil.

This raises compatibility concerns in both directions. Messages encoded
by new binary versions without a `Snapshot` field will be decoded as an
empty field by old binary versions. In other words, old binary versions
can't tell the difference. However, messages encoded by old binary versions
with an empty Snapshot field will be decoded as a non-nil, empty field by
new binary versions. As a result, new binary versions need to be prepared
to handle such messages.

While Message.Snapshot is not intentionally part of the external interface
of this library, it was possible for users of the library to access it and
manipulate it. As such, this change may be considered a breaking change.

Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
2022-11-09 17:35:52 +00:00
demoManito
72cf0cc04a etcd: modify declaring empty slices
declare an empty slice to var s []int replace  s :=[]int{}, https://github.com/golang/go/wiki/CodeReviewComments#declaring-empty-slices

Signed-off-by: demoManito <1430482733@qq.com>
2022-09-16 14:41:14 +08:00
Piotr Tabor
85b18c9b3e Rename WrapApply to Apply. 2022-05-20 14:32:04 +02:00
Piotr Tabor
5097b33ab9 Rename etcdserver/etcderrors package to etcdserver/errors. 2022-05-20 14:32:04 +02:00
Piotr Tabor
63b2f63cc1 Rename package alising "apply2" -> apply. 2022-05-20 14:32:04 +02:00
Piotr Tabor
c78bf655a7 Simplify imports and improve comments. 2022-05-20 14:32:04 +02:00
Piotr Tabor
47a771871b Move apply to its own package (no dependency on etcdserver). 2022-05-20 14:32:04 +02:00
Piotr Tabor
fc6a6c3c27 Move etcdserver/errors.go to sepatate package to avoid cyclic dependencies. 2022-05-20 14:32:04 +02:00
Piotr Tabor
651de5a057 Rename EtcdServer.Id with EtcdServer.MemberId.
It was misleading and error prone vs. ClusterId.
2022-05-20 14:32:04 +02:00
ahrtr
fb2eeb9027 verify consistent_index in snapshot must be equal to the snapshot index
Usually the consistent_index should be greater than the index of the
latest snapshot with suffix .snap. But for the snapshot coming from the
leader, the consistent_index should be equal to the snapshot index.
2022-05-03 20:02:47 +08:00
ahrtr
1b3d6cb0c8 set an separate applyTimeout for the waitAppliedIndex 2022-04-10 14:44:55 +08:00
ahrtr
bfd5170f66 add a txPostLockHook into the backend
Previously the SetConsistentIndex() is called during the apply workflow,
but it's outside the db transaction. If a commit happens between SetConsistentIndex
and the following apply workflow, and etcd crashes for whatever reason right
after the commit, then etcd commits an incomplete transaction to db.
Eventually etcd runs into the data inconsistency issue.

In this commit, we move the SetConsistentIndex into a txPostLockHook, so
it will be executed inside the transaction lock.
2022-04-07 05:35:13 +08:00
Marek Siarkowicz
804fddf921 tests: Use zaptest.NewLogger in tests 2022-04-04 13:03:15 +02:00
Piotr Tabor
f80f477073
Merge pull request #13644 from Juneezee/refactor/t.TempDir
*: use `T.TempDir` to create temporary test directory
2022-02-21 19:52:37 +01:00
Marek Siarkowicz
8c91d60a6f server: Switch to publishV3 2022-02-14 23:06:45 +01:00
Eng Zer Jun
e9edc5432c
*: use T.TempDir to create temporary test directory
The directory created by `T.TempDir()` and is automatically removed when
the test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-01-25 00:00:46 +08:00
Eng Zer Jun
2a151c8982
*: move from io/ioutil to io and os packages
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-28 00:05:28 +08:00
Marek Siarkowicz
83a325ac46 server: Move all functions needed for storage bootstrap to storage package
This is prerequestite to move storage bootstrap, splitted to separate PR
to make it easier to review.
2021-08-03 13:09:15 +02:00
Marek Siarkowicz
a0554a6bd3 etcdserver: Create AuthBackend interface 2021-07-20 18:09:53 +02:00
Marek Siarkowicz
5b6f4579fb server: Rename buckets to schema 2021-07-12 15:37:21 +02:00
Marek Siarkowicz
5e40a8b00c server: Create storage package and move mvcc files to it 2021-07-12 15:37:21 +02:00