1139 Commits

Author SHA1 Message Date
wafuwafu13
2ffa9e7c91 tests(etcdserver): refactor
Signed-off-by: wafuwafu13 <mariobaske@i.softbank.jp>
2022-12-16 10:09:04 +09:00
wafuwafu13
8dcfca0097 tests(etcdserver): add server_access_control_test.go
Signed-off-by: wafuwafu13 <mariobaske@i.softbank.jp>
2022-12-15 21:46:48 +09:00
Benjamin Wang
c4f7ac28a2 deps: bump go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc from 0.32.0 to 0.37.0 in /server
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-14 04:32:18 +08:00
Benjamin Wang
32840bae73 deps: bump go.opentelemetry.io/otel from 1.7.0 to 1.11.2
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-14 04:27:04 +08:00
Benjamin Wang
8f320bfa00 deps: bump github.com/golang-jwt/jwt/v4 from 4.4.2 to 4.4.3 in /server
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-14 04:19:10 +08:00
Benjamin Wang
2c192f4205 deps: bump go.uber.org/multierr from 1.8.0 to 1.9.0 in /server
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-14 04:00:49 +08:00
Benjamin Wang
c2a7a5870d deps: bump github.com/prometheus/client_golang from 1.12.2 to 1.14.0
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-14 03:42:11 +08:00
Benjamin Wang
bc41c0963b deps: bump github.com/coreos/go-systemd/v22 from 22.3.2 to 22.5.0
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-14 03:38:55 +08:00
Benjamin Wang
259a73d67a deps: bump github.com/spf13/cobra from 1.4.0 to 1.6.1
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-14 03:33:24 +08:00
Benjamin Wang
1a0af6fee6 deps: bump go.uber.org/zap from 1.21.0 to 1.24.0
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-14 03:33:24 +08:00
Benjamin Wang
1ba246e1d8 bump golang.org/x/ to v0.4.0
Found 1 known vulnerability.

Vulnerability #1: GO-2022-1144
  An attacker can cause excessive memory growth in a Go server
  accepting HTTP/2 requests. HTTP/2 server connections contain a
  cache of HTTP header keys sent by the client. While the total
  number of entries in this cache is capped, an attacker sending
  very large keys can cause the server to allocate approximately
  64 MiB per open connection.

  Call stacks in your code:
Error:       tools/etcd-dump-metrics/main.go:158:5: go.etcd.io/etcd/v3/tools/etcd-dump-metrics.main calls go.etcd.io/etcd/server/v3/embed.StartEtcd, which eventually calls golang.org/x/net/http2.Server.ServeConn

  Found in: golang.org/x/net/http2@v0.2.0
  Fixed in: golang.org/x/net/http2@v0.4.0
  More info: https://pkg.go.dev/vuln/GO-2022-1144
Error: Process completed with exit code 3.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-09 08:54:48 +08:00
Wei Fu
f59896c735 chore: use Getter in WarnOfExpensiveReadOnlyTxnRequest
The pb provides an accessor method to get field and it will not panic if
the owner is nil. And add non-empty RangeRespone into the test case.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2022-12-07 17:54:52 +08:00
Benjamin Wang
daad3a2154 etcdserver: fix nil pointer panic for readonly txn
FYI. https://github.com/etcd-io/etcd/issues/14891#issuecomment-1337191993

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-06 14:53:47 +08:00
Benjamin Wang
cc58edecf2 raft: add raft into the tools/mod
Previously etcdservers depends on raft/raftpb/raft.proto directly.
After moving raft to a separate repo, we need to add raft to the
tools/mod, and get raft included in the -I protc flags.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-02 15:36:41 +08:00
Benjamin Wang
394956ca4e doc: cleanup etcd/raft in all documents
TODO:
1. Update Documentation/contributor-guide/modules.svg;
2. Update bill-of-materials.json when raft and raftexample are removed in future;

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-02 14:13:18 +08:00
Benjamin Wang
faff80a2b3 etcdserve: format the source code
gofmt -w ./server

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-02 13:00:59 +08:00
Benjamin Wang
87e1ca2b9a etcdserver: cleanup go.mod and go.sum files
Executed commands below,
1. Removed go.etcd.io/raft/v3 => ../raft;
2. go get go.etcd.io/raft/v3@eaa6808e1f7ab2247c13778250f70520b0527ff1
3. go mod tidy

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-02 12:59:23 +08:00
Benjamin Wang
e9aa275b36 etcdserver: update etcdserver to use the new raft module go.etcd.io/raft/v3
Just replaced all go.etcd.io/etcd/raft/v3 with go.etcd.io/raft/v3
under directory server.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-02 09:33:45 +08:00
Benjamin Wang
3f86db5e53 bump golang.org/x imports to address CVEs
Reference: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-32149

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-30 05:03:43 +08:00
Benjamin Wang
fae9435b66 test: fix unit test Instability
When two members in a 5 member cluster are corrupted, and they
have different hashes, etcd will raise alarm for both members,
but the order isn't guaranteed. But if the two corrupted members
have the same hash, then the order is guaranteed. The leader
always raise alarm in the same order as the member list.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-29 06:51:50 +08:00
Benjamin Wang
319651b784
Merge pull request #14862 from ahrtr/add_log_open_wal_failure_20221128
add more debug info for opening WAL files failure
2022-11-29 06:10:16 +08:00
Benjamin Wang
686846ec65 add more debug info for opening WAL files failure
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-28 18:58:01 +08:00
Benjamin Wang
d545d603e9 test: update both unit test and e2e/integration test for CompactHashCheck
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-26 20:13:20 +08:00
Benjamin Wang
6049af072c etcdserver: intentionally set memberID as 0 when can't identify the corrupted member
If quorum doesn't exist, we don't know which members data are
corrupted. In such situation, we intentionally set the memberID
as 0, it means it affects the whole cluster.
It's align with what we did for 3.4 and 3.5 in
https://github.com/etcd-io/etcd/issues/14849

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-26 19:35:38 +08:00
Benjamin Wang
e95e82f0b9 etcdserver: added a summary for the CompactHashCheck method
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-26 19:35:38 +08:00
Benjamin Wang
85fc09d09b etcdserver: resolve review comments in PR 14828
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-26 19:35:38 +08:00
Benjamin Wang
8b98fee9ce etcdserver: detect corrupted member based on quorum
When the leader detects data inconsistency by comparing hashes,
currently it assumes that the follower is the corrupted member.
It isn't correct, the leader might be the corrupted member as well.

We should depend on quorum to identify the corrupted member.
For example, for 3 member cluster, if 2 members have the same hash,
the the member with different hash is the corrupted one. For 5 member
cluster, if 3 members have the same same, the corrupted member is one
of the left two members; it's also possible that both the left members
are corrupted.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-26 19:35:38 +08:00
Benjamin Wang
285e44378f bump github.com/stretchr/testify from v1.7.2 to v1.8.1
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-26 10:51:32 +08:00
Marek Siarkowicz
f18ac02e0f
Merge pull request #14834 from fuweid/bump-grpc-1.51
bump grpc to v1.51.0 from v1.47.0
2022-11-24 09:07:56 +01:00
Bhargav Ravuri
2feec4fe68 comments: fix comments as per goword in go test files
Comments fixed as per goword in go test files that shell
function go_srcs_in_module lists as per changes on #14827

Helps in #14827

Signed-off-by: Bhargav Ravuri <bhargav.ravuri@infracloud.io>
2022-11-23 23:05:42 +05:30
Wei Fu
cf285ea3c7 bump grpc to v1.51.0 from v1.47.0
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2022-11-23 22:16:22 +08:00
Andrew Sims
f656fa0f49 add missing copyright headers
Signed-off-by: Andrew Sims <andrew.cameron.sims@gmail.com>
2022-11-23 19:13:43 +11:00
Benjamin Wang
ebfcaaed34 test: fix the test failures in e2e/warning_logging_test.go
1. Fixed the test failures which are caused by recent test framework rafactoring;
2. renamed the file to promote_experimental_flag_test.go.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-19 01:51:06 +08:00
Bogdan Kanivets
7e8ebf7727 server: added duplicate warning-unary-request-duration flag
--warning-unary-request-duration is a duplicate of --experimental-warning-unary-request-duration
experimental-warning-unary-request-duration will be removed in v3.7.

fixes https://github.com/etcd-io/etcd/issues/13783

Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
2022-11-18 18:06:00 +08:00
Sasha Melentyev
c3b6cbdb73 all: goimports -w .
Signed-off-by: Sasha Melentyev <sasha@melentyev.io>
2022-11-17 19:07:04 +03:00
Benjamin Wang
8438ddae3e
Merge pull request #14767 from sashamelentyev/print
all: Change Printf and friends to Print if there is no formatting
2022-11-16 10:38:20 +08:00
Sasha Melentyev
2c9c209eb6 all: Changing Printf and friends to Print if there is no formatting
Signed-off-by: Sasha Melentyev <sasha@melentyev.io>
2022-11-15 22:11:23 +03:00
Marek Siarkowicz
f96957adba tests: Add compact failpoints
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-11-15 14:59:03 +01:00
Benjamin Wang
39ca876f38
Merge pull request #14757 from ahrtr/add_failpoint_20221115
etcdserver: add failpoints for backend
2022-11-15 17:14:14 +08:00
Benjamin Wang
4f824336ad etcdserver: add two failpoints for backend
1. before and after create boltDB transaction;
2. before and after writebuf back to read buffer;

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-15 08:09:05 +08:00
Sasha Melentyev
006e747a44 all: Change time unit
Signed-off-by: Sasha Melentyev <sasha@melentyev.io>
2022-11-15 01:15:01 +03:00
Benjamin Wang
3f18816e7d etcdserver: add gofail points before and after OnPreCommitUnsafe
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-14 11:02:18 +08:00
Benjamin Wang
f77b8a735f etcdserver: populate HashRevision when responding to leader or client's HashKV request
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-14 08:33:44 +08:00
Benjamin Wang
5a3ef953eb etcdserver: call the OnPreCommitUnsafe in unsafeCommit
`unsafeCommit` is called by both `(*batchTxBuffered) commit` and
`(*backend) defrag`. When users perform the defragmentation
operation, etcd doesn't update the consistent index. If etcd
crashes(e.g. panicking) in the process for whatever reason, then
etcd replays the WAL entries starting from the latest snapshot,
accordingly it may re-apply entries which might have already been
applied, eventually the revision isn't consistent with other members.

Refer to discussion in https://github.com/etcd-io/etcd/pull/14685

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-11 10:57:15 +08:00
Nathan VanBenschoten
0f9d7a4f95 raft: make Message.Snapshot nullable, halve struct size
This commit makes the rarely used `raftpb.Message.Snapshot` field nullable.
In doing so, it reduces the memory size of a `raftpb.Message` message from
264 bytes to 128 bytes — a 52% reduction in size.

While this commit does not change the protobuf encoding, it does change
how that encoding is used. `(gogoproto.nullable) = false` instruct the
generated proto marshaling logic to always encode a value for the field,
even if that value is empty. `(gogoproto.nullable) = true` instructs the
generated proto marshaling logic to omit an encoded value for the field
if the field is nil.

This raises compatibility concerns in both directions. Messages encoded
by new binary versions without a `Snapshot` field will be decoded as an
empty field by old binary versions. In other words, old binary versions
can't tell the difference. However, messages encoded by old binary versions
with an empty Snapshot field will be decoded as a non-nil, empty field by
new binary versions. As a result, new binary versions need to be prepared
to handle such messages.

While Message.Snapshot is not intentionally part of the external interface
of this library, it was possible for users of the library to access it and
manipulate it. As such, this change may be considered a breaking change.

Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
2022-11-09 17:35:52 +00:00
Marek Siarkowicz
f482f7b4a4
Merge pull request #14705 from serathius/raftapi
raft: Remove dependency on etcd api
2022-11-08 14:52:57 +01:00
Marek Siarkowicz
2a1055c7f3 raft: Remove dependency on etcd api
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-11-08 13:56:46 +01:00
Benjamin Wang
2ac149b96a etcdserver: fix log typo when checking version compatiblity
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-08 18:27:46 +08:00
Benjamin Wang
c967715d93 auth: protect all maintainence APIs when auth is enabled
All maintenance APIs require admin privilege when auth is enabled,
otherwise, the request will be rejected. If auth isn't enabled,
then no such requirement any more.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-03 04:39:42 +08:00
Cenk Alti
580a86ebe5
server: add more context to panic message
Signed-off-by: Cenk Alti <cenkalti@gmail.com>
2022-10-31 20:29:15 -04:00