1161 Commits

Author SHA1 Message Date
Benjamin Wang
a3925d1ed6
Merge pull request #15129 from ahrtr/bump_dependency_20230117
Bump some dependencies
2023-01-17 07:16:46 +08:00
Benjamin Wang
ac98432f0b dependency: bump github.com/dustin/go-humanize from v1.0.0 to v1.0.1
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2023-01-17 06:37:19 +08:00
Benjamin Wang
8ed20e85d2 etcdserver: return membership.ErrIDNotFound when the memberID not found
When promoting a learner, we need to wait until the leader's applied ID
catches up to the commitId. Afterwards, check whether the learner ID
exist or not, and return `membership.ErrIDNotFound` directly in the API
if the member ID not found, to avoid the request being unnecessarily
delivered to raft.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2023-01-17 06:18:15 +08:00
Benjamin Wang
ef917f1efd
Merge pull request #15068 from ahrtr/last_wal_rec_corrupt_main_20220107
etcdserver: process the case of the last WAL record being partially synced to disk [backport 3.5]
2023-01-11 18:06:46 +08:00
Benjamin Wang
30fc7611e2 tidy up dependencies
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2023-01-10 09:11:34 +08:00
dependabot[bot]
d104408036 build(deps): bump golang.org/x/net from 0.4.0 to 0.5.0 in /server
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](https://github.com/golang/net/compare/v0.4.0...v0.5.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-10 09:10:07 +08:00
Benjamin Wang
bd9f1584d4 process the scenaro of the last WAL record being partially synced to disk
We need to return io.ErrUnexpectedEOF in the error chain, so that
etcdserver can repair it automatically.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2023-01-08 04:46:51 +08:00
Chao Chen
2c46b2b299 externalize snapshot catchup entries to etcd flag
Signed-off-by: Chao Chen <chaochn@amazon.com>
2023-01-04 19:01:07 -08:00
Piotr Tabor
8ec3cbc551 fixup! Add --raw mode to ./etcd-dump-log
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-30 09:22:39 +01:00
Piotr Tabor
e571fb7baa Add --raw mode to ./etcd-dump-log
This mode allows to look at RAW protos for all entries in WAL logs in the given directory.

Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-30 09:22:34 +01:00
Piotr Tabor
58681d3feb Expose types of entries within the WAL log for access from the tools.
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-30 09:19:23 +01:00
Piotr Tabor
0d8aad54ba wal: Expose Decoder as package visible interface.
Such that can be used by tools.

Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-30 09:19:23 +01:00
Piotr Tabor
bee2a08968 wal decoding: Add optional mode to ignore CRC errors.
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-30 09:19:23 +01:00
Piotr Tabor
498619bdda wal decoder: report file & offset in case of CRC mismatch.
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-30 09:19:23 +01:00
Piotr Tabor
9abc895122 Goimports: Apply automated fixing to test files as well.
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-29 13:04:45 +01:00
Piotr Tabor
6f899a7b40
Merge pull request #15052 from ptabor/20221228-goimports-fix
./scripts/fix.sh: Takes care of goimports across the whole project.
2022-12-29 11:31:22 +01:00
Piotr Tabor
9e1abbab6e Fix goimports in all existing files. Execution of ./scripts/fix.sh
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-29 09:41:31 +01:00
Wei Fu
4d0b91947e chore: delete // +build buildtag by go fix
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2022-12-29 14:17:05 +08:00
KiloG
101a2a61ea
etcdserver: fix typo in comment
etcdserver: fix typo in comment
2022-12-28 18:41:08 +08:00
Piotr Tabor
04a75b10bf
Merge pull request #14875 from BarrensZeppelin/patch-lessor
lease: fix potential goroutine leak in lessor_test.go
2022-12-19 09:28:50 +01:00
wafuwafu13
2ffa9e7c91 tests(etcdserver): refactor
Signed-off-by: wafuwafu13 <mariobaske@i.softbank.jp>
2022-12-16 10:09:04 +09:00
wafuwafu13
8dcfca0097 tests(etcdserver): add server_access_control_test.go
Signed-off-by: wafuwafu13 <mariobaske@i.softbank.jp>
2022-12-15 21:46:48 +09:00
Benjamin Wang
c4f7ac28a2 deps: bump go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc from 0.32.0 to 0.37.0 in /server
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-14 04:32:18 +08:00
Benjamin Wang
32840bae73 deps: bump go.opentelemetry.io/otel from 1.7.0 to 1.11.2
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-14 04:27:04 +08:00
Benjamin Wang
8f320bfa00 deps: bump github.com/golang-jwt/jwt/v4 from 4.4.2 to 4.4.3 in /server
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-14 04:19:10 +08:00
Benjamin Wang
2c192f4205 deps: bump go.uber.org/multierr from 1.8.0 to 1.9.0 in /server
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-14 04:00:49 +08:00
Benjamin Wang
c2a7a5870d deps: bump github.com/prometheus/client_golang from 1.12.2 to 1.14.0
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-14 03:42:11 +08:00
Benjamin Wang
bc41c0963b deps: bump github.com/coreos/go-systemd/v22 from 22.3.2 to 22.5.0
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-14 03:38:55 +08:00
Benjamin Wang
259a73d67a deps: bump github.com/spf13/cobra from 1.4.0 to 1.6.1
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-14 03:33:24 +08:00
Benjamin Wang
1a0af6fee6 deps: bump go.uber.org/zap from 1.21.0 to 1.24.0
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-14 03:33:24 +08:00
Benjamin Wang
1ba246e1d8 bump golang.org/x/ to v0.4.0
Found 1 known vulnerability.

Vulnerability #1: GO-2022-1144
  An attacker can cause excessive memory growth in a Go server
  accepting HTTP/2 requests. HTTP/2 server connections contain a
  cache of HTTP header keys sent by the client. While the total
  number of entries in this cache is capped, an attacker sending
  very large keys can cause the server to allocate approximately
  64 MiB per open connection.

  Call stacks in your code:
Error:       tools/etcd-dump-metrics/main.go:158:5: go.etcd.io/etcd/v3/tools/etcd-dump-metrics.main calls go.etcd.io/etcd/server/v3/embed.StartEtcd, which eventually calls golang.org/x/net/http2.Server.ServeConn

  Found in: golang.org/x/net/http2@v0.2.0
  Fixed in: golang.org/x/net/http2@v0.4.0
  More info: https://pkg.go.dev/vuln/GO-2022-1144
Error: Process completed with exit code 3.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-09 08:54:48 +08:00
Wei Fu
f59896c735 chore: use Getter in WarnOfExpensiveReadOnlyTxnRequest
The pb provides an accessor method to get field and it will not panic if
the owner is nil. And add non-empty RangeRespone into the test case.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2022-12-07 17:54:52 +08:00
Benjamin Wang
daad3a2154 etcdserver: fix nil pointer panic for readonly txn
FYI. https://github.com/etcd-io/etcd/issues/14891#issuecomment-1337191993

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-06 14:53:47 +08:00
Benjamin Wang
cc58edecf2 raft: add raft into the tools/mod
Previously etcdservers depends on raft/raftpb/raft.proto directly.
After moving raft to a separate repo, we need to add raft to the
tools/mod, and get raft included in the -I protc flags.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-02 15:36:41 +08:00
Benjamin Wang
394956ca4e doc: cleanup etcd/raft in all documents
TODO:
1. Update Documentation/contributor-guide/modules.svg;
2. Update bill-of-materials.json when raft and raftexample are removed in future;

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-02 14:13:18 +08:00
Benjamin Wang
faff80a2b3 etcdserve: format the source code
gofmt -w ./server

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-02 13:00:59 +08:00
Benjamin Wang
87e1ca2b9a etcdserver: cleanup go.mod and go.sum files
Executed commands below,
1. Removed go.etcd.io/raft/v3 => ../raft;
2. go get go.etcd.io/raft/v3@eaa6808e1f7ab2247c13778250f70520b0527ff1
3. go mod tidy

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-02 12:59:23 +08:00
Benjamin Wang
e9aa275b36 etcdserver: update etcdserver to use the new raft module go.etcd.io/raft/v3
Just replaced all go.etcd.io/etcd/raft/v3 with go.etcd.io/raft/v3
under directory server.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-02 09:33:45 +08:00
Oskar Haarklou Veileborg
08dd8d2e0e lease: increase renew timeout to 10s
Signed-off-by: Oskar Haarklou Veileborg <ohv1020@hotmail.com>
2022-12-01 08:57:23 +01:00
Oskar Haarklou Veileborg
0fd40769b7 lease: fix potential goroutine leak in lessor_test.go
Signed-off-by: Oskar Haarklou Veileborg <ohv1020@hotmail.com>
2022-11-30 10:17:22 +01:00
Benjamin Wang
3f86db5e53 bump golang.org/x imports to address CVEs
Reference: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-32149

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-30 05:03:43 +08:00
Benjamin Wang
fae9435b66 test: fix unit test Instability
When two members in a 5 member cluster are corrupted, and they
have different hashes, etcd will raise alarm for both members,
but the order isn't guaranteed. But if the two corrupted members
have the same hash, then the order is guaranteed. The leader
always raise alarm in the same order as the member list.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-29 06:51:50 +08:00
Benjamin Wang
319651b784
Merge pull request #14862 from ahrtr/add_log_open_wal_failure_20221128
add more debug info for opening WAL files failure
2022-11-29 06:10:16 +08:00
Benjamin Wang
686846ec65 add more debug info for opening WAL files failure
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-28 18:58:01 +08:00
Benjamin Wang
d545d603e9 test: update both unit test and e2e/integration test for CompactHashCheck
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-26 20:13:20 +08:00
Benjamin Wang
6049af072c etcdserver: intentionally set memberID as 0 when can't identify the corrupted member
If quorum doesn't exist, we don't know which members data are
corrupted. In such situation, we intentionally set the memberID
as 0, it means it affects the whole cluster.
It's align with what we did for 3.4 and 3.5 in
https://github.com/etcd-io/etcd/issues/14849

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-26 19:35:38 +08:00
Benjamin Wang
e95e82f0b9 etcdserver: added a summary for the CompactHashCheck method
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-26 19:35:38 +08:00
Benjamin Wang
85fc09d09b etcdserver: resolve review comments in PR 14828
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-26 19:35:38 +08:00
Benjamin Wang
8b98fee9ce etcdserver: detect corrupted member based on quorum
When the leader detects data inconsistency by comparing hashes,
currently it assumes that the follower is the corrupted member.
It isn't correct, the leader might be the corrupted member as well.

We should depend on quorum to identify the corrupted member.
For example, for 3 member cluster, if 2 members have the same hash,
the the member with different hash is the corrupted one. For 5 member
cluster, if 3 members have the same same, the corrupted member is one
of the left two members; it's also possible that both the left members
are corrupted.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-26 19:35:38 +08:00
Benjamin Wang
285e44378f bump github.com/stretchr/testify from v1.7.2 to v1.8.1
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-26 10:51:32 +08:00