15851 Commits

Author SHA1 Message Date
Benjamin Wang
9c9148c4cd raft: postpone MsgReadIndex until first commit in the term
Backport https://github.com/etcd-io/etcd/pull/12762 to 3.4

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-07-22 13:56:27 +08:00
Benjamin Wang
f18d074866
Merge pull request #14254 from ramses/backport-13435
[3.4] Backport: non mutating requests pass through quotaKVServer when NOSPACE
2022-07-22 09:27:00 +08:00
Benjamin Wang
aca5cd1717
Merge pull request #14246 from vivekpatani/release-3.4
[3.4] etcdserver,pkg: remove temp files in snap dir when etcdserver starting
2022-07-22 09:14:23 +08:00
vivekpatani
e4deb09c9e etcdserver,pkg: remove temp files in snap dir when etcdserver starting
- Backporting: https://github.com/etcd-io/etcd/pull/12846
- Reference: https://github.com/etcd-io/etcd/issues/14232

Signed-off-by: vivekpatani <9080894+vivekpatani@users.noreply.github.com>
2022-07-21 15:50:27 -07:00
Chao Chen
96f69dee47 Backport: non mutating requests pass through quotaKVServer when NOSPACE
This is a backport of https://github.com/etcd-io/etcd/pull/13435 and is
part of the work for 3.4.20
https://github.com/etcd-io/etcd/issues/14232.

The original change had a second commit that modifies a changelog file.
The 3.4 branch does not include any changelog file, so that part was not
cherry-picked.

Local Testing:

- `make build`
- `make test`

Both succeed.

Signed-off-by: Ramsés Morales <ramses@gmail.com>
2022-07-21 15:06:09 -07:00
Michał Jasionowski
8d83691d53 etcdserver,integration: Store remaining TTL on checkpoint
To extend lease checkpointing mechanism to cases when the whole etcd
cluster is restarted.

Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-07-21 17:35:21 +02:00
Michał Jasionowski
a30aba8fc2 lease,integration: add checkpoint scheduling after leader change
Current checkpointing mechanism is buggy. New checkpoints for any lease
are scheduled only until the first leader change. Added fix for that
and a test that will check it.

Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-07-21 17:35:15 +02:00
Benjamin Wang
7ee7029c08
Merge pull request #14251 from ahrtr/3.4_maxstream
[3.4] Support configuring MaxConcurrentStreams for http2
2022-07-21 17:43:15 +08:00
Benjamin Wang
6071b1c523 Support configuring MaxConcurrentStreams for http2
Backport https://github.com/etcd-io/etcd/pull/14219 to 3.4

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-07-21 14:25:29 +08:00
Benjamin Wang
40ccb8b454
Merge pull request #14240 from chaochn47/cherry-pick-12335
[3.4] etcdserver: add more detailed traces on linearized reading
2022-07-21 03:32:02 +08:00
Chao Chen
864006b72d print out applied index as uint64
Signed-off-by: Chao Chen <chaochn@amazon.com>
2022-07-20 12:07:51 -07:00
Pierre Zemb
3f9fba9112 etcdserver: add more detailed traces on linearized reading
To improve debuggability of `agreement among raft nodes before
linearized reading`, we added some tracing inside
`linearizableReadLoop`.

This will allow us to know the timing of `s.r.ReadIndex` vs
`s.applyWait.Wait(rs.Index)`.

Signed-off-by: Chao Chen <chaochn@amazon.com>
2022-07-20 12:07:51 -07:00
Benjamin Wang
fc76e90cf2
Merge pull request #14230 from mitake/perm-cache-lock-3.4
server/auth: protect rangePermCache with a RW lock
2022-07-20 18:51:54 +08:00
Benjamin Wang
3ea12d352e
Merge pull request #14241 from vivekpatani/release-3.4
clientv3: fix isOptsWithFromKey/isOptsWithPrefix
2022-07-20 18:51:24 +08:00
Benjamin Wang
6313502fb4
Merge pull request #14239 from chaochn47/backport-13676
backport 3.5: #13676 load all leases from backend
2022-07-20 18:50:46 +08:00
Benjamin Wang
b0e1aaef69
Merge pull request #14236 from chrisayoub/release-3.4
[release-3.4] clientv3: filter learners members during autosync
2022-07-20 12:49:31 +08:00
Chris Ayoub
36a76e8531 clientv3: filter learners members during autosync
This change is to ensure that all members returned during the client's
AutoSync are started and are not learners, which are not valid
etcd members to make requests to.

Signed-off-by: Chris Ayoub <cayoub@hubspot.com>
2022-07-20 00:04:03 -04:00
vivekpatani
4fef7fcb90 clientv3: fix isOptsWithFromKey/isOptsWithPrefix
- Addressing: https://github.com/etcd-io/etcd/issues/13332
- Backporting: https://github.com/etcd-io/etcd/pull/13334

Signed-off-by: vivekpatani <9080894+vivekpatani@users.noreply.github.com>
2022-07-19 17:20:56 -07:00
Chao Chen
fd51434b54 backport 3.5: #13676 load all leases from backend
Signed-off-by: Chao Chen <chaochn@amazon.com>
2022-07-19 16:08:01 -07:00
Benjamin Wang
d58a0c0434
Merge pull request #14177 from ahrtr/3.4_lease_renew_linearizable
[3.4] Support linearizable renew lease for 3.4
2022-07-19 16:39:00 +08:00
Hitoshi Mitake
ecd91da40d server/auth: protect rangePermCache with a RW lock
Signed-off-by: Hitoshi Mitake <h.mitake@gmail.com>
2022-07-19 15:51:48 +09:00
Benjamin Wang
07d2b1d626 support linearizable renew lease for 3.4
Cherry pick https://github.com/etcd-io/etcd/pull/13932 to 3.4.

When etcdserver receives a LeaseRenew request, it may be still in
progress of processing the LeaseGrantRequest on exact the same
leaseID. Accordingly it may return a TTL=0 to client due to the
leaseID not found error. So the leader should wait for the appliedID
to be available before processing client requests.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-07-19 13:34:55 +08:00
Benjamin Wang
4636a5fab4 Bump version to 3.4.19
Signed-off-by: Benjamin Wang <wachao@vmware.com>
v3.4.19
2022-07-12 16:18:45 +08:00
Benjamin Wang
06561ae4bf
Merge pull request #14210 from ahrtr/fix_release_script
[3.4] Fix pipeline failure for release test
2022-07-12 16:06:33 +08:00
Benjamin Wang
be0ce4f15b fix pipeline failure for release test
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-07-12 08:31:59 +08:00
Benjamin Wang
d3dfc9b796
Merge pull request #14204 from lavacat/release-3.4-balancer-tests
clientv3/balance: fixed flaky balancer tests
2022-07-12 06:14:35 +08:00
Bogdan Kanivets
185f203528 clientv3/balance: fixed flaky balancer tests
- added verification step to indirectly verify that all peers are in balancer subconn list

Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
2022-07-11 14:43:58 -07:00
Benjamin Wang
7de53273dd
Merge pull request #14205 from ahrtr/3.4_release_script
[3.4] Update release scripts for release-3.4
2022-07-11 20:06:06 +08:00
Benjamin Wang
6cc9416ae5 backport release test to 3.4
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-07-11 19:47:08 +08:00
Benjamin Wang
e6b3d97712 Update release scripts for release-3.4
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-07-11 16:06:32 +08:00
Marek Siarkowicz
852ac37bc0
Merge pull request #14200 from ahrtr/3.4_pipeline_race
set RACE as true for linux-amd64-unit and linux-amd64-grpcproxy
2022-07-08 10:23:21 +02:00
Benjamin Wang
8c1c5fefdb set RACE as true for linux-amd64-unit and linux-amd64-grpcproxy
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-07-08 08:37:31 +08:00
Marek Siarkowicz
0c6063fa82
Merge pull request #14192 from ahrtr/3.4_bump_yaml
[3.4] Bump gopkg.in/yaml.v2 v2.2.2 -> v2.4.0 due to: CVE-2019-11254
2022-07-05 14:32:09 +02:00
Benjamin Wang
860dc149b2 Bump gopkg.in/yaml.v2 v2.2.8 -> v2.4.0 due to: CVE-2019-11254
Cherry pick https://github.com/etcd-io/etcd/pull/13616 to 3.4.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-07-05 06:26:06 +08:00
Marek Siarkowicz
f0256eeec9
Merge pull request #14179 from lavacat/release-3.4-crypto
[backport 3.4] Update golang.org/x/crypto to latest
2022-07-04 11:57:58 +02:00
Bogdan Kanivets
576a798bf9 [backport 3.4] Update golang.org/x/crypto to latest
Update crypto to address CVE-2022-27191.

The CVE fix is added in 0.0.0-20220315160706-3147a52a75dd but this
change updates to latest.

Backport of https://github.com/etcd-io/etcd/pull/13996

Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
2022-06-30 23:08:13 -07:00
Benjamin Wang
bae61786fc
Merge pull request #14183 from ahrtr/3.4_pipeline_issues_20220630
[3.4] Fix pipeline failures in 3.4
2022-07-01 05:36:29 +08:00
Benjamin Wang
8160e9ebe2 disable test cases on certificate-based authentication which isn't supported by gRPC proxy.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-06-30 14:11:54 +08:00
Benjamin Wang
5b3f269159 replace all 3.4 certificates and keys with the files from 3.5
Fix the following error in integration pipeline,
```
=== RUN   TestTLSReloadCopy
    v3_grpc_test.go:1754: tls: failed to find any PEM data in key input
    v3_grpc_test.go:1754: tls: private key does not match public key
    v3_grpc_test.go:1754: tls: private key does not match public key
    v3_grpc_test.go:1754: tls: private key does not match public key
```

Refer to https://github.com/etcd-io/etcd/runs/7123775361?check_suite_focus=true

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-06-30 13:21:48 +08:00
Benjamin Wang
bb9113097a fix test failure in TestCtlV3WatchClientTLS
Also refer to the following commit in 3.5,
093282f5ea

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-06-30 10:19:03 +08:00
Benjamin Wang
f169e5dcba
Merge pull request #14151 from ahrtr/3.4_skip_TestWatchRequestProgress_proxy
[3.4] Skip WatchRequestProgress test in grpc-proxy mode.
2022-06-29 05:40:05 +08:00
Benjamin Wang
6958ee8ff2 Skip WatchRequestProgress test in grpc-proxy mode.
We shouldn't fail the grpc-server (completely) by a not implemented RPC.
Failing whole server by remote request is anti-pattern and security
risk.

Refer to https://github.com/etcd-io/etcd/runs/7034342964?check_suite_focus=true#step:5:2284

```
=== RUN   TestWatchRequestProgress/1-watcher
panic: not implemented
goroutine 83024 [running]:
go.etcd.io/etcd/proxy/grpcproxy.(*watchProxyStream).recvLoop(0xc009232f00, 0x4a73e1, 0xc00e2406e0)
	/home/runner/work/etcd/etcd/proxy/grpcproxy/watch.go:265 +0xbf2
go.etcd.io/etcd/proxy/grpcproxy.(*watchProxy).Watch.func1(0xc0038a3bc0, 0xc009232f00)
	/home/runner/work/etcd/etcd/proxy/grpcproxy/watch.go:125 +0x70
created by go.etcd.io/etcd/proxy/grpcproxy.(*watchProxy).Watch
	/home/runner/work/etcd/etcd/proxy/grpcproxy/watch.go:123 +0x73b
FAIL	go.etcd.io/etcd/clientv3/integration	222.813s
FAIL
```

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-06-29 05:12:43 +08:00
Marek Siarkowicz
f1c59dcfac
Merge pull request #14170 from ahrtr/3.4_proxy_fix_20220628
Fix deadlock in 'go test -tags cluster_proxy -v ./integration/... ./client'
2022-06-28 17:56:44 +02:00
Benjamin Wang
1c9fa07cd7 Fix deadlock in 'go test -tags cluster_proxy -v ./integration/... ./clientv3/...'
Cherry pick https://github.com/etcd-io/etcd/pull/12319 to 3.4.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-06-28 13:44:47 +08:00
Benjamin Wang
4e88cce06c
Merge pull request #14168 from lavacat/release-3.4-TestGetToken
[backport 3.4] clientv3/integration: Reduce flakines of TestGetTokenWithoutAuth
2022-06-28 04:35:17 +08:00
Bogdan Kanivets
2d99b341ad [backport 3.4] clientv3/integration: Reduce flakines of TestGetTokenWithoutAuth
backport from branch-3.5:
https://github.com/etcd-io/etcd/pull/12200/

Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
2022-06-27 11:31:16 -07:00
Marek Siarkowicz
17fc680454
Merge pull request #14150 from ahrtr/lease_revoke_race_3.4
[3.4] Backport two lease related bug fixes to 3.4
2022-06-24 11:27:09 +02:00
Benjamin Wang
f036529b5d Backport two lease related bug fixes to 3.4
The first bug fix is to resolve the race condition between goroutine
and channel on the same leases to be revoked. It's a classic mistake
in using Golang channel + goroutine. Please refer to
https://go.dev/doc/effective_go#channels

The second bug fix is to resolve the issue that etcd lessor may
continue to schedule checkpoint after stepping down the leader role.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-06-24 09:09:40 +08:00
Benjamin Wang
953376e666
Merge pull request #14136 from ahrtr/3.4_pipeline_issues
[3.4] Fix all the pipeline failues for release 3.4
2022-06-23 04:54:42 +08:00
Benjamin Wang
1abf085cfb fix all the pipeline failues for release 3.4
Items resolved:
1. fix the vet error: possible misuse of reflect.SliceHeader;
2. fix the vet error: call to (*T).Fatal from a non-test goroutine;
3. bump package golang.org/x/crypto, net and sys;
4. bump boltdb from 1.3.3 to 1.3.6;
5. remove the vendor directory;
6. remove go 1.12.17 and 1.15.15, add go 1.16.15 into pipeline;
7. bump go version to 1.16 in go.mod;
8. fix the issue: compile: version go1.16.15 does not match go tool version go1.17.11,
   refer to https://github.com/actions/setup-go/issues/107;
9. fix data race on compactMainRev and watcherGauge;
10. fix test failure for TestLeasingTxnOwnerGet in cluster_proxy mode.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-06-22 05:28:45 +08:00