884 Commits

Author SHA1 Message Date
ahrtr
484d2f01f3 set backend to cindex before recovering the lessor in applySnapshot 2022-04-12 10:36:29 +08:00
ahrtr
1b3d6cb0c8 set an separate applyTimeout for the waitAppliedIndex 2022-04-10 14:44:55 +08:00
ahrtr
fe3a57976e support linearizable renew lease
When etcdserver receives a LeaseRenew request, it may be still in
progress of processing the LeaseGrantRequest on exact the same
leaseID. Accordingly it may return a TTL=0 to client due to the
leaseID not found error. So the leader should wait for the appliedID
to be available before processing client requests.
2022-04-10 14:44:55 +08:00
Marek Siarkowicz
12bdd1c5e4
Merge pull request #13910 from serathius/crypto
*: update golang.org/x/crypto
2022-04-09 09:41:49 +02:00
Marek Siarkowicz
7d3ca1f516
Merge pull request #13906 from donalhunt/main
*: drop use of humanize.Time() in favour of zap.Duration and time.Duration
2022-04-08 23:41:50 +02:00
Donal Hunt
d659403955
Update server/etcdserver/api/v3rpc/maintenance.go
Co-authored-by: Marek Siarkowicz <marek.siarkowicz@protonmail.com>
2022-04-08 18:45:13 +01:00
Marek Siarkowicz
1bb59adb1e *: update golang.org/x/crypto 2022-04-08 16:27:52 +02:00
ahrtr
a3650db574 use readTx in (*store).restore 2022-04-08 15:45:05 +08:00
Donal Hunt
4c8ef011e0 *: drop use of humanize.Time() in favour of time.Duration.String()
humanize.Time() drops precision resulting in some events reporting they took
"now" time to complete. Using time.Duration.String() results in accurate
duration being reported.

Fixes #13905
2022-04-07 23:24:35 +01:00
Marek Siarkowicz
1ea53d527e server: Save consistency index and term to backend even when they decrease
Reason to store CI and term in backend was to make db fully independent
snapshot, it was never meant to interfere with apply logic. Skip of CI
was introduced for v2->v3 migration where we wanted to prevent it from
decreasing when replaying wal in
https://github.com/etcd-io/etcd/pull/5391. By mistake it was added to
apply flow during refactor in
https://github.com/etcd-io/etcd/pull/12855#commitcomment-70713670.

Consistency index and term should only be negotiated and used by raft to make
decisions. Their values should only driven by raft state machine and
backend should only be responsible for storing them.
2022-04-07 19:00:03 +02:00
Piotr Tabor
c83b1ad9ba
Merge pull request #13854 from ahrtr/data_corruption
Fix the data inconsistency issue by moving the SetConsistentIndex into the transaction lock
2022-04-07 14:20:19 +02:00
ahrtr
4033f5c2b9 move the consistentIdx and consistentTerm from Etcdserver to cindex package
Removed the fields consistentIdx and consistentTerm from struct EtcdServer,
and added applyingIndex and applyingTerm into struct consistentIndex in
package cindex. We may remove the two fields completely if we decide to
remove the OnPreCommitUnsafe, and it will depend on the performance test
result.
2022-04-07 15:16:49 +08:00
ahrtr
e155e50886 rename LockWithoutHook to LockOutsideApply and add LockInsideApply 2022-04-07 05:35:13 +08:00
ahrtr
47038593e9 set the consistent_index directly when applyV3 isn't performed 2022-04-07 05:35:13 +08:00
ahrtr
7ac995cdde enhanced authBackend to support authReadTx 2022-04-07 05:35:13 +08:00
ahrtr
a4c5da844d added detailed comment to explain the difference between Lock and LockWithoutHook 2022-04-07 05:35:13 +08:00
ahrtr
bfd5170f66 add a txPostLockHook into the backend
Previously the SetConsistentIndex() is called during the apply workflow,
but it's outside the db transaction. If a commit happens between SetConsistentIndex
and the following apply workflow, and etcd crashes for whatever reason right
after the commit, then etcd commits an incomplete transaction to db.
Eventually etcd runs into the data inconsistency issue.

In this commit, we move the SetConsistentIndex into a txPostLockHook, so
it will be executed inside the transaction lock.
2022-04-07 05:35:13 +08:00
Manuel Rüger
f0f77fc14e go.mod: Bump prometheus/client_golang to v1.12.1
Signed-off-by: Manuel Rüger <manuel@rueg.eu>
2022-04-06 19:03:24 +02:00
ls-2018
5b84b30fce fix unexpose todo
Signed-off-by: ls-2018 <acejilam@gmail.com>
2022-04-06 17:38:46 +08:00
Marek Siarkowicz
ad03f2076a
Merge pull request #13886 from serathius/backend-logger
tests: Pass logger to backend
2022-04-05 16:35:07 +02:00
Marek Siarkowicz
73fc864247 tests: Pass logger to backend 2022-04-05 15:53:38 +02:00
Marek Siarkowicz
1d3517020b server: Add verification of whether lock was called within out outside of apply 2022-04-05 15:34:45 +02:00
Piotr Tabor
6c974a3e31
Merge pull request #13867 from serathius/logs-test
tests: Use zaptest.NewLogger in tests
2022-04-04 14:47:04 +02:00
Marek Siarkowicz
804fddf921 tests: Use zaptest.NewLogger in tests 2022-04-04 13:03:15 +02:00
ahrtr
836bd6bc3a fix WARNING: DATA RACE issue when multiple goroutines access the backend concurrently 2022-04-03 06:13:09 +08:00
Marek Siarkowicz
b1610934e3
Merge pull request #13864 from serathius/logs
Fix inconsistent log format
2022-04-01 11:00:48 +02:00
Marek Siarkowicz
63346bfead server: Use default logging configuration instead of zap production one
This fixes problem where logs json changes format of timestamp.
2022-04-01 10:23:42 +02:00
ahrtr
9b3b383366 the file server/storage/mvcc/util.go isn't used at all, so removing it 2022-03-31 10:14:46 +08:00
Kay Yan
afecd3139c fix the api dependency in pkg, and update cobra to 1.4.0
Signed-off-by: Kay Yan <kay.yan@daocloud.io>
2022-03-25 17:18:56 +08:00
ahrtr
f978da4f4f move the newClientCfg into clientv3 package so as to be reused by both etcdctl and v3discovery 2022-03-24 06:18:25 +08:00
Hitoshi Mitake
43e39d362d
Merge pull request #13301 from mitake/jwt-exp-log
server/auth: avoid logging for JWT token
2022-03-23 22:39:28 +09:00
ahrtr
edce939f6e add one more field storageVersion into StatusResponse
When performing the downgrade operation, users can confirm whether each member
is ready to be downgraded using the field 'storageVersion'. If it's equal to the
'target version' in the downgrade command, then it's ready to be downgraded;
otherwise, the etcd member is still in progress of processing the db file.
2022-03-18 07:04:44 +08:00
ahrtr
1a3822f2c3 Rename ClientConfig to ConfigSpec
The ClientConfig is a fully declarive configuration, so it makes more
sense to rename it to ConfigSpec. It can also mitigate the confusion
between Config and ClientConfig.
2022-03-13 05:41:49 +08:00
ahrtr
3dcbbf62d9 Move clientconfig into clientv3 so that it can be reused by both etcdctl and v3 discovery 2022-03-12 06:38:41 +08:00
ahrtr
3ecd8d2364 update incorrect migration log message
It can be upgrade or downgrade, so we should use word "updated" here.
2022-03-10 04:39:56 +08:00
ahrtr
d578a86a59 fix some log format typos 2022-03-10 03:26:34 +08:00
ahrtr
1ae5aa52de fix some typos related to downgrade 2022-03-09 16:07:18 +08:00
Marek Siarkowicz
d2e5a8cb5d
Merge pull request #13750 from kkkkun/add-timeout
add timeout for http client
2022-03-02 10:15:44 +01:00
Piotr Tabor
7c472a964e
Merge pull request #13752 from kkkkun/delete-duplicate-code
mvcc: clean up duplicate metrics name
2022-03-01 17:09:40 +01:00
Piotr Tabor
088807c08e
Merge pull request #13565 from ahrtr/remove_peer_serve_client_requests
Updated servePeers to remove the grpc server
2022-03-01 16:24:42 +01:00
kkkkun
124bc1ebbc mvcc: clean up duplicate metrics name 2022-03-01 18:00:07 +08:00
Marek Siarkowicz
1406a9919c
Merge pull request #13700 from AdamKorcz/fuzz8
server/etcdserver: fix oss-fuzz issue
2022-03-01 10:48:29 +01:00
kkkkun
59f7764772 add timeout for http client 2022-03-01 11:11:09 +08:00
Marek Siarkowicz
fb55910500 version: bump up to 3.6.0-alpha.0 2022-02-28 13:20:27 +01:00
kkkkun
0ecd4de923 delete duplicate metrics rangeCounterDebug 2022-02-28 16:22:40 +08:00
Piotr Tabor
af7154c1f4
Merge pull request #13704 from kumakichi/main
fix: a little grammar issue
2022-02-25 18:55:45 +01:00
ahrtr
2f36e0c62b Change discovery url to endpoints
Currently the discovery url is just one endpoint. But actually it
should be the same as the etcdctl, which means that it should be
a list of endpoints. When one endpoint is down, the clientv3 can
fail over to the next endpoint automatically.
2022-02-24 09:11:41 +08:00
Marek Siarkowicz
6af760131e
Merge pull request #13687 from serathius/etcdctl
Add downgrade commands
2022-02-22 17:12:23 +01:00
Marek Siarkowicz
42faf9fe06 etcdctl: Use minor versions for downgrade 2022-02-22 16:30:08 +01:00
Marek Siarkowicz
b5e224db7d
Merge pull request #13635 from ahrtr/v3_discovery
support v3 discovery to bootstrap a new etcd cluster
2022-02-21 21:50:40 +01:00