892 Commits

Author SHA1 Message Date
Colleen Murphy
27bd78f6ab Update golang.org/x/crypto to latest
Update crypto to address CVE-2022-27191.

The CVE fix is added in 0.0.0-20220315160706-3147a52a75dd but this
change updates to latest.
2022-04-25 09:52:12 -07:00
Piotr Tabor
887f95d0d3
Merge pull request #13963 from ptabor/20220412-verify-assert
Add verification consistent index is (nearly) never decreasing
2022-04-25 10:13:44 +02:00
Tony Abboud
1d13e0733c
Fix logger spelling in NewZapLoggerBuilder godoc
Just a small docs fix that I happened to notice when setting up a new zap logger for etcd
2022-04-22 12:08:54 -04:00
Piotr Tabor
d69e07dd3a Verification framework and check whether cindex is not decreasing. 2022-04-22 12:32:05 +02:00
ahrtr
6eef7ede40 Update conssitent_index when applying fails
When clients have no permission to perform whatever operation, then
the applying may fail. We should also move consistent_index forward
in this case, otherwise the consitent_index may smaller than the
snapshot index.
2022-04-20 21:44:48 +08:00
Piotr Tabor
55a7c91de9 test.sh: Tools run correctly on OsX
- The location of installed tools is found using `go list`
- ETCD_VERIFY env variable is printed in repro command line.
2022-04-17 23:28:26 +02:00
ahrtr
0dae4b3b1e rollback the opentelemetry bumpping to recover the pipeline failures 2022-04-14 16:13:28 +08:00
Will Beason
eab1e0c5d5 go.mod: upgrade opentelemetry deps
Downstream users of etcd experience build issues when using dependencies
which require more recent (incompatible) versions of opentelemetry. This
commit upgrades the dependencies so that downstream users stop
experiencing these issues.
2022-04-13 07:14:10 -07:00
ahrtr
484d2f01f3 set backend to cindex before recovering the lessor in applySnapshot 2022-04-12 10:36:29 +08:00
ahrtr
1b3d6cb0c8 set an separate applyTimeout for the waitAppliedIndex 2022-04-10 14:44:55 +08:00
ahrtr
fe3a57976e support linearizable renew lease
When etcdserver receives a LeaseRenew request, it may be still in
progress of processing the LeaseGrantRequest on exact the same
leaseID. Accordingly it may return a TTL=0 to client due to the
leaseID not found error. So the leader should wait for the appliedID
to be available before processing client requests.
2022-04-10 14:44:55 +08:00
Marek Siarkowicz
12bdd1c5e4
Merge pull request #13910 from serathius/crypto
*: update golang.org/x/crypto
2022-04-09 09:41:49 +02:00
Marek Siarkowicz
7d3ca1f516
Merge pull request #13906 from donalhunt/main
*: drop use of humanize.Time() in favour of zap.Duration and time.Duration
2022-04-08 23:41:50 +02:00
Donal Hunt
d659403955
Update server/etcdserver/api/v3rpc/maintenance.go
Co-authored-by: Marek Siarkowicz <marek.siarkowicz@protonmail.com>
2022-04-08 18:45:13 +01:00
Marek Siarkowicz
1bb59adb1e *: update golang.org/x/crypto 2022-04-08 16:27:52 +02:00
ahrtr
a3650db574 use readTx in (*store).restore 2022-04-08 15:45:05 +08:00
Donal Hunt
4c8ef011e0 *: drop use of humanize.Time() in favour of time.Duration.String()
humanize.Time() drops precision resulting in some events reporting they took
"now" time to complete. Using time.Duration.String() results in accurate
duration being reported.

Fixes #13905
2022-04-07 23:24:35 +01:00
Marek Siarkowicz
1ea53d527e server: Save consistency index and term to backend even when they decrease
Reason to store CI and term in backend was to make db fully independent
snapshot, it was never meant to interfere with apply logic. Skip of CI
was introduced for v2->v3 migration where we wanted to prevent it from
decreasing when replaying wal in
https://github.com/etcd-io/etcd/pull/5391. By mistake it was added to
apply flow during refactor in
https://github.com/etcd-io/etcd/pull/12855#commitcomment-70713670.

Consistency index and term should only be negotiated and used by raft to make
decisions. Their values should only driven by raft state machine and
backend should only be responsible for storing them.
2022-04-07 19:00:03 +02:00
Piotr Tabor
c83b1ad9ba
Merge pull request #13854 from ahrtr/data_corruption
Fix the data inconsistency issue by moving the SetConsistentIndex into the transaction lock
2022-04-07 14:20:19 +02:00
ahrtr
4033f5c2b9 move the consistentIdx and consistentTerm from Etcdserver to cindex package
Removed the fields consistentIdx and consistentTerm from struct EtcdServer,
and added applyingIndex and applyingTerm into struct consistentIndex in
package cindex. We may remove the two fields completely if we decide to
remove the OnPreCommitUnsafe, and it will depend on the performance test
result.
2022-04-07 15:16:49 +08:00
ahrtr
e155e50886 rename LockWithoutHook to LockOutsideApply and add LockInsideApply 2022-04-07 05:35:13 +08:00
ahrtr
47038593e9 set the consistent_index directly when applyV3 isn't performed 2022-04-07 05:35:13 +08:00
ahrtr
7ac995cdde enhanced authBackend to support authReadTx 2022-04-07 05:35:13 +08:00
ahrtr
a4c5da844d added detailed comment to explain the difference between Lock and LockWithoutHook 2022-04-07 05:35:13 +08:00
ahrtr
bfd5170f66 add a txPostLockHook into the backend
Previously the SetConsistentIndex() is called during the apply workflow,
but it's outside the db transaction. If a commit happens between SetConsistentIndex
and the following apply workflow, and etcd crashes for whatever reason right
after the commit, then etcd commits an incomplete transaction to db.
Eventually etcd runs into the data inconsistency issue.

In this commit, we move the SetConsistentIndex into a txPostLockHook, so
it will be executed inside the transaction lock.
2022-04-07 05:35:13 +08:00
Manuel Rüger
f0f77fc14e go.mod: Bump prometheus/client_golang to v1.12.1
Signed-off-by: Manuel Rüger <manuel@rueg.eu>
2022-04-06 19:03:24 +02:00
ls-2018
5b84b30fce fix unexpose todo
Signed-off-by: ls-2018 <acejilam@gmail.com>
2022-04-06 17:38:46 +08:00
Marek Siarkowicz
ad03f2076a
Merge pull request #13886 from serathius/backend-logger
tests: Pass logger to backend
2022-04-05 16:35:07 +02:00
Marek Siarkowicz
73fc864247 tests: Pass logger to backend 2022-04-05 15:53:38 +02:00
Marek Siarkowicz
1d3517020b server: Add verification of whether lock was called within out outside of apply 2022-04-05 15:34:45 +02:00
Piotr Tabor
6c974a3e31
Merge pull request #13867 from serathius/logs-test
tests: Use zaptest.NewLogger in tests
2022-04-04 14:47:04 +02:00
Marek Siarkowicz
804fddf921 tests: Use zaptest.NewLogger in tests 2022-04-04 13:03:15 +02:00
ahrtr
836bd6bc3a fix WARNING: DATA RACE issue when multiple goroutines access the backend concurrently 2022-04-03 06:13:09 +08:00
Marek Siarkowicz
b1610934e3
Merge pull request #13864 from serathius/logs
Fix inconsistent log format
2022-04-01 11:00:48 +02:00
Marek Siarkowicz
63346bfead server: Use default logging configuration instead of zap production one
This fixes problem where logs json changes format of timestamp.
2022-04-01 10:23:42 +02:00
ahrtr
9b3b383366 the file server/storage/mvcc/util.go isn't used at all, so removing it 2022-03-31 10:14:46 +08:00
Kay Yan
afecd3139c fix the api dependency in pkg, and update cobra to 1.4.0
Signed-off-by: Kay Yan <kay.yan@daocloud.io>
2022-03-25 17:18:56 +08:00
ahrtr
f978da4f4f move the newClientCfg into clientv3 package so as to be reused by both etcdctl and v3discovery 2022-03-24 06:18:25 +08:00
Hitoshi Mitake
43e39d362d
Merge pull request #13301 from mitake/jwt-exp-log
server/auth: avoid logging for JWT token
2022-03-23 22:39:28 +09:00
ahrtr
edce939f6e add one more field storageVersion into StatusResponse
When performing the downgrade operation, users can confirm whether each member
is ready to be downgraded using the field 'storageVersion'. If it's equal to the
'target version' in the downgrade command, then it's ready to be downgraded;
otherwise, the etcd member is still in progress of processing the db file.
2022-03-18 07:04:44 +08:00
ahrtr
1a3822f2c3 Rename ClientConfig to ConfigSpec
The ClientConfig is a fully declarive configuration, so it makes more
sense to rename it to ConfigSpec. It can also mitigate the confusion
between Config and ClientConfig.
2022-03-13 05:41:49 +08:00
ahrtr
3dcbbf62d9 Move clientconfig into clientv3 so that it can be reused by both etcdctl and v3 discovery 2022-03-12 06:38:41 +08:00
ahrtr
3ecd8d2364 update incorrect migration log message
It can be upgrade or downgrade, so we should use word "updated" here.
2022-03-10 04:39:56 +08:00
ahrtr
d578a86a59 fix some log format typos 2022-03-10 03:26:34 +08:00
ahrtr
1ae5aa52de fix some typos related to downgrade 2022-03-09 16:07:18 +08:00
Marek Siarkowicz
d2e5a8cb5d
Merge pull request #13750 from kkkkun/add-timeout
add timeout for http client
2022-03-02 10:15:44 +01:00
Piotr Tabor
7c472a964e
Merge pull request #13752 from kkkkun/delete-duplicate-code
mvcc: clean up duplicate metrics name
2022-03-01 17:09:40 +01:00
Piotr Tabor
088807c08e
Merge pull request #13565 from ahrtr/remove_peer_serve_client_requests
Updated servePeers to remove the grpc server
2022-03-01 16:24:42 +01:00
kkkkun
124bc1ebbc mvcc: clean up duplicate metrics name 2022-03-01 18:00:07 +08:00
Marek Siarkowicz
1406a9919c
Merge pull request #13700 from AdamKorcz/fuzz8
server/etcdserver: fix oss-fuzz issue
2022-03-01 10:48:29 +01:00