1201 Commits

Author SHA1 Message Date
Marek Siarkowicz
722ec487df server: Split metrics and health code 2022-05-05 09:52:14 +02:00
Marek Siarkowicz
600ee13ac0 server: Cover V3 health with tests 2022-05-05 09:52:14 +02:00
Marek Siarkowicz
e9dec74ded server: Refactor health checks 2022-05-05 09:52:14 +02:00
Marek Siarkowicz
191aed645e server: Run health check tests in subtests 2022-05-05 09:52:14 +02:00
Marek Siarkowicz
e4e391792a server: Rename test case expect fields 2022-05-05 09:52:13 +02:00
Marek Siarkowicz
0fb194d6f2 server: Use named struct initialization in healthcheck test 2022-05-05 09:52:13 +02:00
Marek Siarkowicz
0096d2ecdb server: Remove unused NewClientHandler 2022-05-05 09:52:13 +02:00
ahrtr
fb2eeb9027 verify consistent_index in snapshot must be equal to the snapshot index
Usually the consistent_index should be greater than the index of the
latest snapshot with suffix .snap. But for the snapshot coming from the
leader, the consistent_index should be equal to the snapshot index.
2022-05-03 20:02:47 +08:00
Colleen Murphy
27bd78f6ab Update golang.org/x/crypto to latest
Update crypto to address CVE-2022-27191.

The CVE fix is added in 0.0.0-20220315160706-3147a52a75dd but this
change updates to latest.
2022-04-25 09:52:12 -07:00
Piotr Tabor
887f95d0d3
Merge pull request #13963 from ptabor/20220412-verify-assert
Add verification consistent index is (nearly) never decreasing
2022-04-25 10:13:44 +02:00
Tony Abboud
1d13e0733c
Fix logger spelling in NewZapLoggerBuilder godoc
Just a small docs fix that I happened to notice when setting up a new zap logger for etcd
2022-04-22 12:08:54 -04:00
Piotr Tabor
d69e07dd3a Verification framework and check whether cindex is not decreasing. 2022-04-22 12:32:05 +02:00
ahrtr
6eef7ede40 Update conssitent_index when applying fails
When clients have no permission to perform whatever operation, then
the applying may fail. We should also move consistent_index forward
in this case, otherwise the consitent_index may smaller than the
snapshot index.
2022-04-20 21:44:48 +08:00
Piotr Tabor
55a7c91de9 test.sh: Tools run correctly on OsX
- The location of installed tools is found using `go list`
- ETCD_VERIFY env variable is printed in repro command line.
2022-04-17 23:28:26 +02:00
ahrtr
0dae4b3b1e rollback the opentelemetry bumpping to recover the pipeline failures 2022-04-14 16:13:28 +08:00
Will Beason
eab1e0c5d5 go.mod: upgrade opentelemetry deps
Downstream users of etcd experience build issues when using dependencies
which require more recent (incompatible) versions of opentelemetry. This
commit upgrades the dependencies so that downstream users stop
experiencing these issues.
2022-04-13 07:14:10 -07:00
ahrtr
484d2f01f3 set backend to cindex before recovering the lessor in applySnapshot 2022-04-12 10:36:29 +08:00
ahrtr
1b3d6cb0c8 set an separate applyTimeout for the waitAppliedIndex 2022-04-10 14:44:55 +08:00
ahrtr
fe3a57976e support linearizable renew lease
When etcdserver receives a LeaseRenew request, it may be still in
progress of processing the LeaseGrantRequest on exact the same
leaseID. Accordingly it may return a TTL=0 to client due to the
leaseID not found error. So the leader should wait for the appliedID
to be available before processing client requests.
2022-04-10 14:44:55 +08:00
Marek Siarkowicz
12bdd1c5e4
Merge pull request #13910 from serathius/crypto
*: update golang.org/x/crypto
2022-04-09 09:41:49 +02:00
Marek Siarkowicz
7d3ca1f516
Merge pull request #13906 from donalhunt/main
*: drop use of humanize.Time() in favour of zap.Duration and time.Duration
2022-04-08 23:41:50 +02:00
Donal Hunt
d659403955
Update server/etcdserver/api/v3rpc/maintenance.go
Co-authored-by: Marek Siarkowicz <marek.siarkowicz@protonmail.com>
2022-04-08 18:45:13 +01:00
Marek Siarkowicz
1bb59adb1e *: update golang.org/x/crypto 2022-04-08 16:27:52 +02:00
ahrtr
a3650db574 use readTx in (*store).restore 2022-04-08 15:45:05 +08:00
Donal Hunt
4c8ef011e0 *: drop use of humanize.Time() in favour of time.Duration.String()
humanize.Time() drops precision resulting in some events reporting they took
"now" time to complete. Using time.Duration.String() results in accurate
duration being reported.

Fixes #13905
2022-04-07 23:24:35 +01:00
Marek Siarkowicz
1ea53d527e server: Save consistency index and term to backend even when they decrease
Reason to store CI and term in backend was to make db fully independent
snapshot, it was never meant to interfere with apply logic. Skip of CI
was introduced for v2->v3 migration where we wanted to prevent it from
decreasing when replaying wal in
https://github.com/etcd-io/etcd/pull/5391. By mistake it was added to
apply flow during refactor in
https://github.com/etcd-io/etcd/pull/12855#commitcomment-70713670.

Consistency index and term should only be negotiated and used by raft to make
decisions. Their values should only driven by raft state machine and
backend should only be responsible for storing them.
2022-04-07 19:00:03 +02:00
Piotr Tabor
c83b1ad9ba
Merge pull request #13854 from ahrtr/data_corruption
Fix the data inconsistency issue by moving the SetConsistentIndex into the transaction lock
2022-04-07 14:20:19 +02:00
ahrtr
4033f5c2b9 move the consistentIdx and consistentTerm from Etcdserver to cindex package
Removed the fields consistentIdx and consistentTerm from struct EtcdServer,
and added applyingIndex and applyingTerm into struct consistentIndex in
package cindex. We may remove the two fields completely if we decide to
remove the OnPreCommitUnsafe, and it will depend on the performance test
result.
2022-04-07 15:16:49 +08:00
ahrtr
e155e50886 rename LockWithoutHook to LockOutsideApply and add LockInsideApply 2022-04-07 05:35:13 +08:00
ahrtr
47038593e9 set the consistent_index directly when applyV3 isn't performed 2022-04-07 05:35:13 +08:00
ahrtr
7ac995cdde enhanced authBackend to support authReadTx 2022-04-07 05:35:13 +08:00
ahrtr
a4c5da844d added detailed comment to explain the difference between Lock and LockWithoutHook 2022-04-07 05:35:13 +08:00
ahrtr
bfd5170f66 add a txPostLockHook into the backend
Previously the SetConsistentIndex() is called during the apply workflow,
but it's outside the db transaction. If a commit happens between SetConsistentIndex
and the following apply workflow, and etcd crashes for whatever reason right
after the commit, then etcd commits an incomplete transaction to db.
Eventually etcd runs into the data inconsistency issue.

In this commit, we move the SetConsistentIndex into a txPostLockHook, so
it will be executed inside the transaction lock.
2022-04-07 05:35:13 +08:00
Manuel Rüger
f0f77fc14e go.mod: Bump prometheus/client_golang to v1.12.1
Signed-off-by: Manuel Rüger <manuel@rueg.eu>
2022-04-06 19:03:24 +02:00
ls-2018
5b84b30fce fix unexpose todo
Signed-off-by: ls-2018 <acejilam@gmail.com>
2022-04-06 17:38:46 +08:00
Marek Siarkowicz
ad03f2076a
Merge pull request #13886 from serathius/backend-logger
tests: Pass logger to backend
2022-04-05 16:35:07 +02:00
Marek Siarkowicz
73fc864247 tests: Pass logger to backend 2022-04-05 15:53:38 +02:00
Marek Siarkowicz
1d3517020b server: Add verification of whether lock was called within out outside of apply 2022-04-05 15:34:45 +02:00
Piotr Tabor
6c974a3e31
Merge pull request #13867 from serathius/logs-test
tests: Use zaptest.NewLogger in tests
2022-04-04 14:47:04 +02:00
Marek Siarkowicz
804fddf921 tests: Use zaptest.NewLogger in tests 2022-04-04 13:03:15 +02:00
ahrtr
836bd6bc3a fix WARNING: DATA RACE issue when multiple goroutines access the backend concurrently 2022-04-03 06:13:09 +08:00
Marek Siarkowicz
b1610934e3
Merge pull request #13864 from serathius/logs
Fix inconsistent log format
2022-04-01 11:00:48 +02:00
Marek Siarkowicz
63346bfead server: Use default logging configuration instead of zap production one
This fixes problem where logs json changes format of timestamp.
2022-04-01 10:23:42 +02:00
ahrtr
9b3b383366 the file server/storage/mvcc/util.go isn't used at all, so removing it 2022-03-31 10:14:46 +08:00
Kay Yan
afecd3139c fix the api dependency in pkg, and update cobra to 1.4.0
Signed-off-by: Kay Yan <kay.yan@daocloud.io>
2022-03-25 17:18:56 +08:00
qsyqian
2ed87b9f2f skip compact when rev not change at period compact mode 2022-03-24 20:59:25 +08:00
ahrtr
f978da4f4f move the newClientCfg into clientv3 package so as to be reused by both etcdctl and v3discovery 2022-03-24 06:18:25 +08:00
Hitoshi Mitake
43e39d362d
Merge pull request #13301 from mitake/jwt-exp-log
server/auth: avoid logging for JWT token
2022-03-23 22:39:28 +09:00
ahrtr
edce939f6e add one more field storageVersion into StatusResponse
When performing the downgrade operation, users can confirm whether each member
is ready to be downgraded using the field 'storageVersion'. If it's equal to the
'target version' in the downgrade command, then it's ready to be downgraded;
otherwise, the etcd member is still in progress of processing the db file.
2022-03-18 07:04:44 +08:00
ahrtr
1a3822f2c3 Rename ClientConfig to ConfigSpec
The ClientConfig is a fully declarive configuration, so it makes more
sense to rename it to ConfigSpec. It can also mitigate the confusion
between Config and ClientConfig.
2022-03-13 05:41:49 +08:00