Piotr Tabor
85341e08f2
Merge pull request #12968 from serathius/logger-simplify
...
server: Simplify passing logger setup by passing only logger
2021-05-15 15:58:00 +02:00
Marek Siarkowicz
41ed74824e
server: Simplify passing logger setup by passing only logger
2021-05-14 13:14:48 +02:00
Piotr Tabor
ab586cd463
Persists Term in the (bbolt) Backend.
...
Additional layer of protection, that allows to validate whether we
start replaying log not only from the proper 'index', but also of the
right 'term'.
2021-05-13 21:29:01 +02:00
Piotr Tabor
e44fb40be5
Merge pull request #12962 from ptabor/20210513-write-conf-state
...
Save raftpb.ConfState in the backend.
2021-05-13 19:22:28 +02:00
Gyuho Lee
e2d67f2e3b
Merge pull request #12956 from gyuho/rename-to-main
...
*: rename "master" branch references to "main" in source code
2021-05-13 08:26:33 -07:00
Piotr Tabor
865df75714
Save raftpb.ConfState in the backend.
...
This makes (bbolt) backend a full feature snapshot in term of WAL/raft,
i.e. carries:
- commit : (applied_index)
- confState
Benefits:
- Backend will be a sufficient point in time definition sufficient to
start replaying WAL. We have applied_index & confState in consistent
state.
- In case of emergency a backend state can be used for recovery
2021-05-13 14:29:36 +02:00
Piotr Tabor
3cb1ba4b2b
Merge pull request #12954 from serathius/logger-new-ctx-client
...
client: Add logger argument to NewCtxClient
2021-05-13 09:03:38 +02:00
Gyuho Lee
77c8033739
server: rename "master" branch references
...
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2021-05-12 10:37:35 -07:00
Piotr Tabor
ead81df948
Disallow -v2-deprecation>'not-yet' combined with --enable-v2
2021-05-12 18:09:34 +02:00
Piotr Tabor
f3b4a3e578
Detecting whether v2store is "empty" (metadata only).
2021-05-12 18:09:34 +02:00
Marek Siarkowicz
1189ee3f3d
client: Add logger argument to NewCtxClient
2021-05-12 16:40:55 +02:00
Marek Siarkowicz
efc8505739
etcdserver: Implement running defrag if freeable space will exceed privided threshold
2021-05-11 14:00:29 +02:00
Lili Cosic
1a718a958e
Add initial Tracing with OpenTelemetry
2021-05-10 10:44:40 +02:00
Piotr Tabor
aeb9b5fc73
Merge pull request #12855 from ptabor/20210409-backend-hooks
...
(no)StoreV2 (Part 4): Backend hooks: precommit updates consistency_index
2021-05-08 09:34:31 +02:00
Gyuho Lee
344c9f3930
Merge pull request #12896 from wilsonwang371/profiling-txn2
...
server: make applier use ReadTx() in Txn() instead of ConcurrentReadTx()
2021-05-06 01:59:14 -07:00
tangcong
b32bc914ff
learner support snapshot RPC ( #12890 )
...
* learner support snapshot RPC
* CHANGELOG: update for 12890
2021-05-04 13:26:33 -07:00
Wilson Wang
98083ea914
server: add experimental flag for using shared buffer in transacton write
2021-05-04 11:59:08 -07:00
Piotr Tabor
2fb6f0a74b
Simplify lease management after cindex update is moved to 'hooks'.
2021-05-04 18:21:23 +02:00
Piotr Tabor
2dbecea5b2
Simplify KVStore interaction with cindex thanks to hooks.
2021-05-04 18:21:23 +02:00
Piotr Tabor
fe3254aee3
Remove explicit authStore->ConsistencyIndex updates, as they are taken care by hook.
2021-05-04 15:38:23 +02:00
Piotr Tabor
50051675f9
Integrate backend::hooks with consistent_index.
...
Every transaction committed to backend is writing most recent consistent_index.
Makes sure that even automatically trigger commits of batch-transactions
stays "really" consistent a.d. the most recent WAL log index applied.
2021-05-04 15:38:23 +02:00
Piotr Tabor
a3fc535a5a
Merge pull request #12914 from ptabor/20210430-read-membership
...
No-storeV2: Read membership information from the backend (Part5)
2021-05-04 10:15:01 +02:00
Piotr Tabor
451f65d661
Merge pull request #12908 from ptabor/20210429-client-retry-logging
...
Clientv3 (retry interceptor) logs should use the configured logger
2021-04-29 19:25:04 +02:00
Piotr Tabor
cedbea6c81
Merge pull request #12904 from wpedrak/limit_mlocked_memory
...
server: replace mlockall with `Mlock` in `--experimental-memory-mlock`
2021-04-29 18:21:24 +02:00
Piotr Tabor
ffea1537d4
ClientV3 tests use integration.NewClient that configures proper logger.
2021-04-29 18:18:34 +02:00
Piotr Tabor
205a1a442a
Read membership information from the backend.
2021-04-29 13:45:45 +02:00
wpedrak
1145c57601
server: replace mlockall with Mlock in --experimental-memory-mlock
...
Implementation of `--experimental-memory-mlock` backed by `mlockall` syscall is replaced by `Mlock` flag (backed by mlock syscall) of bboltDB.
2021-04-29 12:08:52 +02:00
wpedrak
927b3a3152
server: replace mlockall with Mlock in --experimental-memory-mlock
...
Implementation of `--experimental-memory-mlock` backed by `mlockall` syscall is replaced by `Mlock` flag (backed by mlock syscall) of bboltDB.
2021-04-29 12:08:20 +02:00
Piotr Tabor
e90504fe62
Unify shared code (and constants) with cindex package.
2021-04-29 11:51:25 +02:00
Piotr Tabor
f53b70facb
Embed: In case KVStoreHash verification fails, close the backend.
...
In case of failed verification, the server used to keep opened backend
(so the file was locked on OS level).
2021-04-29 11:51:25 +02:00
Piotr Tabor
911204cd76
Fix ETCDCTL_API=2 etcdctl backup --with-v3 consistent index consistency
...
Prior to this CL, `ETCDCTL_API=2 etcdctl backup --with-v3` was readacting WAL log
(by removal of some entries), but was NOT updating consistent_index in the backend.
Also the WAL editing logic was buggy, as it didn't took in consideration the fact
that when TERM changes, there can be entries with duplicated indexes in
the log. So its NOT sufficient to subtract number of removed entries to
get accurate log indexes.
The PR replaces removing and shifting of WAL entries with replacing them with an no-op entries.
Thanks to this consistent-index references are staying up to date.
The PR also:
- updates 'verification' logic to check whether consistent_index does not lag befor last snapshot
- env-gated execution of verification framework in `etcdctl backup`.
Tested with:
```
(./build.sh && cd tests && EXPECT_DEBUG=TRUE 'env' 'go' 'test' '-timeout=300m' 'go.etcd.io/etcd/tests/v3/e2e' -run=TestCtlV2Backup --count=1000 2>&1 | tee TestCtlV2BackupV3.log)
```
2021-04-29 11:51:24 +02:00
Wilson Wang
8d8d0377a2
server: applier uses ReadTx instead of ConcurrentTx
2021-04-28 11:06:24 -07:00
Piotr Tabor
067521981e
v2 etcdctl backup: producing consistent state of membership
2021-04-27 19:34:34 +02:00
Piotr Tabor
a70386a1a4
Simplify membership interface: Does not pass the 'unused' token.
2021-04-27 17:17:31 +02:00
Piotr Tabor
7ae3d25f91
Membership: Add additional methods to trim/manage membership data in backend.
2021-04-27 17:17:31 +02:00
Piotr Tabor
768da490ed
sever: v2store deprecation: Fix etcdctl snapshot restore to restore
...
correct 'backend' (bbolt) context in aspect of membership.
Prior to this change the 'restored' backend used to still contain:
- old memberid (mvcc deletion used, why the membership is in bolt
bucket, but not mvcc part):
```
mvs := mvcc.NewStore(s.lg, be, lessor, ci, mvcc.StoreConfig{CompactionBatchLimit: math.MaxInt32})
defer mvs.Close()
txn := mvs.Write(traceutil.TODO())
btx := be.BatchTx()
del := func(k, v []byte) error {
txn.DeleteRange(k, nil)
return nil
}
// delete stored members from old cluster since using new members
btx.UnsafeForEach([]byte("members"), del)
```
- didn't get new members added.
2021-04-27 17:17:30 +02:00
Piotr Tabor
9a4b2bdccc
Errors: context cancelled or context deadline exceeded are exposed as codes.Canceled, codes.DeadlineExceeded instead of 'codes.Unknown'
2021-04-22 14:35:24 +02:00
Piotr Tabor
d7d110b5a8
mvcc/backend tests: Refactor: Do not mix testing&prod code.
2021-04-21 09:43:13 +02:00
Piotr Tabor
ea287dd9f8
Merge pull request #12854 from ptabor/20210410-shouldApplyV3
...
(no)StoreV2 (Part 3): Applying consistency fix: ClusterVersionSet (and co) might get not applied on v2store
2021-04-21 09:31:38 +02:00
Sam Batschelet
0f2c940f64
Merge pull request #12880 from chaochn47/exclude_alarms_from_health_check
...
etcdhttp/metrics.go: exclude alarms from health check conditionally with `?exclude=NOSPACE`
2021-04-20 21:18:15 -04:00
Chao Chen
140ea4fa29
etcdhttp/metrics.go: exclude alarms from health check conditionally with ?exclude=NOSPACE
2021-04-20 13:17:09 -07:00
Piotr Tabor
06ba0fc5a2
Merge pull request #12846 from pyiyun/fix-snaptmpfile-bug
...
etcdserver: remove temp files in snap dir when etcdserver starting
2021-04-17 12:58:46 +02:00
chao
80586c5b47
etcdserver/util.go: reduce memory when logging range requests ( #12871 )
...
* etcdserver/util.go: reduce memory when logging range requests
Fixes #12835
* Update CHANGELOG-3.5.md
2021-04-16 16:39:34 -07:00
Piotr Tabor
17b982382e
Fix TestSnapshotV3RestoreMultiMemberAdd flakes (leaks)
...
- most important: unix's socket transport should not keep idle
connections. For top-level Transport we close them using:
f3c518025e/server/etcdserver/api/rafthttp/transport.go (L226)
2021-04-16 20:17:28 +02:00
pyiyun
28a490b09c
etcdserver: remove temp files in snap dir when etcdServer starting
...
When etcd exits abnormally, tmp files will remain in snap dir, so clean up tmp files in snap dir when etcdserver starting.
Fixes #12837
2021-04-16 20:30:04 +08:00
Piotr Tabor
d69e46ea47
Make ShouldApplyV3 an enum - not bool
2021-04-13 23:01:03 +02:00
Piotr Tabor
b1c04ce043
Applying consistency fix: ClusterVersionSet (and co) might get no applied on v2store
...
ClusterVersionSet, ClusterMemberAttrSet, DowngradeInfoSet functions are
writing both to V2store and backend. Prior this CL there were
in a branch not executed if shouldApplyV3 was false,
e.g. during restore when Backend is up-to-date (has high
consistency-index) while v2store requires replay from WAL log.
The most serious consequence of this bug was that v2store after restore
could have different index (revision) than the same exact store before restore,
so potentially different content between replicas.
Also this change is supressing double-applying of Membership
(ClusterConfig) changes on Backend (store v3) - that lackilly are not
part of MVCC/KeyValue store, so they didn't caused Revisions to be
bumped.
Inspired by jingyih@ comment:
https://github.com/etcd-io/etcd/pull/12820#issuecomment-815299406
2021-04-12 09:43:48 +02:00
wpedrak
3991a8c9fa
etcdserver: replace forceVersionC with FirstCommitInTermNotify
2021-04-09 11:30:42 +02:00
wpedrak
3d485faac5
etcdserver: resend ReadIndex request on empty apply request
...
Empty apply indicates first commit in current term. It is first time when follower is sure, that it's ReadIndex request can be processed.
2021-04-09 11:30:42 +02:00
Piotr Tabor
cc0f812f51
server_test.go: Use context.Background() instead of TODO in tests.
...
Suggected in: https://golang.org/pkg/context/#Background
2021-04-08 01:15:16 +02:00
