56 Commits

Author SHA1 Message Date
Marek Siarkowicz
bf3e7033e9 etcdserver: Move Read/Update methods on Meta bucket to one place
There are still some left like compact keys, but they will require more
work to avoid circular dependency.
2021-07-05 13:23:53 +02:00
Piotr Tabor
ac0243bd70
Merge pull request #13160 from serathius/compact
etcdserver: Extract functions for setting and reading compaction information in backend
2021-07-03 11:37:09 +02:00
Piotr Tabor
1208505290
Merge pull request #13161 from serathius/membership
etcdserver: Membership uses MembershipStorage interface instead of directly accessing Backend
2021-07-03 11:33:38 +02:00
Piotr Tabor
33b2cdb957
Merge pull request #13162 from serathius/auth
etcdserver: Move read/update methods on Auth bucket to one place
2021-07-03 11:33:07 +02:00
Piotr Tabor
a8e6016fc6
Merge pull request #13163 from serathius/authusers
etcdserver: Move all get/put/delete on AuthUsers and AuthRoles to buckets module
2021-07-03 11:32:24 +02:00
Piotr Tabor
6825b4da61
Merge pull request #13164 from serathius/alarm
etcdserver: Move put/read/delete on Alarm bucket to bucket package
2021-07-03 11:31:37 +02:00
Marek Siarkowicz
e5a026822b etcdserver: Move put/read/delete on Alarm bucket to bucket package 2021-07-01 13:35:10 +02:00
Marek Siarkowicz
9e511665c5 etcdserver: Move get/put/delete on Lease bucket to bucket package 2021-07-01 13:34:45 +02:00
Marek Siarkowicz
0c701fb9f3 etcdserver: Move all get/put/delete on AuthUsers and AuthRoles to buckets module 2021-07-01 12:12:15 +02:00
Marek Siarkowicz
221c887819 etcdserver: Extract functions for setting and reading compaction information in backend 2021-07-01 11:26:27 +02:00
Marek Siarkowicz
b2e08fbfd4 etcdserver: Move read/update methods on Auth bucket to one place 2021-06-29 18:02:11 +02:00
Marek Siarkowicz
50507d5f3c etcdserver: Membership uses MembershipStorage interface instead of directly accessing Backend 2021-06-29 16:14:06 +02:00
Marek Siarkowicz
f79d09d48b etcdserver: Move all named keys to buckets module 2021-06-28 16:40:50 +02:00
Marek Siarkowicz
e2740b4afa server,etcdutl: Preserve etcd version in backend allowing etcdutl to read it from snapshot 2021-06-25 14:06:56 +02:00
Marek Siarkowicz
182aef6e6b etcdserver: Fix invalid count returned on Range with Limit 2021-06-01 15:04:28 +02:00
Piotr Tabor
71934ff244
Merge pull request #13010 from ipixiu/imp/store-scheduleCompaction-lock
limit tx lock scope in store.scheduleCompaction
2021-05-26 08:51:44 +02:00
Sahdev Zala
05674c87fb
Merge pull request #13018 from ipixiu/feature/bolt-compaction-sleep-interval
add bolt compaction sleep interval
2021-05-25 10:06:47 -04:00
Wilson Wang
9c82e8c72b server: set multiple concurrentReadTx instances share one txReadBuffer. 2021-05-24 15:16:53 -07:00
AlexStocks
184b0e5d49 add sleep interval 2021-05-24 16:22:00 +08:00
AlexStocks
6d68f7841a limit tx lock scope 2021-05-19 23:47:46 +08:00
Piotr Tabor
46b49a6ecf
Merge pull request #12991 from ptabor/20210519-mlmhl-buffer-seq
Represent bucket as object instead of []byte name.
2021-05-18 19:32:01 +02:00
Piotr Tabor
66752fef2f Represent bucket as object instead of []byte name.
Thanks to this change:
  - all the maps bucket -> buffer are indexed by int's instead of
string. No need to do: byte[] -> string -> hash conversion on each
access.
  - buckets are strongly typed in backend/mvcc API.
2021-05-18 18:58:53 +02:00
Piotr Tabor
62b731909f
Merge pull request #12587 from mlmhl/buffer-search
backend: fix buffer range bug
2021-05-18 10:54:30 +02:00
Piotr Tabor
eebe67d87d Rename seq to bucket2seq. 2021-05-17 17:29:44 +02:00
Piotr Tabor
4a2ffc2cbe Minor formatting fix on top of https://github.com/etcd-io/etcd/pull/12568 2021-05-17 17:08:00 +02:00
mlmhl
79eafb9719 backend: set seq flag for each bucket buffer 2021-05-17 17:08:00 +02:00
Piotr Tabor
d99d0df5a5 Adding etcdutl test coverage. 2021-05-17 11:54:03 +02:00
Piotr Tabor
ab586cd463 Persists Term in the (bbolt) Backend.
Additional layer of protection, that allows to validate whether we
start replaying log not only from the proper 'index', but also of the
right 'term'.
2021-05-13 21:29:01 +02:00
Piotr Tabor
865df75714 Save raftpb.ConfState in the backend.
This makes (bbolt) backend a full feature snapshot in term of WAL/raft,
i.e. carries:
  - commit : (applied_index)
  - confState

Benefits:
  - Backend will be a sufficient point in time definition sufficient to
start replaying WAL. We have applied_index & confState in consistent
state.
  - In case of emergency a backend state can be used for recovery
2021-05-13 14:29:36 +02:00
Piotr Tabor
aeb9b5fc73
Merge pull request #12855 from ptabor/20210409-backend-hooks
(no)StoreV2 (Part 4): Backend hooks:  precommit updates consistency_index
2021-05-08 09:34:31 +02:00
Gyuho Lee
344c9f3930
Merge pull request #12896 from wilsonwang371/profiling-txn2
server: make applier use ReadTx() in Txn() instead of ConcurrentReadTx()
2021-05-06 01:59:14 -07:00
Piotr Tabor
a78d072b9a Simplify KVstore dependency on cindex. 2021-05-04 18:21:24 +02:00
Piotr Tabor
2dbecea5b2 Simplify KVStore interaction with cindex thanks to hooks. 2021-05-04 18:21:23 +02:00
Piotr Tabor
d53d2db1e2 Tests: Backend hooks support. 2021-05-04 15:38:22 +02:00
Piotr Tabor
9f11b16b2d backend: Hooks interface & implementation. 2021-05-04 15:38:22 +02:00
Wilson Wang
56154216b7 update variable declaration location 2021-05-03 10:30:15 -07:00
wpedrak
6623c008ee server: reapply Mlock flag after defrag 2021-05-03 11:01:02 +02:00
Piotr Tabor
cedbea6c81
Merge pull request #12904 from wpedrak/limit_mlocked_memory
server: replace mlockall with `Mlock` in `--experimental-memory-mlock`
2021-04-29 18:21:24 +02:00
wpedrak
927b3a3152 server: replace mlockall with Mlock in --experimental-memory-mlock
Implementation of `--experimental-memory-mlock` backed by `mlockall` syscall is replaced by `Mlock` flag (backed by mlock syscall) of bboltDB.
2021-04-29 12:08:20 +02:00
Piotr Tabor
e90504fe62 Unify shared code (and constants) with cindex package. 2021-04-29 11:51:25 +02:00
Wilson Wang
8d8d0377a2 server: applier uses ReadTx instead of ConcurrentTx 2021-04-28 11:06:24 -07:00
Piotr Tabor
7ae3d25f91 Membership: Add additional methods to trim/manage membership data in backend. 2021-04-27 17:17:31 +02:00
Makdon
ddcb463822
etcdserver/mvcc: update trace.Step condition 2021-04-25 23:07:45 +08:00
Piotr Tabor
d7d110b5a8 mvcc/backend tests: Refactor: Do not mix testing&prod code. 2021-04-21 09:43:13 +02:00
Piotr Tabor
b1c04ce043 Applying consistency fix: ClusterVersionSet (and co) might get no applied on v2store
ClusterVersionSet, ClusterMemberAttrSet, DowngradeInfoSet functions are
writing both to V2store and backend. Prior this CL there were
in a branch not executed if shouldApplyV3 was false,
e.g. during restore when Backend is up-to-date (has high
consistency-index) while v2store requires replay from WAL log.

The most serious consequence of this bug was that v2store after restore
could have different index (revision) than the same exact store before restore,
so potentially different content between replicas.

Also this change is supressing double-applying of Membership
(ClusterConfig) changes on Backend (store v3) - that lackilly are not
part of MVCC/KeyValue store, so they didn't caused Revisions to be
bumped.

Inspired by jingyih@ comment:
https://github.com/etcd-io/etcd/pull/12820#issuecomment-815299406
2021-04-12 09:43:48 +02:00
Piotr Tabor
3bb7acc8cf Migrate dependencies pkg/foo -> client/pkg/foo 2021-04-07 00:38:47 +02:00
Piotr Tabor
03f55eeb2c Make NewTmpBackend use testing tmp location (so cleanup). 2021-03-26 13:54:55 +01:00
Gábor Lipták
49d19efd82
Correct ineffassign 2021-03-22 14:11:30 -04:00
wpedrak
dac6e37ea1 *: over 20 staticcheck fixes 2021-03-18 15:06:17 +01:00
Piotr Tabor
948e32ae15 Delete etcd_debug metrics scheduled for deletion in 3.5. 2021-03-12 16:30:47 +01:00