213 Commits

Author SHA1 Message Date
Chao Chen
405862e807 Fix event loss after compaction
Signed-off-by: Chao Chen <chaochn@amazon.com>
2024-03-15 14:22:37 -07:00
Abhishek Kr Srivastav
fe113026fa Increased timeout to fix HashKV test
Signed-off-by: Abhishek Kr Srivastav <Abhishek.kr.srivastav@ibm.com>
2024-02-23 15:58:57 +05:30
Siyuan Zhang
3565a822de Add VerifyTxConsistency to backend.
Signed-off-by: Siyuan Zhang <sizhang@google.com>

Update server/storage/backend/verify.go

Co-authored-by: Benjamin Wang <benjamin.wang@broadcom.com>

Update server/storage/backend/verify.go

Co-authored-by: Benjamin Wang <benjamin.wang@broadcom.com>
2024-02-22 11:31:16 -08:00
Ivan Valdes
be28833212
etcdutl: Fix snapshot restore memory alloc issue
When running the snapshot command, allow receiving an initial memory map
allocation for the database, avoiding future memory allocation issues.

Co-authored-by: Benjamin Wang <benjamin.wang@broadcom.com>
Co-authored-by: Fatih USTA <fatihusta86@gmail.com>
Signed-off-by: Ivan Valdes <ivan@vald.es>
2024-02-13 13:36:13 -08:00
Ivan Valdes
d69adf45f9
server: Implement WithMmapSize option for backend config
Accept a third argument for NewDefaultBackend for overrides to the
BackendConfig.
Add a new function, WithMmapSize, which modifies the backend config to
provide a custom InitiamMmapSize.

Signed-off-by: Ivan Valdes <ivan@vald.es>
2024-02-13 13:24:44 -08:00
Benjamin Wang
c12e15d91a Add verification on keys: should be always mononically increasing
Signed-off-by: Benjamin Wang <benjamin.ahrtr@gmail.com>
2024-01-25 18:00:26 +00:00
Benjamin Wang
c647614b6a test: fix TestHashKVWhenCompacting: ensure all goroutine finished
Signed-off-by: Benjamin Wang <benjamin.ahrtr@gmail.com>
2024-01-25 13:30:27 +00:00
Benjamin Wang
dd0b0526e1
Merge pull request #17284 from ahrtr/test_TestScheduleCompaction_20240119
Fix test issue in TestScheduleCompaction
2024-01-20 13:29:04 +00:00
Siyuan Zhang
30b242f800 Revert "add test cases of duplicate keys in one tx."
This reverts commit 8d29b245a39172bd72643106ea7d9e4df727bd7e.

Signed-off-by: Siyuan Zhang <sizhang@google.com>
2024-01-19 08:58:48 -08:00
Siyuan Zhang
93afd8e430 Revert "add key dedupe when a write buffer writeback to an empty read buffer bucket."
This reverts commit 971a4f57ccff71dfc10c1f9eb7a6a4d64bec1d2e.

Signed-off-by: Siyuan Zhang <sizhang@google.com>
2024-01-19 08:58:27 -08:00
Benjamin Wang
82562abc3c Fix test issue in TestScheduleCompaction
It always reuse the same memory space. Each next iteration
will always overwrite previous data

Signed-off-by: Benjamin Wang <benjamin.ahrtr@gmail.com>
2024-01-19 15:29:51 +00:00
Benjamin Wang
690916f018 replace CreateBucket with CreateBucketIfNotExists
Signed-off-by: Benjamin Wang <benjamin.ahrtr@gmail.com>
2024-01-16 09:40:20 +00:00
Benjamin Wang
4bc55e5208
Merge pull request #17204 from rahulbapumore/branch1
mvcc: Printing etcd backend database related metrics inside scheduleCompaction function
2024-01-15 15:37:08 +00:00
Rahul More
21bbc82710 mvcc: Printing etcd backend database related metrics inside
scheduleCompaction function

To improve traceability of backend database usage, Added below parameter
related to backend database usage metrics inside scheduledCompaction
function.
current-db-size-bytes
current-db-size
current-db-size-in-use-bytes
current-db-size-in-use

Signed-off-by: Rahul More <rahulbapumore@gmail.com>
2024-01-15 17:36:38 +05:30
Benjamin Wang
9ded10ec76
Merge pull request #17237 from ahrtr/bbolt_logger_20240112
Specify logger for bbolt
2024-01-13 11:03:14 +00:00
Benjamin Wang
cfbf672b3c specify logger for bbolt
Signed-off-by: Benjamin Wang <benjamin.ahrtr@gmail.com>
2024-01-12 16:38:18 +00:00
Siyuan Zhang
971a4f57cc add key dedupe when a write buffer writeback to an empty read buffer bucket.
Signed-off-by: Siyuan Zhang <sizhang@google.com>
2024-01-11 11:31:09 -08:00
Siyuan Zhang
8d29b245a3 add test cases of duplicate keys in one tx.
Signed-off-by: Siyuan Zhang <sizhang@google.com>
2024-01-11 11:31:03 -08:00
Benjamin Wang
b3bf59a355
Merge pull request #17195 from siyuanfoundation/txBuf1
Fix delete inconsistencies in read buffer
2024-01-11 08:21:47 +00:00
Siyuan Zhang
db61c9695d commit bbolt transaction if there is any pending deleting operations
Signed-off-by: Siyuan Zhang <sizhang@google.com>
2024-01-10 09:33:21 -08:00
Siyuan Zhang
b3a5307640 add tests to test tx delete consistency.
Signed-off-by: Siyuan Zhang <sizhang@google.com>
2024-01-10 09:33:13 -08:00
Marek Siarkowicz
d39d86a214 Improve logs around recovering from snapshot backend
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-12-20 16:26:27 +01:00
Allen Ray
395376d3ab Refactor common revision code to pkg
Signed-off-by: Allen Ray <alray@redhat.com>
2023-10-09 14:53:16 -04:00
Wei Fu
4704a5af3a *: fix unused issue
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-09-25 19:37:18 +08:00
Wei Fu
aa97484166 *: enable goimports in verify-lint
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-09-21 21:14:09 +08:00
Wei Fu
5e3910d96c *: fix govet-shadow lint
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-09-19 20:24:01 +08:00
Benjamin Wang
4ec581a6a3 re-generate all the *.pb.gw.go files
Run ./scripts/genproto.sh

Signed-off-by: Benjamin Wang <wachao@vmware.com>
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-09-18 11:22:15 +01:00
Wei Fu
522b698928 upgrade grpc-gateway from v1 to v2
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2023-09-18 11:22:15 +01:00
Benjamin Wang
700411d838
Merge pull request #16601 from fuweid/fix-nakedret-lint
*: fix nakedret lint
2023-09-18 10:00:25 +01:00
Wei Fu
e72c2c40d4 *: fix nakedret lint
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-09-17 23:52:41 +08:00
chenyahui
c0aa3b613b Use any instead of interface{}
Signed-off-by: chenyahui <cyhone@qq.com>
2023-09-17 17:41:58 +08:00
redwrasse
b79cbcc3db mvcc: cleanup comment in HashStorage.Store
Signed-off-by: redwrasse <mail@redwrasse.io>
2023-09-11 21:54:42 -07:00
Benjamin Wang
a13fc7e51c
Merge pull request #16508 from new-dream/main-concurrentreadtxn
server: optimizing memory overhead of copy operation in ConcurrentReadTxn
2023-09-09 09:35:00 +01:00
Wei Fu
a6f7bbe560 server/storage/mvcc: deflake TestHashKVWhenCompacting
The HashByRev-goroutines exit since receive `donec` notification. The
Check-computed-hashes goroutine could not have chance to get the hash
result and be stuck forever. We should add case for donec when we wait
for hash result.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-09-06 18:12:42 +08:00
new-dream
b05d75ab0b server: optimizing memory overhead of copy operation in ConcurrentReadTxn
Signed-off-by: new-dream <111836360+new-dream@users.noreply.github.com>
2023-08-30 23:03:15 +08:00
James Blair
cb0df72b70
Use crypto/rand.Read instead of deprecated math/rand.Read.
Signed-off-by: James Blair <mail@jamesblair.net>
2023-08-22 21:48:27 +12:00
Marek Siarkowicz
53cbd81009 Separate Writer interface from BatchTx interfaces
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-07-31 10:18:01 +02:00
Marek Siarkowicz
29769984e6 Remove RLock/RUnlock from BatchTx
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-07-28 11:39:50 +02:00
Marek Siarkowicz
b4f8a7be51 server: Remove Lock/Unlock from ReadTx
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-07-27 13:34:43 +02:00
caojiamingalan
bc97a94564 Follow up https://github.com/etcd-io/etcd/pull/16068#discussion_r1263664700
Replace unnecessary Lock()/Unlock()s with RLock()/RUnlock()s

Signed-off-by: caojiamingalan <alan.c.19971111@gmail.com>
2023-07-14 20:08:25 -05:00
caojiamingalan
b9e30bf878 etcdserver: add e2e test to reproduce the incorrect hash issue when resuming scheduled compaction.
check ScheduledCompactKeyName and FinishedCompactKeyName
before writing hash to hashstore. If they do not match, then it means this compaction has once been interrupted and its hash value is invalid. In such cases, we won't write the hash values to the hashstore, and avoids the incorrect corruption alarm.

Signed-off-by: caojiamingalan <alan.c.19971111@gmail.com>
2023-06-07 19:54:09 -05:00
Chao Chen
9e1e378e9e update code comments
Signed-off-by: Chao Chen <chaochn@amazon.com>
2023-05-24 12:54:27 -07:00
Benjamin Wang
19ec574f45
Merge pull request #15915 from hexfusion/bytes
server/storage/schema: prefer equal to compare for equality comparisons
2023-05-18 10:53:42 +08:00
Sam Batschelet
a708e94749 server/storage/schema: prefer equal to compare for equality comparisons
Signed-off-by: Sam Batschelet <sbatschelet@gmail.com>
2023-05-16 21:25:34 -04:00
qsyqian
0cd1b7beaa log time when open backend db took too long
Signed-off-by: qsyqian <qsyqian@gmail.com>
2023-05-08 15:23:29 +08:00
Peter Wortmann
74feb229c7 etcdserver: Guarantee order of requested progress notifications
Progress notifications requested using ProgressRequest were sent
directly using the ctrlStream, which means that they could race
against watch responses in the watchStream.

This would especially happen when the stream was not synced - e.g. if
you requested a progress notification on a freshly created unsynced
watcher, the notification would typically arrive indicating a revision
for which not all watch responses had been sent.

This changes the behaviour so that v3rpc always goes through the watch
stream, using a new RequestProgressAll function that closely matches
the behaviour of the v3rpc code - i.e.

1. Generate a message with WatchId -1, indicating the revision for
   *all* watchers in the stream

2. Guarantee that a response is (eventually) sent

The latter might require us to defer the response until all watchers
are synced, which is likely as it should be. Note that we do *not*
guarantee that the number of progress notifications matches the number
of requests, only that eventually at least one gets sent.

Signed-off-by: Peter Wortmann <peter.wortmann@skao.int>
2023-04-05 11:54:10 +01:00
Marek Siarkowicz
7052d8998a server: Test watch restore
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-03-20 11:15:41 +01:00
Bogdan Kanivets
a690707c57 mvcc: update minRev when watcher stays synced
Problem: during restore in watchableStore.Restore, synced watchers are moved to unsynced.
minRev will be behind since it's not updated when watcher stays synced.

Solution: update minRev

fixes: https://github.com/etcd-io/etcd/issues/15271
Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-03-20 11:11:48 +01:00
Wei Fu
eb09e00541 chore: refactor cleanup fn in mvcc test
The tmp path is cleanup by go testing so that the `cleanup` doesn't need
to call `os.Remove`.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-03-19 21:46:03 +08:00
Wei Fu
830d9e9eaa test: fix TestRestoreContinueUnfinishedCompaction
The original testcase uses `return` statement which skips `restore`
case. It's aimed to enable `restore` testcase.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-03-18 10:58:05 +08:00