This is a backporting cherry-pick of the following commits:
- add flagsline
Signed-off-by: Jalin Wang <JalinWang@outlook.com>
- etcdserver: rename defaultCompactionSleepInterval var (#18495)
etcdserver: rename `minimumBatchInterval` to `defaultCompactionSleepInterval` and `defaultCompactBatchLimit` to `defaultCompactionBatchLimit`
Signed-off-by: Jalin Wang <JalinWang@outlook.com>
(cherry picked from commit 2c53be7c5d91e9b618342d63d2d06c3b265abee4)
- test: add CompactionSleepInterval in FakeStore's config
After setting the ComparionSleepInterval, we can use time.Ticker
instead of time.After to optimize the scheduleComparison(),
otherwise it will fail in the 'TestStoreCompact(t)' test.
Signed-off-by: guozhao <guozhao@360.cn>
(cherry picked from commit fab8474ef8370a089fe0174e28c197b5b93898df)
- add sleep interval
(cherry picked from commit 184b0e5d4964f1115590acec50fa5e584a2f7770)
Signed-off-by: Jalin Wang <JalinWang@outlook.com>
Before this patch, the tombstone can be deleted if its revision is equal
compacted revision. It causes that the watch subscriber won't get this
DELETE event. Based on Compact API[1], we should keep tombstone revision
if it's not less than the compaction revision.
> CompactionRequest compacts the key-value store up to a given revision.
> All superseded keys with a revision less than the compaction revision
> will be removed.
[1]: https://etcd.io/docs/latest/dev-guide/api_reference_v3/
Signed-off-by: Wei Fu <fuweid89@gmail.com>
(cherry picked from commit bbdc94181a6d67904b575ad936c20d1be10e220c)
Signed-off-by: Wei Fu <fuweid89@gmail.com>
When running the snapshot command, allow receiving an initial memory map
allocation for the database, avoiding future memory allocation issues.
Backports commit: be2883321240340479f9e334bc4a3b2959ad7639 / PR: #17277
Co-authored-by: Fatih USTA <fatihusta86@gmail.com>
Signed-off-by: Ivan Valdes <ivan@vald.es>
Accept a third argument for NewDefaultBackend for overrides to the
BackendConfig.
Add a new function, WithMmapSize, which modifies the backend config to
provide a custom InitiamMmapSize.
Backports commit: d69adf45f9e95661966204460f019b8164aaef12 / PR: #17277
Signed-off-by: Ivan Valdes <ivan@vald.es>
Signed-off-by: Wei Fu <fuweid89@gmail.com>
(cherry picked from commit 9ea234913a99670d18b66aa23915781f89713177)
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Fix#15919.
Check ScheduledCompactKeyName and FinishedCompactKeyName
before writing hash to hashstore.
If they do not match, then it means this compaction has once been interrupted and its hash value is invalid. In such cases, we won't write the hash values to the hashstore, and avoids the incorrect corruption alarm.
Signed-off-by: caojiamingalan <alan.c.19971111@gmail.com>
Progress notifications requested using ProgressRequest were sent
directly using the ctrlStream, which means that they could race
against watch responses in the watchStream.
This would especially happen when the stream was not synced - e.g. if
you requested a progress notification on a freshly created unsynced
watcher, the notification would typically arrive indicating a revision
for which not all watch responses had been sent.
This changes the behaviour so that v3rpc always goes through the watch
stream, using a new RequestProgressAll function that closely matches
the behaviour of the v3rpc code - i.e.
1. Generate a message with WatchId -1, indicating the revision for
*all* watchers in the stream
2. Guarantee that a response is (eventually) sent
The latter might require us to defer the response until all watchers
are synced, which is likely as it should be. Note that we do *not*
guarantee that the number of progress notifications matches the number
of requests, only that eventually at least one gets sent.
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Problem: during restore in watchableStore.Restore, synced watchers are moved to unsynced.
minRev will be behind since it's not updated when watcher stays synced.
Solution: update minRev
fixes: https://github.com/etcd-io/etcd/issues/15271
Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
`unsafeCommit` is called by both `(*batchTxBuffered) commit` and
`(*backend) defrag`. When users perform the defragmentation
operation, etcd doesn't update the consistent index. If etcd
crashes(e.g. panicking) in the process for whatever reason, then
etcd replays the WAL entries starting from the latest snapshot,
accordingly it may re-apply entries which might have already been
applied, eventually the revision isn't consistent with other members.
Refer to discussion in https://github.com/etcd-io/etcd/pull/14685
Signed-off-by: Benjamin Wang <wachao@vmware.com>
Problem: We pass grpc context down to applier in readonly serializable txn.
This context can be cancelled for example due to timeout.
This will trigger panic inside applyTxn
Solution: Only panic for transactions with write operations
fixes https://github.com/etcd-io/etcd/issues/14110
main PR https://github.com/etcd-io/etcd/pull/14149
Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
Previously the SetConsistentIndex() is called during the apply workflow,
but it's outside the db transaction. If a commit happens between SetConsistentIndex
and the following apply workflow, and etcd crashes for whatever reason right
after the commit, then etcd commits an incomplete transaction to db.
Eventually etcd runs into the data inconsistency issue.
In this commit, we move the SetConsistentIndex into a txPostLockHook, so
it will be executed inside the transaction lock.
