Mirroristas/etcd
2
0
Fork 0
mirror of https://github.com/etcd-io/etcd.git synced 2024-09-27 06:25:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
7,815 Commits 15 Branches 491 Tags 148 MiB
Commit Graph

268 Commits

Author SHA1 Message Date
Xiang Li
a1129dd5a5 storage: support multiple watching per watcher 
We want to support multiple watchings per one watcher chan. Then
we can have one single go routine to watch multiple keys/prefixs.
 2015-11-03 12:36:11 -08:00
Yicheng Qin
59b5dabc66 storage: extend wait timeout for execution 
Extend timeout to pass always in traivs.
 2015-10-30 17:44:31 -07:00
Xiang Li
f71bcfa8ce storage: move watcher interface into watcher.go 2015-10-28 21:10:58 -07:00
Yicheng Qin
4b8ee2d66e storage: skip old entry in ConsistentWatchableStore 
This avoids to apply the same entry twice when restoring from disk.
 2015-10-26 23:26:06 -07:00
Yicheng Qin
263b270708 etcdserver: commit v3 storage before releasing WAL 
This ensures that v3 storage could always find the following log entries
when restart.
 2015-10-26 21:06:08 -07:00
Yicheng Qin
15ed6d8268 etcdserver: save consistent index into v3 storage 
This helps to recover consistent index when restart in the future.
 2015-10-24 09:27:24 -07:00
Yicheng Qin
41cb39b68a storage: Get -> ConsistentIndex in ConsistentIndexGetter 
To make the method name more specific in the context.
 2015-10-23 16:40:55 -07:00
Yicheng Qin
4fb4bc3ca8 storage: add consistentWatchableStore 
consistentWatchableStore maintains an index that is always consistent
with the latest txn. The index could be used to indicate the progress
of the store so far when recovery.
 2015-10-22 22:54:51 -07:00
Yicheng Qin
8b08fff1e9 Merge pull request #3731 from yichengq/storage-kv 
storage: fix WatchableKV interface and refine comment
 2015-10-21 17:27:24 -07:00
Gyu-Ho Lee
f73d0ed1d9 storage: use map for watchable store unsynced 
This is for `TODO: use map to reduce cancel cost`.
I switched slice to map, and benchmark results show
that map implementation performs better, as follows:

```
[1]:
benchmark                                   old ns/op     new ns/op     delta
BenchmarkWatchableStoreUnsyncedCancel       215212        1307          -99.39%
BenchmarkWatchableStoreUnsyncedCancel-2     120453        710           -99.41%
BenchmarkWatchableStoreUnsyncedCancel-4     120765        748           -99.38%
BenchmarkWatchableStoreUnsyncedCancel-8     121391        719           -99.41%

benchmark                                   old allocs     new allocs     delta
BenchmarkWatchableStoreUnsyncedCancel       0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-2     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-4     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-8     0              0              +0.00%

benchmark                                   old bytes     new bytes     delta
BenchmarkWatchableStoreUnsyncedCancel       200           1             -99.50%
BenchmarkWatchableStoreUnsyncedCancel-2     138           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-4     138           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-8     139           0             -100.00%

[2]:
benchmark                                   old ns/op     new ns/op     delta
BenchmarkWatchableStoreUnsyncedCancel       212550        1117          -99.47%
BenchmarkWatchableStoreUnsyncedCancel-2     120927        691           -99.43%
BenchmarkWatchableStoreUnsyncedCancel-4     120752        699           -99.42%
BenchmarkWatchableStoreUnsyncedCancel-8     121012        688           -99.43%

benchmark                                   old allocs     new allocs     delta
BenchmarkWatchableStoreUnsyncedCancel       0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-2     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-4     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-8     0              0              +0.00%

benchmark                                   old bytes     new bytes     delta
BenchmarkWatchableStoreUnsyncedCancel       197           1             -99.49%
BenchmarkWatchableStoreUnsyncedCancel-2     138           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-4     138           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-8     139           0             -100.00%

[3]:
benchmark                                   old ns/op     new ns/op     delta
BenchmarkWatchableStoreUnsyncedCancel       214268        1183          -99.45%
BenchmarkWatchableStoreUnsyncedCancel-2     120763        759           -99.37%
BenchmarkWatchableStoreUnsyncedCancel-4     120321        708           -99.41%
BenchmarkWatchableStoreUnsyncedCancel-8     121628        680           -99.44%

benchmark                                   old allocs     new allocs     delta
BenchmarkWatchableStoreUnsyncedCancel       0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-2     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-4     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-8     0              0              +0.00%

benchmark                                   old bytes     new bytes     delta
BenchmarkWatchableStoreUnsyncedCancel       200           1             -99.50%
BenchmarkWatchableStoreUnsyncedCancel-2     139           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-4     138           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-8     139           0             -100.00%

[4]:
benchmark                                   old ns/op     new ns/op     delta
BenchmarkWatchableStoreUnsyncedCancel       208332        1089          -99.48%
BenchmarkWatchableStoreUnsyncedCancel-2     121011        691           -99.43%
BenchmarkWatchableStoreUnsyncedCancel-4     120678        681           -99.44%
BenchmarkWatchableStoreUnsyncedCancel-8     121303        721           -99.41%

benchmark                                   old allocs     new allocs     delta
BenchmarkWatchableStoreUnsyncedCancel       0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-2     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-4     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-8     0              0              +0.00%

benchmark                                   old bytes     new bytes     delta
BenchmarkWatchableStoreUnsyncedCancel       194           1             -99.48%
BenchmarkWatchableStoreUnsyncedCancel-2     139           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-4     139           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-8     139           0             -100.00%

[5]:
benchmark                                   old ns/op     new ns/op     delta
BenchmarkWatchableStoreUnsyncedCancel       211900        1097          -99.48%
BenchmarkWatchableStoreUnsyncedCancel-2     121795        753           -99.38%
BenchmarkWatchableStoreUnsyncedCancel-4     123182        700           -99.43%
BenchmarkWatchableStoreUnsyncedCancel-8     122820        688           -99.44%

benchmark                                   old allocs     new allocs     delta
BenchmarkWatchableStoreUnsyncedCancel       0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-2     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-4     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-8     0              0              +0.00%

benchmark                                   old bytes     new bytes     delta
BenchmarkWatchableStoreUnsyncedCancel       198           1             -99.49%
BenchmarkWatchableStoreUnsyncedCancel-2     140           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-4     141           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-8     141           0             -100.00%
```
 2015-10-21 15:30:15 -07:00
Yicheng Qin
027c073d55 storage: refine Watch comment in WatchableKV 
Explain explicitly how these arguments are used.
 2015-10-21 14:47:52 -07:00
Yicheng Qin
2673e657e6 storage: fix WatchableKV interface 
We delete endRev from the watch functionality, so the interface needs
to be fixed.
 2015-10-21 11:50:25 -07:00
Yicheng Qin
0f7374ce89 storage: KV field -> store field in watchableStore 
We need to access the underlying store to use its RangeEvents function.
It is not good to use unnecessary type conversion.

The underlying store is also needed for further store upon
watchableStore.
 2015-10-20 19:23:20 -07:00
Xiang Li
5efdd7bc6d storage/backend: avoid creating new bolt.tx during a batchTx 2015-10-17 21:31:34 -07:00
Xiang Li
6556bf1643 storage: remove the endRev of watcher 2015-10-17 15:59:49 -07:00
Xiang Li
21179d929f Merge pull request #3616 from yichengq/storage-txn 
storage: hold batchTx lock during KV txn
 2015-10-05 17:12:52 -07:00
Xiang Li
0aa2f1192a storage: add metrics for db total size 2015-10-05 16:56:30 -07:00
Xiang Li
ba949ae2be Merge pull request #3640 from xiang90/watch_metrics 
storage: add metrics for watchers
 2015-10-05 11:46:50 -07:00
Xiang Li
09157d4f1a storage: add metrics for watchers 2015-10-05 11:32:56 -07:00
Yicheng Qin
c97dda766e storage: hold batchTx lock during KV txn 
One txn is treated as atomic, and might contain multiple Put/Delete/Range
operations. For now, between these operations, we might call forecCommit
to sync the change to disk, or backend may commit it in background.
Thus the snapshot state might contains an unfinished multiple objects
transaction, which is dangerous if database is restored from the snapshot.

This PR makes KV txn hold batchTx lock during the process and avoids
commit to happen.
 2015-10-03 16:01:05 -07:00
Yicheng Qin
36f4303fc3 storage/etcdserver: update KV.Snapshot function 
When using Snapshot function, it is expected:
1. know the size of snapshot before writing data
2. split snapshot-ready phase and write-data phase. so we could cut
snapshot first and write data later.

Update its interface to fit the requirement of etcdserver.
 2015-10-03 10:15:23 -07:00
Xiang Li
385e17583f storage: fix hash by iterating kv 2015-09-23 11:28:33 -07:00
Gyu-Ho Lee
4113509828 storage/watchable_store: defer to Unlock s.mu 
New PR from https://github.com/coreos/etcd/pull/3575.
This add `defer` to `s.mu`. Current code does not `Unlock`
in the correct scope, I think.

(Sorry, I accidentally deleted my fork so the changes
might not sound continuous from my previous pull requests.)
 2015-09-22 23:25:07 -07:00
Yicheng Qin
d72914c36f storage: clarify comment for store.RangeEvents and fix related bugs 
Change to the function:
1. specify the meaning of startRev and endRev parameters
2. specify the meaning of returned nextRev

Moreover, it adds unit tests for the function.
 2015-09-19 23:17:03 -07:00
Yicheng Qin
5709b66dfb storage: add unit test for index.RangeEvents 2015-09-19 23:08:24 -07:00
Yicheng Qin
87b5143b15 storage: fix missing continue in keyIndex.since 
It should continue to skip following operations.

The test from rev14 to rev0 fails if it doesn't call continue and append
all revisions of the same main rev to the list.
 2015-09-19 23:01:18 -07:00
Yicheng Qin
158d6e0e03 storage: fix calculating generation in keyIndex.since 
It should skip last empty generation when the key is just tombstoned.

The rev15 and rev16 in the test fails if it doesn't skip last empty generation
and find previous generations.
 2015-09-19 22:58:45 -07:00
Yicheng Qin
be80d11948 storage: enhance test for keyIndex.Get and keyIndex.Compact 
It covers the case that one key is set multiple times in one main
revision now.
 2015-09-17 18:26:17 -07:00
Yicheng Qin
f7efbe8b14 storage/backend: extend wait timeout for commit to finish 
It needs to take more time on travis. Fix:

```
--- FAIL: TestBackendBatchIntervalCommit (0.01s)
		backend_test.go:113: bucket test does not exit
```
 2015-09-16 14:14:51 -07:00
Jonathan Boulle
7848ac3979 *: add missing license headers 2015-09-15 14:09:01 -07:00
Xiang Li
6d1f0ce89f storage: add rev into kv interface 2015-09-15 12:11:00 -07:00
Yicheng Qin
c082488e23 Merge pull request #3507 from yichengq/watch 
storage: support basic watch
 2015-09-15 00:04:36 -07:00
Yicheng Qin
ec43e0a4c3 storage: introduce WatchableKV and watch feature 
WatchableKV is an interface upon KV, and supports watch feature.
 2015-09-14 23:53:03 -07:00
Xiang Li
451cce4a90 Merge pull request #3516 from xiang90/hash_improved 
storage: support hash state
 2015-09-13 21:46:12 -07:00
Xiang Li
714b5e0b08 storage: support hash state 2015-09-13 21:34:58 -07:00
Dmitry Smirnov
b2f4a5f587 *: fix spelling issues (codespell). 
Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>
 2015-09-11 10:22:29 +10:00
Xiang Li
3f18ded10a *: v3api index->revision 2015-09-04 10:41:20 -07:00
Yicheng Qin
215f27c2f5 storage: add mock tests for store struct 2015-09-04 08:53:49 -07:00
Yicheng Qin
5ae2eb4731 storage: avoid one extra round of wait 
It could exit early if it knows that there is no more keys.
 2015-09-03 19:12:27 -07:00
Yicheng Qin
9175df7c71 storage: correct revision for range when deleteRange 
to make it logically reasonable.
 2015-09-03 19:12:27 -07:00
Yicheng Qin
797a4796d9 storage: remove check for DELETE type KeyValue 
kvindex always returns kvs that exist at given revision, so there is no
need to check for whether the KeyValue range from backend is DELETE type.
 2015-09-03 19:12:27 -07:00
Yicheng Qin
00e31f13a6 storage: remove unnecessary rev parameter 2015-09-03 19:12:27 -07:00
Xiang Li
ef7cf058a2 *: update gogoproto 2015-09-03 15:32:25 -07:00
Tamir Duberstein
45390b9fb8 *: regenerate proto to use local import path 
Using Go-style import paths in protos is not idiomatic. Normally, this
detail would be internal to etcd, but the path from which gogoproto
is imported affects downstream consumers (e.g. cockroachdb).

In cockroach, we want to avoid including `$GOPATH/src` in our protoc
include path for various reasons. This patch puts etcd on the same
convention, which allows this for cockroach.

More information: https://github.com/cockroachdb/cockroach/pull/2339#discussion_r38663417

This commit also regenerates all the protos, which seem to have
drifted a tiny bit.
 2015-09-03 13:38:28 -04:00
Yicheng Qin
1eaf169057 Merge pull request #3395 from yichengq/backend-test 
storage/backend: add unit tests for backend and batchTx
 2015-09-03 07:23:38 -07:00
Yicheng Qin
44fd734038 storage/backend: add unit tests for backend and batchTx 2015-09-02 16:57:13 -07:00
Yicheng Qin
a21166c3aa storage: extend timeout to wait for put complete 
travis is sometimes slow, and it could fail to complete the put in 10ms.
 2015-09-01 09:03:03 -07:00
Yicheng Qin
4b9b0cbcc1 storage: add newBackend and newBatchTx 
This is for ease of testing.
 2015-08-31 13:25:10 -07:00
Xiang Li
b9632e0f8d storage: register txnCounter 2015-08-28 15:17:16 -07:00
Xiang Li
dd443be41b storage: report total number of keys 2015-08-28 15:16:53 -07:00