Mirroristas/etcd
2
0
Fork 0
mirror of https://github.com/etcd-io/etcd.git synced 2024-09-27 06:25:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
6,555 Commits 15 Branches 491 Tags 148 MiB
Commit Graph

158 Commits

Author SHA1 Message Date
Xiang Li
1dc0e664f0 *: fix proto and regenerate all go files 2015-12-30 20:11:19 -08:00
Xiang Li
1a0201a31a *: update api proto 2015-12-29 12:31:53 -08:00
Gyu-Ho Lee
c3655cbfd9 Merge pull request #4084 from gyuho/revisioin_document 
storage: add/fix revision/generation description
 2015-12-28 20:45:36 -07:00
Gyu-Ho Lee
5284c1c3bb storage: add/fix revision/generation description 
This adds some struct descriptions to revision/generation structs and corrects
some spelling errors.
 2015-12-28 19:32:03 -08:00
Gyu-Ho Lee
6c5dc28d0f Merge pull request #4043 from gyuho/storage_range_all_unsynced 
storage: range all unsynced at once
 2015-12-28 14:45:40 -07:00
Gyu-Ho Lee
d64eb94580 storage: clean up test variable names, minor typos in comments 
This just changes variable name to be more consistent: `N` rather than `Size`.
And fix some minor grammatical errors.
 2015-12-28 11:49:57 -08:00
Gyu-Ho Lee
ecc3e15a46 storage: delete RangeHistory 
This has been replaced by operations inside `syncWatchings`.
 2015-12-28 11:37:32 -08:00
Gyu-Ho Lee
78b0b8a4a0 storage: range all unsynced at once 
This is for https://github.com/coreos/etcd/issues/3848.
It replaces RangeHistory method for more efficient event
sending.
 2015-12-28 11:37:26 -08:00
Xiang Li
aec356e416 Merge pull request #4064 from xiang90/reduce_fysnc 
backend: do not commit unless there is a pending change
 2015-12-28 07:31:50 -08:00
Gyu-Ho Lee
df0c2e6842 storage: remove unnecessary math/rand seed 
As @jonboulle pointed out at
https://github.com/coreos/etcd/pull/4070/files#r48441847:

> math/rand is unrelated to crypto/rand; the latter reads from /dev/urandom and
> is relying on the kernel's PRNG. Just remove the seed entirely.
 2015-12-25 09:55:11 -08:00
Xiang Li
8bc59b66d1 backend: do not commit unless there is a pending change 
Reduce the nubmer of fsync etcd issues when the cluster is
idle.
 2015-12-23 18:58:37 -08:00
Gyu-Ho Lee
28d0e473a7 storage: add more tests for synced, unsynced for watchable store 
This adds more tests on functions that updates synced and unsynced in watchable
store. Preparatory change for https://github.com/coreos/etcd/issues/3848.
 2015-12-21 20:20:08 -08:00
Gyu-Ho Lee
84d777305d storage: use only one mutex for store struct 
Mutex is a variable, which means there needs to be only one mutex
value per scope. We don't need a separate mutex inside store struct,
**if we assume that `TxnPut` and `TxnRange` are called ONLY ONCE
per transaction (between `TxnBegin` and `TxnEnd`)**, as documented.
 2015-12-21 11:42:40 -08:00
Gyu-Ho Lee
2377ef870a storage: remove unnecessary test variable assignment 
There is no need to assign a separate variable since 'base' is already defined
as a local variable within the loop.
 2015-12-20 14:23:27 -08:00
Xiang Li
9b0b15c9be Merge pull request #4006 from mitake/kvtest-deadlock 
storage, test: unlock transaction in the retry loop
 2015-12-16 18:06:14 -08:00
Hitoshi Mitake
af2569c2d8 storage, test: unlock transaction in the retry loop 2015-12-17 10:35:03 +09:00
Sean Russell
928c0d3601 storage/backend: fixes Windows compile error 
The type is "Options," not "Option" -- at least, in the vendored version of boltdb in the repository.
 2015-12-16 11:10:46 -08:00
Gyu-Ho Lee
891cdd56b4 storage: newFakeStore to return only 1 
This changes newFakeStore function to return only 1 because two other arguments
are already in the first return storage struct.
 2015-12-13 17:46:47 -08:00
Gyu-Ho Lee
d55ab7798b storage: change var names from 'index' 
Having the same variable name as 'index' type
can be confusing and shadowing other variables.
This gives different variable names to 'index' variables.
 2015-12-12 16:37:24 -08:00
Gyu-Ho Lee
52d21331e2 storage: fixes shadowed variables 
Fixes for https://github.com/coreos/etcd/issues/3954.
 2015-12-12 09:20:34 -08:00
Gyu-Ho Lee
bed6b90e61 storagepb: minor updates from genproto 
I ran genproto with the most recent protocol buffer
and it adds one line of missing comment.
 2015-12-04 16:16:54 -08:00
Xiang Li
127b529582 Merge pull request #3959 from xiang90/fix_storage_test 
storage: make RestoreContinueUnfinishedCompaction more reliable
 2015-12-04 13:03:29 -08:00
Xiang Li
a0eca5fd37 storage: make RestoreContinueUnfinishedCompaction more reliable 2015-12-04 11:48:46 -08:00
Gyu-Ho Lee
ff2e8b55ae storage: add tests for unsafeAddWatching 
This adds map operation tests for unsafeAddWatching, which
could have been failed without https://github.com/coreos/etcd/pull/3939.
It tests if unsafeAddWatching is correctly updating synced map.
 2015-12-02 14:18:07 -08:00
Xiang Li
9f908ce67f storage: add missing return for unsafeAddWatching 2015-12-01 14:55:20 -08:00
Yicheng Qin
deb1da5f57 storage: add watch ID to identify watchings 
One watcher includes multiple watchings, and their events are
sent out through one channel. For the received event, user would like to
know which watching it belongs to.

Introduce a watch ID. When watching on some key, user will get a watch
ID. The watch ID is attached to all events that is observed by this
watch.
 2015-11-21 11:19:17 -08:00
Gyu-Ho Lee
48aebd9b09 storage: use map for watchableStore synced 
This is for coreos#3859 switching slice to map for synced watchings.
For a large amount of synced watchings, map implementation performs better.
When putting 1 million watchers on the same key and canceling them one by
one: original implementation takes 9m7.268221091s, while the one with map
takes only 430.531637ms.
 2015-11-21 00:42:09 -08:00
Xiang Li
cf2d20c5c9 Merge pull request #3851 from yichengq/storage-kv-data 
storage: save the KeyValue instead of Event in backend
 2015-11-19 10:24:50 -08:00
Yicheng Qin
1214f77519 storage: set revBytes capacity to avoid malloc when appending mark 
This is a performance optimization.
 2015-11-19 09:53:31 -08:00
Gyu-Ho Lee
81229dbea9 *: add missing package descriptions 
This adds and updates package descriptions in etcd projects.
And also deletes some duplicate LICENSE statements.
 2015-11-17 20:54:10 -08:00
Yicheng Qin
2f74f76025 storage: remove the event concept from key-value layer 
The point is to decouple the key-value storage layer and the
event notification layer clearly. It gives the watchableKV the
flexibility to define whatever event structure it wants without
breaking the ondisk format at key-value storage layer.

Changes:

1. change the format of key and value stored in backend

Store KeyValue struct instead of Event struct in backend value for
better abstraction as xiang suggests. And record the corresponded
action in the backend key.

2. Remove word 'event' from functions
 2015-11-17 20:35:49 -08:00
Gyu-Ho Lee
7638423f85 storage/backend: support MAP_POPULATE for unix 
This adds build constraints in order to pass memory-map flags to bolt.Option.
If backend package passes syscall.MAP_POPULATE flag, the boltdb does read-ahead
which speeds up entire-database read, which then leads to faster storage
Restore. Benchmark result shows for 4GB, it opens and loads 6x faster in SSD,
and 12x faster in HDD. For 2GB, 1.6x faster with MAP_POPULATE in SSD.
 2015-11-15 16:47:38 -08:00
Yicheng Qin
dadfdf6af8 Merge pull request #3802 from yichengq/fix-storage-watch 
storage: delete key instead of setting it to false
 2015-11-05 11:40:46 -08:00
Gyu-Ho Lee
6e5eb03544 storage: move watcherGauge to watchable_store 
watcherGauge should be increased everytime we creates Watcher, not per watch
method call.
 2015-11-04 13:17:47 -08:00
Gyu-Ho Lee
4ebf28aa2e storage: add metrics to watcher 
This adds metrics to watcher, and changes some order in MustRegister function
calls in init (same order that we define the gauges).
 2015-11-04 13:01:52 -08:00
Yicheng Qin
c8e622f517 storage: make putm/delm a set with empty value 
This cleans the code, and reduces the allocation space.
 2015-11-03 19:10:45 -08:00
Yicheng Qin
6dbfc21846 storage: delete key instead of setting it to false 
When getting the watched events, it iterate all keys in putm and delm
to generate the events. If we don't delete the key from putm/delm,
it would range on the key that is not actually put or deleted. This is
incorrect.

Fix the panic that happens when single put/delete is watched.
 2015-11-03 19:00:39 -08:00
Xiang Li
1f1d8e9282 Merge pull request #3800 from xiang90/watch_server 
*: serve watch service
 2015-11-03 16:32:29 -08:00
Xiang Li
10de2e6dbe *: serve watch service 
Implement watch service and hook it up
with grpc server in etcdmain.
 2015-11-03 15:58:34 -08:00
Gyu-Ho Lee
bdc280c4a7 storage: apply same naming in metrics.go 
This is PR following up with Xiang's https://github.com/coreos/etcd/pull/3795,
and to make the naming consistent with its interface change.
 2015-11-03 15:19:18 -08:00
Xiang Li
a1129dd5a5 storage: support multiple watching per watcher 
We want to support multiple watchings per one watcher chan. Then
we can have one single go routine to watch multiple keys/prefixs.
 2015-11-03 12:36:11 -08:00
Yicheng Qin
59b5dabc66 storage: extend wait timeout for execution 
Extend timeout to pass always in traivs.
 2015-10-30 17:44:31 -07:00
Xiang Li
f71bcfa8ce storage: move watcher interface into watcher.go 2015-10-28 21:10:58 -07:00
Yicheng Qin
4b8ee2d66e storage: skip old entry in ConsistentWatchableStore 
This avoids to apply the same entry twice when restoring from disk.
 2015-10-26 23:26:06 -07:00
Yicheng Qin
263b270708 etcdserver: commit v3 storage before releasing WAL 
This ensures that v3 storage could always find the following log entries
when restart.
 2015-10-26 21:06:08 -07:00
Yicheng Qin
15ed6d8268 etcdserver: save consistent index into v3 storage 
This helps to recover consistent index when restart in the future.
 2015-10-24 09:27:24 -07:00
Yicheng Qin
41cb39b68a storage: Get -> ConsistentIndex in ConsistentIndexGetter 
To make the method name more specific in the context.
 2015-10-23 16:40:55 -07:00
Yicheng Qin
4fb4bc3ca8 storage: add consistentWatchableStore 
consistentWatchableStore maintains an index that is always consistent
with the latest txn. The index could be used to indicate the progress
of the store so far when recovery.
 2015-10-22 22:54:51 -07:00
Yicheng Qin
8b08fff1e9 Merge pull request #3731 from yichengq/storage-kv 
storage: fix WatchableKV interface and refine comment
 2015-10-21 17:27:24 -07:00
Gyu-Ho Lee
f73d0ed1d9 storage: use map for watchable store unsynced 
This is for `TODO: use map to reduce cancel cost`.
I switched slice to map, and benchmark results show
that map implementation performs better, as follows:

```
[1]:
benchmark                                   old ns/op     new ns/op     delta
BenchmarkWatchableStoreUnsyncedCancel       215212        1307          -99.39%
BenchmarkWatchableStoreUnsyncedCancel-2     120453        710           -99.41%
BenchmarkWatchableStoreUnsyncedCancel-4     120765        748           -99.38%
BenchmarkWatchableStoreUnsyncedCancel-8     121391        719           -99.41%

benchmark                                   old allocs     new allocs     delta
BenchmarkWatchableStoreUnsyncedCancel       0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-2     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-4     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-8     0              0              +0.00%

benchmark                                   old bytes     new bytes     delta
BenchmarkWatchableStoreUnsyncedCancel       200           1             -99.50%
BenchmarkWatchableStoreUnsyncedCancel-2     138           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-4     138           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-8     139           0             -100.00%

[2]:
benchmark                                   old ns/op     new ns/op     delta
BenchmarkWatchableStoreUnsyncedCancel       212550        1117          -99.47%
BenchmarkWatchableStoreUnsyncedCancel-2     120927        691           -99.43%
BenchmarkWatchableStoreUnsyncedCancel-4     120752        699           -99.42%
BenchmarkWatchableStoreUnsyncedCancel-8     121012        688           -99.43%

benchmark                                   old allocs     new allocs     delta
BenchmarkWatchableStoreUnsyncedCancel       0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-2     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-4     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-8     0              0              +0.00%

benchmark                                   old bytes     new bytes     delta
BenchmarkWatchableStoreUnsyncedCancel       197           1             -99.49%
BenchmarkWatchableStoreUnsyncedCancel-2     138           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-4     138           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-8     139           0             -100.00%

[3]:
benchmark                                   old ns/op     new ns/op     delta
BenchmarkWatchableStoreUnsyncedCancel       214268        1183          -99.45%
BenchmarkWatchableStoreUnsyncedCancel-2     120763        759           -99.37%
BenchmarkWatchableStoreUnsyncedCancel-4     120321        708           -99.41%
BenchmarkWatchableStoreUnsyncedCancel-8     121628        680           -99.44%

benchmark                                   old allocs     new allocs     delta
BenchmarkWatchableStoreUnsyncedCancel       0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-2     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-4     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-8     0              0              +0.00%

benchmark                                   old bytes     new bytes     delta
BenchmarkWatchableStoreUnsyncedCancel       200           1             -99.50%
BenchmarkWatchableStoreUnsyncedCancel-2     139           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-4     138           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-8     139           0             -100.00%

[4]:
benchmark                                   old ns/op     new ns/op     delta
BenchmarkWatchableStoreUnsyncedCancel       208332        1089          -99.48%
BenchmarkWatchableStoreUnsyncedCancel-2     121011        691           -99.43%
BenchmarkWatchableStoreUnsyncedCancel-4     120678        681           -99.44%
BenchmarkWatchableStoreUnsyncedCancel-8     121303        721           -99.41%

benchmark                                   old allocs     new allocs     delta
BenchmarkWatchableStoreUnsyncedCancel       0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-2     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-4     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-8     0              0              +0.00%

benchmark                                   old bytes     new bytes     delta
BenchmarkWatchableStoreUnsyncedCancel       194           1             -99.48%
BenchmarkWatchableStoreUnsyncedCancel-2     139           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-4     139           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-8     139           0             -100.00%

[5]:
benchmark                                   old ns/op     new ns/op     delta
BenchmarkWatchableStoreUnsyncedCancel       211900        1097          -99.48%
BenchmarkWatchableStoreUnsyncedCancel-2     121795        753           -99.38%
BenchmarkWatchableStoreUnsyncedCancel-4     123182        700           -99.43%
BenchmarkWatchableStoreUnsyncedCancel-8     122820        688           -99.44%

benchmark                                   old allocs     new allocs     delta
BenchmarkWatchableStoreUnsyncedCancel       0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-2     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-4     0              0              +0.00%
BenchmarkWatchableStoreUnsyncedCancel-8     0              0              +0.00%

benchmark                                   old bytes     new bytes     delta
BenchmarkWatchableStoreUnsyncedCancel       198           1             -99.49%
BenchmarkWatchableStoreUnsyncedCancel-2     140           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-4     141           0             -100.00%
BenchmarkWatchableStoreUnsyncedCancel-8     141           0             -100.00%
```
 2015-10-21 15:30:15 -07:00