204 Commits

Author SHA1 Message Date
Xiang Li
6556bf1643 storage: remove the endRev of watcher 2015-10-17 15:59:49 -07:00
Xiang Li
21179d929f Merge pull request #3616 from yichengq/storage-txn
storage: hold batchTx lock during KV txn
2015-10-05 17:12:52 -07:00
Xiang Li
0aa2f1192a storage: add metrics for db total size 2015-10-05 16:56:30 -07:00
Xiang Li
ba949ae2be Merge pull request #3640 from xiang90/watch_metrics
storage: add metrics for watchers
2015-10-05 11:46:50 -07:00
Xiang Li
09157d4f1a storage: add metrics for watchers 2015-10-05 11:32:56 -07:00
Yicheng Qin
c97dda766e storage: hold batchTx lock during KV txn
One txn is treated as atomic, and might contain multiple Put/Delete/Range
operations. For now, between these operations, we might call forecCommit
to sync the change to disk, or backend may commit it in background.
Thus the snapshot state might contains an unfinished multiple objects
transaction, which is dangerous if database is restored from the snapshot.

This PR makes KV txn hold batchTx lock during the process and avoids
commit to happen.
2015-10-03 16:01:05 -07:00
Yicheng Qin
36f4303fc3 storage/etcdserver: update KV.Snapshot function
When using Snapshot function, it is expected:
1. know the size of snapshot before writing data
2. split snapshot-ready phase and write-data phase. so we could cut
snapshot first and write data later.

Update its interface to fit the requirement of etcdserver.
2015-10-03 10:15:23 -07:00
Xiang Li
385e17583f storage: fix hash by iterating kv 2015-09-23 11:28:33 -07:00
Gyu-Ho Lee
4113509828 storage/watchable_store: defer to Unlock s.mu
New PR from https://github.com/coreos/etcd/pull/3575.
This add `defer` to `s.mu`. Current code does not `Unlock`
in the correct scope, I think.

(Sorry, I accidentally deleted my fork so the changes
might not sound continuous from my previous pull requests.)
2015-09-22 23:25:07 -07:00
Yicheng Qin
d72914c36f storage: clarify comment for store.RangeEvents and fix related bugs
Change to the function:
1. specify the meaning of startRev and endRev parameters
2. specify the meaning of returned nextRev

Moreover, it adds unit tests for the function.
2015-09-19 23:17:03 -07:00
Yicheng Qin
5709b66dfb storage: add unit test for index.RangeEvents 2015-09-19 23:08:24 -07:00
Yicheng Qin
87b5143b15 storage: fix missing continue in keyIndex.since
It should continue to skip following operations.

The test from rev14 to rev0 fails if it doesn't call continue and append
all revisions of the same main rev to the list.
2015-09-19 23:01:18 -07:00
Yicheng Qin
158d6e0e03 storage: fix calculating generation in keyIndex.since
It should skip last empty generation when the key is just tombstoned.

The rev15 and rev16 in the test fails if it doesn't skip last empty generation
and find previous generations.
2015-09-19 22:58:45 -07:00
Yicheng Qin
be80d11948 storage: enhance test for keyIndex.Get and keyIndex.Compact
It covers the case that one key is set multiple times in one main
revision now.
2015-09-17 18:26:17 -07:00
Yicheng Qin
f7efbe8b14 storage/backend: extend wait timeout for commit to finish
It needs to take more time on travis. Fix:

```
--- FAIL: TestBackendBatchIntervalCommit (0.01s)
		backend_test.go:113: bucket test does not exit
```
2015-09-16 14:14:51 -07:00
Jonathan Boulle
7848ac3979 *: add missing license headers 2015-09-15 14:09:01 -07:00
Xiang Li
6d1f0ce89f storage: add rev into kv interface 2015-09-15 12:11:00 -07:00
Yicheng Qin
c082488e23 Merge pull request #3507 from yichengq/watch
storage: support basic watch
2015-09-15 00:04:36 -07:00
Yicheng Qin
ec43e0a4c3 storage: introduce WatchableKV and watch feature
WatchableKV is an interface upon KV, and supports watch feature.
2015-09-14 23:53:03 -07:00
Xiang Li
451cce4a90 Merge pull request #3516 from xiang90/hash_improved
storage: support hash state
2015-09-13 21:46:12 -07:00
Xiang Li
714b5e0b08 storage: support hash state 2015-09-13 21:34:58 -07:00
Dmitry Smirnov
b2f4a5f587 *: fix spelling issues (codespell).
Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>
2015-09-11 10:22:29 +10:00
Xiang Li
3f18ded10a *: v3api index->revision 2015-09-04 10:41:20 -07:00
Yicheng Qin
215f27c2f5 storage: add mock tests for store struct 2015-09-04 08:53:49 -07:00
Yicheng Qin
5ae2eb4731 storage: avoid one extra round of wait
It could exit early if it knows that there is no more keys.
2015-09-03 19:12:27 -07:00
Yicheng Qin
9175df7c71 storage: correct revision for range when deleteRange
to make it logically reasonable.
2015-09-03 19:12:27 -07:00
Yicheng Qin
797a4796d9 storage: remove check for DELETE type KeyValue
kvindex always returns kvs that exist at given revision, so there is no
need to check for whether the KeyValue range from backend is DELETE type.
2015-09-03 19:12:27 -07:00
Yicheng Qin
00e31f13a6 storage: remove unnecessary rev parameter 2015-09-03 19:12:27 -07:00
Xiang Li
ef7cf058a2 *: update gogoproto 2015-09-03 15:32:25 -07:00
Tamir Duberstein
45390b9fb8 *: regenerate proto to use local import path
Using Go-style import paths in protos is not idiomatic. Normally, this
detail would be internal to etcd, but the path from which gogoproto
is imported affects downstream consumers (e.g. cockroachdb).

In cockroach, we want to avoid including `$GOPATH/src` in our protoc
include path for various reasons. This patch puts etcd on the same
convention, which allows this for cockroach.

More information: https://github.com/cockroachdb/cockroach/pull/2339#discussion_r38663417

This commit also regenerates all the protos, which seem to have
drifted a tiny bit.
2015-09-03 13:38:28 -04:00
Yicheng Qin
1eaf169057 Merge pull request #3395 from yichengq/backend-test
storage/backend: add unit tests for backend and batchTx
2015-09-03 07:23:38 -07:00
Yicheng Qin
44fd734038 storage/backend: add unit tests for backend and batchTx 2015-09-02 16:57:13 -07:00
Yicheng Qin
a21166c3aa storage: extend timeout to wait for put complete
travis is sometimes slow, and it could fail to complete the put in 10ms.
2015-09-01 09:03:03 -07:00
Yicheng Qin
4b9b0cbcc1 storage: add newBackend and newBatchTx
This is for ease of testing.
2015-08-31 13:25:10 -07:00
Xiang Li
b9632e0f8d storage: register txnCounter 2015-08-28 15:17:16 -07:00
Xiang Li
dd443be41b storage: report total number of keys 2015-08-28 15:16:53 -07:00
Yicheng Qin
054fab84ee storage/backend: remove startc var
This makes start logic cleaner.
2015-08-28 13:52:31 -07:00
Xiang Li
b5838edb93 storage: add initial metrics for kv 2015-08-28 13:41:42 -07:00
Yicheng Qin
f04884f74d storage/backend: fix off-by-one error for pending var
Or it may commit until batchLimit + 1.
2015-08-27 22:51:32 -07:00
Yicheng Qin
7ed929fb3d storage/backend: fix limit doesn't effect in range 2015-08-27 22:51:32 -07:00
Matt Keller
19a28c8efd storage: Fixed backend test
./backend_test.go:23: multiple-value batchTx.UnsafeRange() in single-value context
2015-08-27 15:20:29 -04:00
Yicheng Qin
0d38c13990 storage: use temp path to handle test file 2015-08-26 15:01:41 -07:00
Yicheng Qin
2d01eb4e11 storage: add tests for kvstore_compaction 2015-08-26 15:01:13 -07:00
Yicheng Qin
0813139140 storage: add more tests for index 2015-08-26 12:53:30 -07:00
Yicheng Qin
3723f01b48 storage: add more unit tests for keyIndex 2015-08-26 12:53:30 -07:00
Yicheng Qin
ad8a291dc1 storage: return error when tombstone on new generation
It is not allowed to put tombstone on an empty generation.
2015-08-26 12:53:30 -07:00
Yicheng Qin
ffa87f9678 storage: fix the comment in generation.walk 2015-08-26 12:53:30 -07:00
Xiang Li
e8f40b0412 storage/backend: add commitAndStop
After the upgrade of boltdb, db.Close waits for all txn to finish.
CommitAndStop commits the current txn and stop creating new ones.
2015-08-25 10:57:25 -07:00
Xiang Li
e3ef1d363a Merge pull request #3366 from xiang90/v3_proto
update v3 proto and doc
2015-08-24 11:22:29 -07:00
Xiang Li
3a60d490d1 storagepb: fix comment location 2015-08-24 10:42:16 -07:00