57 Commits

Author SHA1 Message Date
Xiang Li
558640d91e backend: support shrink db 2016-03-02 14:35:28 -08:00
Xiang Li
40c598cfa6 Merge pull request #4494 from xiang90/sched
*: fix schedule.Wait race
2016-02-11 15:56:01 -08:00
Xiang Li
bfa5e310a9 *: detach keys from leases
1. deatch when a key is removed
2. deatch when the key's lease changes
3. potentially deatch when restroing a tombstone key
2016-02-11 15:31:25 -08:00
Xiang Li
870e4c2681 *: fix schedule.Wait race 2016-02-11 15:21:33 -08:00
Xiang Li
d314345e6d *: add a scheduler and use it to schedule compaction 2016-02-10 14:27:08 -08:00
Anthony Romano
ecba9b61cf storage: start initial revision at 1
When the start revision was 0, there was no way to safely watch
starting from the first store revision.
2016-01-14 17:05:56 -08:00
Gyu-Ho Lee
497bbd3010 *: FatalStack to stacktrace tests after timeout
Related to https://github.com/coreos/etcd/issues/4065.
2016-01-14 10:20:14 -08:00
Xiang Li
0cba12d991 storage: extend timeout for slow CI
1. extend timeout

2. print out stacktrace. When it fails again, we can get more confidence that the
failure is caused by slow IO.
2016-01-13 16:04:19 -08:00
Xiang Li
f5753f2f51 *: support lease Attach
Now we can attach keys to leases. And revoking the lease removes all
the attached keys of that lease.
2016-01-09 11:01:58 -08:00
Anthony Romano
8774d53459 Merge pull request #4158 from heyitsanthony/nolease-to-leasepkg
lease: move storage.NoLease to lease package
2016-01-07 10:43:51 -08:00
Anthony Romano
f9af744be3 lease: move storage.NoLease to lease package 2016-01-07 10:33:44 -08:00
Xiang Li
1714290f4e storage: support recovering from backend
We want the KV to support recovering from backend to avoid
additional pointer swap. Or we have to do coordination between
etcdserver and API layer, since API layer might have access to
kv pointer and use a closed kv.
2016-01-06 21:16:55 -08:00
Xiang Li
5dd3f91903 *: make backend outside kv
KV and lease will share the same backend. Thus we need to make
backend outside KV.
2016-01-05 19:55:29 -08:00
Xiang Li
1e61243fd7 Merge pull request #4134 from xiang90/lease
lease: modify API and persist lease to disk
2016-01-05 10:30:52 -08:00
Xiang Li
09b420f08c *: move leaseID typedef to lease pkg 2016-01-05 10:18:17 -08:00
Anthony Romano
384cc76299 pkg/testutil: make Recorder an interface
Provides two implementations of Recorder-- one that is non-blocking
like the original version and one that provides a blocking channel
to avoid busy waiting or racing in tests when no other synchronization
is available.
2016-01-05 09:39:18 -08:00
Xiang Li
4336278b44 *: support put with lease 2016-01-04 15:54:06 -08:00
Hitoshi Mitake
0b2d31f3bc storage: decouple default parameters and storage creation
newStore() uses constants for some important parameters
e.g. batchInerval. It is not suitable for other storage users
(e.g. tools/benchmark). This commit decouples the default parameters
from storage creation. etcd should use a newly added function
newDefaultStore() for creating a store with default parameters.
2015-12-31 22:28:59 +09:00
Gyu-Ho Lee
6c5dc28d0f Merge pull request #4043 from gyuho/storage_range_all_unsynced
storage: range all unsynced at once
2015-12-28 14:45:40 -07:00
Gyu-Ho Lee
ecc3e15a46 storage: delete RangeHistory
This has been replaced by operations inside `syncWatchings`.
2015-12-28 11:37:32 -08:00
Gyu-Ho Lee
df0c2e6842 storage: remove unnecessary math/rand seed
As @jonboulle pointed out at
https://github.com/coreos/etcd/pull/4070/files#r48441847:

> math/rand is unrelated to crypto/rand; the latter reads from /dev/urandom and
> is relying on the kernel's PRNG. Just remove the seed entirely.
2015-12-25 09:55:11 -08:00
Gyu-Ho Lee
84d777305d storage: use only one mutex for store struct
Mutex is a variable, which means there needs to be only one mutex
value per scope. We don't need a separate mutex inside store struct,
**if we assume that `TxnPut` and `TxnRange` are called ONLY ONCE
per transaction (between `TxnBegin` and `TxnEnd`)**, as documented.
2015-12-21 11:42:40 -08:00
Hitoshi Mitake
af2569c2d8 storage, test: unlock transaction in the retry loop 2015-12-17 10:35:03 +09:00
Gyu-Ho Lee
891cdd56b4 storage: newFakeStore to return only 1
This changes newFakeStore function to return only 1 because two other arguments
are already in the first return storage struct.
2015-12-13 17:46:47 -08:00
Gyu-Ho Lee
d55ab7798b storage: change var names from 'index'
Having the same variable name as 'index' type
can be confusing and shadowing other variables.
This gives different variable names to 'index' variables.
2015-12-12 16:37:24 -08:00
Xiang Li
a0eca5fd37 storage: make RestoreContinueUnfinishedCompaction more reliable 2015-12-04 11:48:46 -08:00
Yicheng Qin
2f74f76025 storage: remove the event concept from key-value layer
The point is to decouple the key-value storage layer and the
event notification layer clearly. It gives the watchableKV the
flexibility to define whatever event structure it wants without
breaking the ondisk format at key-value storage layer.

Changes:

1. change the format of key and value stored in backend

Store KeyValue struct instead of Event struct in backend value for
better abstraction as xiang suggests. And record the corresponded
action in the backend key.

2. Remove word 'event' from functions
2015-11-17 20:35:49 -08:00
Yicheng Qin
59b5dabc66 storage: extend wait timeout for execution
Extend timeout to pass always in traivs.
2015-10-30 17:44:31 -07:00
Xiang Li
6556bf1643 storage: remove the endRev of watcher 2015-10-17 15:59:49 -07:00
Xiang Li
21179d929f Merge pull request #3616 from yichengq/storage-txn
storage: hold batchTx lock during KV txn
2015-10-05 17:12:52 -07:00
Xiang Li
0aa2f1192a storage: add metrics for db total size 2015-10-05 16:56:30 -07:00
Yicheng Qin
c97dda766e storage: hold batchTx lock during KV txn
One txn is treated as atomic, and might contain multiple Put/Delete/Range
operations. For now, between these operations, we might call forecCommit
to sync the change to disk, or backend may commit it in background.
Thus the snapshot state might contains an unfinished multiple objects
transaction, which is dangerous if database is restored from the snapshot.

This PR makes KV txn hold batchTx lock during the process and avoids
commit to happen.
2015-10-03 16:01:05 -07:00
Yicheng Qin
36f4303fc3 storage/etcdserver: update KV.Snapshot function
When using Snapshot function, it is expected:
1. know the size of snapshot before writing data
2. split snapshot-ready phase and write-data phase. so we could cut
snapshot first and write data later.

Update its interface to fit the requirement of etcdserver.
2015-10-03 10:15:23 -07:00
Xiang Li
385e17583f storage: fix hash by iterating kv 2015-09-23 11:28:33 -07:00
Yicheng Qin
d72914c36f storage: clarify comment for store.RangeEvents and fix related bugs
Change to the function:
1. specify the meaning of startRev and endRev parameters
2. specify the meaning of returned nextRev

Moreover, it adds unit tests for the function.
2015-09-19 23:17:03 -07:00
Jonathan Boulle
7848ac3979 *: add missing license headers 2015-09-15 14:09:01 -07:00
Xiang Li
6d1f0ce89f storage: add rev into kv interface 2015-09-15 12:11:00 -07:00
Yicheng Qin
ec43e0a4c3 storage: introduce WatchableKV and watch feature
WatchableKV is an interface upon KV, and supports watch feature.
2015-09-14 23:53:03 -07:00
Xiang Li
3f18ded10a *: v3api index->revision 2015-09-04 10:41:20 -07:00
Yicheng Qin
215f27c2f5 storage: add mock tests for store struct 2015-09-04 08:53:49 -07:00
Yicheng Qin
0d38c13990 storage: use temp path to handle test file 2015-08-26 15:01:41 -07:00
Yicheng Qin
9c0c314425 storage: add functional tests for the package
It adds and reorganize tests to construct functional tests.
2015-08-23 20:59:06 -07:00
Tyler Neely
acd7a92f03 storage: reversion -> revision 2015-08-20 08:39:07 -07:00
Xiang Li
53a77fa519 *: tnx -> txn 2015-07-24 23:21:09 +08:00
Xiang Li
ccca2b04da storage: save version 2015-06-29 13:15:09 -07:00
Yicheng Qin
148394f66f storage: fix schedule compaction bug in recover process
It uses wrong schedule compaction reversion before.
2015-06-18 11:11:37 -07:00
Yicheng Qin
26a09d8479 storage: enhance TestRestore and kill TODO 2015-06-18 10:37:12 -07:00
Yicheng Qin
80a59f00b7 storage: fix limit mismatch in Range func 2015-06-17 17:43:08 -07:00
Yicheng Qin
93f477944b storage: return ErrFutureRev if rev is a future one 2015-06-17 17:42:43 -07:00
Yicheng Qin
94924d04db storage: add TestRangeBadRev 2015-06-17 16:22:28 -07:00