48 Commits

Author SHA1 Message Date
Gyu-Ho Lee
ecc3e15a46 storage: delete RangeHistory
This has been replaced by operations inside `syncWatchings`.
2015-12-28 11:37:32 -08:00
Gyu-Ho Lee
84d777305d storage: use only one mutex for store struct
Mutex is a variable, which means there needs to be only one mutex
value per scope. We don't need a separate mutex inside store struct,
**if we assume that `TxnPut` and `TxnRange` are called ONLY ONCE
per transaction (between `TxnBegin` and `TxnEnd`)**, as documented.
2015-12-21 11:42:40 -08:00
Yicheng Qin
2f74f76025 storage: remove the event concept from key-value layer
The point is to decouple the key-value storage layer and the
event notification layer clearly. It gives the watchableKV the
flexibility to define whatever event structure it wants without
breaking the ondisk format at key-value storage layer.

Changes:

1. change the format of key and value stored in backend

Store KeyValue struct instead of Event struct in backend value for
better abstraction as xiang suggests. And record the corresponded
action in the backend key.

2. Remove word 'event' from functions
2015-11-17 20:35:49 -08:00
Yicheng Qin
263b270708 etcdserver: commit v3 storage before releasing WAL
This ensures that v3 storage could always find the following log entries
when restart.
2015-10-26 21:06:08 -07:00
Yicheng Qin
15ed6d8268 etcdserver: save consistent index into v3 storage
This helps to recover consistent index when restart in the future.
2015-10-24 09:27:24 -07:00
Xiang Li
6556bf1643 storage: remove the endRev of watcher 2015-10-17 15:59:49 -07:00
Xiang Li
21179d929f Merge pull request #3616 from yichengq/storage-txn
storage: hold batchTx lock during KV txn
2015-10-05 17:12:52 -07:00
Xiang Li
0aa2f1192a storage: add metrics for db total size 2015-10-05 16:56:30 -07:00
Yicheng Qin
c97dda766e storage: hold batchTx lock during KV txn
One txn is treated as atomic, and might contain multiple Put/Delete/Range
operations. For now, between these operations, we might call forecCommit
to sync the change to disk, or backend may commit it in background.
Thus the snapshot state might contains an unfinished multiple objects
transaction, which is dangerous if database is restored from the snapshot.

This PR makes KV txn hold batchTx lock during the process and avoids
commit to happen.
2015-10-03 16:01:05 -07:00
Yicheng Qin
36f4303fc3 storage/etcdserver: update KV.Snapshot function
When using Snapshot function, it is expected:
1. know the size of snapshot before writing data
2. split snapshot-ready phase and write-data phase. so we could cut
snapshot first and write data later.

Update its interface to fit the requirement of etcdserver.
2015-10-03 10:15:23 -07:00
Xiang Li
385e17583f storage: fix hash by iterating kv 2015-09-23 11:28:33 -07:00
Yicheng Qin
d72914c36f storage: clarify comment for store.RangeEvents and fix related bugs
Change to the function:
1. specify the meaning of startRev and endRev parameters
2. specify the meaning of returned nextRev

Moreover, it adds unit tests for the function.
2015-09-19 23:17:03 -07:00
Jonathan Boulle
7848ac3979 *: add missing license headers 2015-09-15 14:09:01 -07:00
Xiang Li
6d1f0ce89f storage: add rev into kv interface 2015-09-15 12:11:00 -07:00
Yicheng Qin
c082488e23 Merge pull request #3507 from yichengq/watch
storage: support basic watch
2015-09-15 00:04:36 -07:00
Yicheng Qin
ec43e0a4c3 storage: introduce WatchableKV and watch feature
WatchableKV is an interface upon KV, and supports watch feature.
2015-09-14 23:53:03 -07:00
Xiang Li
714b5e0b08 storage: support hash state 2015-09-13 21:34:58 -07:00
Xiang Li
3f18ded10a *: v3api index->revision 2015-09-04 10:41:20 -07:00
Yicheng Qin
9175df7c71 storage: correct revision for range when deleteRange
to make it logically reasonable.
2015-09-03 19:12:27 -07:00
Yicheng Qin
797a4796d9 storage: remove check for DELETE type KeyValue
kvindex always returns kvs that exist at given revision, so there is no
need to check for whether the KeyValue range from backend is DELETE type.
2015-09-03 19:12:27 -07:00
Yicheng Qin
00e31f13a6 storage: remove unnecessary rev parameter 2015-09-03 19:12:27 -07:00
Xiang Li
b5838edb93 storage: add initial metrics for kv 2015-08-28 13:41:42 -07:00
Yicheng Qin
9960651c3f storage: let range work in the process of txn
range should work in the process of txn to help check the status during the
txn.
2015-08-23 20:59:06 -07:00
Yicheng Qin
6d97dcaf3f storage: ensure that desired compaction is persisted
It needs to persist the desired compaction, so it won't forget the compaction
if it crashes later.
2015-08-23 20:59:06 -07:00
Yicheng Qin
353f10ca2b storage: reject to compact on future rev
Compaction on future rev is unreasonable.
2015-08-23 20:59:06 -07:00
Yicheng Qin
47b243be5d storage: let TxnDeleteRange return rev if no error
If it doesn't return error, it should return valid rev.
2015-08-23 20:59:06 -07:00
Tyler Neely
acd7a92f03 storage: reversion -> revision 2015-08-20 08:39:07 -07:00
Xiang Li
53a77fa519 *: tnx -> txn 2015-07-24 23:21:09 +08:00
Xiang Li
883bb47dcf Merge pull request #3074 from xiang90/storage_restore
storage: correctly restore create and ver
2015-06-30 09:20:19 -07:00
Xiang Li
f8b947a00b storage: fix small issues 2015-06-29 22:02:21 -07:00
Xiang Li
433f2ee1bc storage: correctly restore create and ver
Add a restore func to correctly restore create reversion and
version of keys for the index.
2015-06-29 13:44:43 -07:00
Xiang Li
ccca2b04da storage: save version 2015-06-29 13:15:09 -07:00
Xiang Li
c069119abe Merge pull request #3067 from xiang90/storage_created_mod
storage: save created index and modified index
2015-06-27 23:11:05 -07:00
Xiang Li
4581064060 storage: save created index and modified index 2015-06-26 12:10:26 -07:00
Yicheng Qin
9f2e4c8a57 storage: remove unnecessary ForceCommit in kvstore.Close
s.b.Close will commit pending ops, so there is no need to FroceCommit
it in kvstore.Close()
2015-06-18 13:36:23 -07:00
Yicheng Qin
7cba42fb73 storage: wait for compact goroutine to exit before close backend
If backend is closed, the operations on backend in compact
goroutine will panic. So this PR waits for compact goroutine to exit
before close backend.

This fixes the TestWorkflow failure too.
2015-06-18 12:18:39 -07:00
Yicheng Qin
148394f66f storage: fix schedule compaction bug in recover process
It uses wrong schedule compaction reversion before.
2015-06-18 11:11:37 -07:00
Yicheng Qin
80a59f00b7 storage: fix limit mismatch in Range func 2015-06-17 17:43:08 -07:00
Yicheng Qin
93f477944b storage: return ErrFutureRev if rev is a future one 2015-06-17 17:42:43 -07:00
Yicheng Qin
7b1a93e1ef storage: put storage info keys into information bucket
They used to be in key bucket, and make recover failed because they
cannot be parsed as normal key.
2015-06-17 14:37:29 -07:00
Xiang Li
ba9a46aa02 storage: initial snapshot and restore
Snapshot takes an io.Writer and writes the entire backend data to
the given writer. Snapshot writes a consistent view and does not
block other storage operations.

Restore restores the in-memory states (index and book keeping) of
the storage from the backend data.
2015-06-10 11:32:10 -07:00
Xiang Li
f47ed4a364 storage: initial compact 2015-06-05 09:22:44 -07:00
Xiang Li
d417b36e5d storage: refactor key_index 2015-05-31 15:24:04 -07:00
Xiang Li
69d02410cf stroage: adopt KV interface 2015-05-27 14:24:23 -07:00
Xiang Li
cbb8b9bb08 stroage: add tnx id 2015-05-27 10:35:51 -07:00
Xiang Li
9c1aec6877 storage: add rangeKeys func 2015-05-27 09:58:21 -07:00
Xiang Li
93ecf36855 storage: support tnx 2015-05-27 09:31:11 -07:00
Xiang Li
9db360387d storage: support Range 2015-05-27 09:31:11 -07:00