239 Commits

Author SHA1 Message Date
Ben Cox
c20cc05fc5
mvcc: Export a "Last DB compaction" timestamp metric (#12176)
This is to aid with debugging the effectiveness of systems that
manually take care of cluster compaction, and have greater visibity
into recent compactions.

It can be handy to alert on the exactly how long it was since a
compaction (and also to put on dashboards) had happened.

---

Tested using a test cluster, the final result looks like this:

```
	root@etcd-1:~# ETCDCTL_API=3 /tmp/test-etcd/etcdctl --endpoints=192.168.232.10:2379 compact 1012
	compacted revision 1012

	root@etcd-1:~# curl -s 192.168.232.10:2379/metrics | grep last
	# HELP etcd_debugging_mvcc_db_compaction_last The unix time since the last db compaction.  Resets to 0 on start.
	# TYPE etcd_debugging_mvcc_db_compaction_last gauge
	etcd_debugging_mvcc_db_compaction_last 1.595873939e+09
```
2020-08-26 16:27:10 -07:00
Ankur Gargi
2b79442d8e
backend: Create base type for readTx and concurrentReadTx (#11976)
backend: Create base type for readTx and concurrentReadTx

backend: Implemented review comments to rename rTx to baseReadTx and remove TODO

backend: Resolved comments around baseReadTx

backend: Implemented a review comment
2020-06-25 10:26:17 -07:00
CFC4N
26f6aee395
mvcc: chanLen 1024 is to biger,and it used more memory. (#11987)
128 seems to be enough.
Sometimes the consumption speed is more than the production speed.

See https://github.com/etcd-io/etcd/issues/11906 for more detail.
2020-06-10 14:35:48 -07:00
tangcong
26c930f27d mvcc: push down RangeOptions.limit argv into index tree 2020-06-09 01:11:06 +08:00
David Crawshaw
66cb045d47 etcdserver, et al: add --unsafe-no-fsync flag
This makes it possible to run an etcd node for testing and development
without placing lots of load on the file system.

Fixes #11930.

Signed-off-by: David Crawshaw <crawshaw@tailscale.com>
2020-05-26 09:21:49 +10:00
Jack Kleeman
ec13797407
mvcc: avoid negative watcher count metrics (#11882)
The watch count metrics are not robust to duplicate cancellations. These
cause the count to be decremented twice, leading eventually to negative
counts. We are seeing this in production. The duplicate cancellations
themselves are not themselves a big problem (except performance), but
they are caused by the new proactive cancellation logic (#11850) which
cancels proactively even immediately before initiating a Close, thus
nearly guaranteeing a Close-cancel race, as discussed in
watchable_store.go. We can avoid this in most cases by not sending a
cancellation when we are going to Close.
2020-05-12 16:50:53 -07:00
Ted Yu
2dbbe6ce85 mvcc: Obtain txLock once in readTx#UnsafeRange
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
2020-05-09 16:28:47 -07:00
Gyuho Lee
d472f67c77
Merge pull request #11817 from tangcong/fix-mvcc-deadlock-bug
mvcc: fix a deadlock bug in mvcc
2020-05-07 16:03:45 -07:00
Brandon Philips
d8bbf7f58d
Merge pull request #11816 from gaurav1086/fix_race_condition_kv_test
mvcc: fix race in TestKVTxnBlockWriteOperations
2020-04-28 16:11:26 -07:00
Gaurav Singh
e5ddc91228 mvcc: fix race in TestKVTxnBlockWriteOperations 2020-04-28 10:53:25 -04:00
Brandon Philips
96cce208c2 go.mod: use go.etcd.io/etcd/v3 versioning
This change makes the etcd package compatible with the existing Go
ecosystem for module versioning.

Used this tool to update package imports:
  https://github.com/KSubedi/gomove
2020-04-28 00:57:35 +00:00
tangcong
8fa7783875 mvcc: fix deadlock bug 2020-04-27 06:42:06 +08:00
tangcong
730f3f1d78 mvcc: reduce count-only range overhead 2020-04-10 19:23:38 +08:00
Gyuho Lee
3ac7a11515
Merge pull request #11699 from tangcong/refactor-consistentindex
*: refactor consistentindex
2020-03-25 18:07:44 -07:00
tangcong
7b2018683a *: refactor consistent index 2020-03-25 10:59:15 +08:00
fengpf
1f70c0b075 *: fix goroutines typos 2020-03-24 15:33:47 +08:00
mlmhl
0eee733220
mvcc: no need to hold tx lock when converting kvs to watch events (#11701)
Method `kvsToEvents` can take a long time if a large number of events need to be synchronized. However, this method does not access any shared resources, so it no need to hold the tx lock when executing. So we can move it outside to reduce lock holding time.
2020-03-19 21:01:36 -07:00
tangcong
06ad53321e *: fix auth revision corruption bug 2020-02-29 13:31:37 +08:00
jingyih
b6ee807e93 mvcc/backend: check for nil boltOpenOptions
Check if boltOpenOptions is nil before use it.
2020-02-14 21:26:52 -08:00
Joe Betz
213f7f7877
mvcc/backend: Delete orphaned db.tmp files before defrag 2020-02-12 22:40:28 -08:00
jingyih
6b389bf23c mvcc: remove capnslog 2020-02-07 07:44:44 -08:00
sfzhu93
cad92706cf
in multiple packages: fixed goroutine leak bugs in tests (#11569) 2020-01-30 10:45:59 -08:00
Ted Yu
c3497c84ed mvcc: use correct error variable in defragdb (#11527)
Co-authored-by: yutedz <54038175+yutedz@users.noreply.github.com>
2020-01-22 11:03:23 -08:00
Gyuho Lee
a592b808c5
Merge pull request #11538 from andyxning/fix_typo
mvcc: fix error log typo
2020-01-17 13:10:11 -08:00
Andy Xie
f260df559d mvcc: fix error log typo 2020-01-17 17:27:04 +08:00
Ted Yu
8aa7a3dc09 mvcc: log error if tmpdb is not removed (#11523)
Co-authored-by: yutedz <54038175+yutedz@users.noreply.github.com>
2020-01-16 17:34:45 -08:00
zimash
e5a35c827c mvcc: incomplete fix in #11502 (#11533) 2020-01-16 12:05:10 -08:00
Ted Yu
250b0acac4 mvcc: check the error return in defragdb (#11524)
Co-authored-by: yutedz <54038175+yutedz@users.noreply.github.com>
2020-01-15 16:57:12 -08:00
Ted Yu
574ee42ad7 mvcc: rollback tmptx in case of error (#11525)
Co-authored-by: yutedz <54038175+yutedz@users.noreply.github.com>
2020-01-15 13:54:52 -08:00
wangtuanjie
a96bc48268 mvcc: fix snapshot sendRateBytes unit (#11502) 2020-01-14 11:29:05 -08:00
Ted Yu
7f3dd59d22 mvcc: Log warning when compaction encounters error (#11447) 2019-12-13 15:59:49 -08:00
sun
c5887cdc49 mvcc: go style err check (#11419) 2019-12-04 18:46:17 -08:00
宇慕
655df65e93 mvcc/kvstore:fixcompactbug 2019-11-27 17:55:42 +08:00
yoyinzyc
a5e747b2d4 mvcc: update to "etcd_debugging_mvcc_total_put_size_in_bytes" 2019-11-19 15:41:58 -08:00
yoyinzyc
28f40d7794 mvcc: add "etcd_mvcc_put_size_in_bytes" to monitor the throughput of put request. 2019-11-19 13:55:37 -08:00
Jingyi Hu
cbc1340af5
Merge pull request #11330 from shenjiangc/master
mvcc/kvstore: Optimize compaction, slove conflict for #11150
2019-11-04 20:56:01 -08:00
shenjiangc
72cc5dcf39 mvcc/kvstore:when the number key-value is greater than one million, compact take too long and blocks other requests 2019-11-05 11:48:32 +08:00
Ziheng Liu
235d7c2e51 mvcc: Add Unlock before panic to prevent double lock 2019-10-28 22:38:34 -04:00
Ye Kuang
49f4a418f9 backend: fix comment for bucketBuffer.merge() 2019-10-17 08:48:12 +09:00
yoyinzyc
57aa68af5a etcdserver: trace compaction request; add return parameter 'trace' to applierV3.Compaction()
mvcc: trace compaction request; add input parameter 'trace' to KV.Compact()
2019-10-07 09:55:27 -07:00
yoyinzyc
401df4bb8e etcdserver: add put request steps.
mvcc: add put request steps; add trace to KV.Write() as input parameter.
2019-10-01 14:08:06 -07:00
yoyinzyc
1d6ef8370e pkg: use zap logger to format the structure log output. 2019-09-30 13:11:21 -07:00
yoyinzyc
3830b3ef11 pkg: add field to record additional detail of trace; add stepThreshold
to reduce log volume.
2019-09-30 13:11:21 -07:00
yoyinzyc
f4e7fc56a7 pkg: create package traceutil for tracing. mvcc: add tracing
steps:range from the in-memory index tree; range from boltdb.
etcdserver: add tracing steps: agreement among raft nodes before
linerized reading; authentication; filter and sort kv pairs; assemble
the response.
2019-09-30 13:06:02 -07:00
Jingyi Hu
88d998be4c
Merge pull request #10975 from lzhfromustc/currentRev
Add critical section to protect s.currentRev
2019-09-06 12:27:53 -07:00
Ziheng Liu
077dd74827 mvcc: add a TODO in (*store).Hash, to warn against potential data race of currentRev and suggest feasible fix 2019-09-06 15:05:43 -04:00
Jingyi Hu
0f8c46a0f3 mvcc: add store revision metrics
Add experimental metrics etcd_debugging_mvcc_current_revision and
etcd_debugging_mvcc_compact_revision.
2019-09-05 15:40:06 -07:00
Gyuho Lee
f279bfb775 mvcc: keep 64-bit alignment in "store" struct
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2019-08-15 13:01:32 -07:00
Joe Betz
9b51febaf5 *: Add experimental-compaction-batch-limit flag 2019-08-15 11:47:23 -07:00
Joe Betz
d57bc6e724 mvcc: Optimize compaction for short commit pauses 2019-08-15 11:47:23 -07:00