Piotr Tabor
e90504fe62
Unify shared code (and constants) with cindex package.
2021-04-29 11:51:25 +02:00
Piotr Tabor
f53b70facb
Embed: In case KVStoreHash verification fails, close the backend.
...
In case of failed verification, the server used to keep opened backend
(so the file was locked on OS level).
2021-04-29 11:51:25 +02:00
Piotr Tabor
911204cd76
Fix ETCDCTL_API=2 etcdctl backup --with-v3 consistent index consistency
...
Prior to this CL, `ETCDCTL_API=2 etcdctl backup --with-v3` was readacting WAL log
(by removal of some entries), but was NOT updating consistent_index in the backend.
Also the WAL editing logic was buggy, as it didn't took in consideration the fact
that when TERM changes, there can be entries with duplicated indexes in
the log. So its NOT sufficient to subtract number of removed entries to
get accurate log indexes.
The PR replaces removing and shifting of WAL entries with replacing them with an no-op entries.
Thanks to this consistent-index references are staying up to date.
The PR also:
- updates 'verification' logic to check whether consistent_index does not lag befor last snapshot
- env-gated execution of verification framework in `etcdctl backup`.
Tested with:
```
(./build.sh && cd tests && EXPECT_DEBUG=TRUE 'env' 'go' 'test' '-timeout=300m' 'go.etcd.io/etcd/tests/v3/e2e' -run=TestCtlV2Backup --count=1000 2>&1 | tee TestCtlV2BackupV3.log)
```
2021-04-29 11:51:24 +02:00
Piotr Tabor
067521981e
v2 etcdctl backup: producing consistent state of membership
2021-04-27 19:34:34 +02:00
Piotr Tabor
a70386a1a4
Simplify membership interface: Does not pass the 'unused' token.
2021-04-27 17:17:31 +02:00
Piotr Tabor
7ae3d25f91
Membership: Add additional methods to trim/manage membership data in backend.
2021-04-27 17:17:31 +02:00
Piotr Tabor
768da490ed
sever: v2store deprecation: Fix etcdctl snapshot restore to restore
...
correct 'backend' (bbolt) context in aspect of membership.
Prior to this change the 'restored' backend used to still contain:
- old memberid (mvcc deletion used, why the membership is in bolt
bucket, but not mvcc part):
```
mvs := mvcc.NewStore(s.lg, be, lessor, ci, mvcc.StoreConfig{CompactionBatchLimit: math.MaxInt32})
defer mvs.Close()
txn := mvs.Write(traceutil.TODO())
btx := be.BatchTx()
del := func(k, v []byte) error {
txn.DeleteRange(k, nil)
return nil
}
// delete stored members from old cluster since using new members
btx.UnsafeForEach([]byte("members"), del)
```
- didn't get new members added.
2021-04-27 17:17:30 +02:00
Piotr Tabor
9a4b2bdccc
Errors: context cancelled or context deadline exceeded are exposed as codes.Canceled, codes.DeadlineExceeded instead of 'codes.Unknown'
2021-04-22 14:35:24 +02:00
Piotr Tabor
d7d110b5a8
mvcc/backend tests: Refactor: Do not mix testing&prod code.
2021-04-21 09:43:13 +02:00
Piotr Tabor
ea287dd9f8
Merge pull request #12854 from ptabor/20210410-shouldApplyV3
...
(no)StoreV2 (Part 3): Applying consistency fix: ClusterVersionSet (and co) might get not applied on v2store
2021-04-21 09:31:38 +02:00
Sam Batschelet
0f2c940f64
Merge pull request #12880 from chaochn47/exclude_alarms_from_health_check
...
etcdhttp/metrics.go: exclude alarms from health check conditionally with `?exclude=NOSPACE`
2021-04-20 21:18:15 -04:00
Chao Chen
140ea4fa29
etcdhttp/metrics.go: exclude alarms from health check conditionally with ?exclude=NOSPACE
2021-04-20 13:17:09 -07:00
Piotr Tabor
06ba0fc5a2
Merge pull request #12846 from pyiyun/fix-snaptmpfile-bug
...
etcdserver: remove temp files in snap dir when etcdserver starting
2021-04-17 12:58:46 +02:00
chao
80586c5b47
etcdserver/util.go: reduce memory when logging range requests ( #12871 )
...
* etcdserver/util.go: reduce memory when logging range requests
Fixes #12835
* Update CHANGELOG-3.5.md
2021-04-16 16:39:34 -07:00
Piotr Tabor
17b982382e
Fix TestSnapshotV3RestoreMultiMemberAdd flakes (leaks)
...
- most important: unix's socket transport should not keep idle
connections. For top-level Transport we close them using:
f3c518025e/server/etcdserver/api/rafthttp/transport.go (L226)
2021-04-16 20:17:28 +02:00
pyiyun
28a490b09c
etcdserver: remove temp files in snap dir when etcdServer starting
...
When etcd exits abnormally, tmp files will remain in snap dir, so clean up tmp files in snap dir when etcdserver starting.
Fixes #12837
2021-04-16 20:30:04 +08:00
Piotr Tabor
d69e46ea47
Make ShouldApplyV3 an enum - not bool
2021-04-13 23:01:03 +02:00
Piotr Tabor
b1c04ce043
Applying consistency fix: ClusterVersionSet (and co) might get no applied on v2store
...
ClusterVersionSet, ClusterMemberAttrSet, DowngradeInfoSet functions are
writing both to V2store and backend. Prior this CL there were
in a branch not executed if shouldApplyV3 was false,
e.g. during restore when Backend is up-to-date (has high
consistency-index) while v2store requires replay from WAL log.
The most serious consequence of this bug was that v2store after restore
could have different index (revision) than the same exact store before restore,
so potentially different content between replicas.
Also this change is supressing double-applying of Membership
(ClusterConfig) changes on Backend (store v3) - that lackilly are not
part of MVCC/KeyValue store, so they didn't caused Revisions to be
bumped.
Inspired by jingyih@ comment:
https://github.com/etcd-io/etcd/pull/12820#issuecomment-815299406
2021-04-12 09:43:48 +02:00
wpedrak
3991a8c9fa
etcdserver: replace forceVersionC with FirstCommitInTermNotify
2021-04-09 11:30:42 +02:00
wpedrak
3d485faac5
etcdserver: resend ReadIndex request on empty apply request
...
Empty apply indicates first commit in current term. It is first time when follower is sure, that it's ReadIndex request can be processed.
2021-04-09 11:30:42 +02:00
Piotr Tabor
cc0f812f51
server_test.go: Use context.Background() instead of TODO in tests.
...
Suggected in: https://golang.org/pkg/context/#Background
2021-04-08 01:15:16 +02:00
Piotr Tabor
931af493cf
Merge pull request #12830 from ptabor/20210405-split-pkg
...
Split client/pkg as dedicated low-dependencies module for client
2021-04-08 01:12:17 +02:00
Piotr Tabor
3bb7acc8cf
Migrate dependencies pkg/foo -> client/pkg/foo
2021-04-07 00:38:47 +02:00
Piotr Tabor
55ccbe62a2
membership/cluster_test: Use zaptest logger.
2021-03-26 13:54:59 +01:00
Piotr Tabor
03f55eeb2c
Make NewTmpBackend use testing tmp location (so cleanup).
2021-03-26 13:54:55 +01:00
Piotr Tabor
4d4c84e014
server: Written Snapshot's to WAL contains populated ConfState.
...
This will (among others) allow etcd-3.6 to not depend on store_v2 .snap files at all,
as WAL + db file will be self-sufficient.
2021-03-25 00:31:44 +01:00
Piotr Tabor
7d7a9c6f23
codec.go: should use google runtime
...
golang-proto runtime can deal with both: gogo & golang generated
protobufs. It does not work vice-versa with protobuf-1.5.1.
2021-03-24 22:06:47 +01:00
Joel Smith
19f7c6ef3e
*: Update gogo/protobuf to v1.3.2, rerun ./scripts/genproto.sh
...
While it appears that etcd is not vulnerable to CVE-2021-3121,
it is a good idea to update to the new generator so that new
vulnerable code isn't generated in any future APIs. Also, this
lays the issue to rest of whether there is any issue with
etcd and CVE-2021-3121.
2021-03-23 11:48:06 -06:00
Piotr Tabor
30ce6067da
Merge pull request #12780 from wpedrak/read_index_retry
...
Read index retry
2021-03-23 13:47:41 +01:00
wpedrak
dac6e37ea1
*: over 20 staticcheck fixes
2021-03-18 15:06:17 +01:00
Piotr Tabor
a84bd093b0
Integration with grpc-settable logger.
2021-03-16 22:50:41 +01:00
wpedrak
e9779231ec
server: add 500ms retries to ReadIndex requests for l-reads
...
It is second approach (with first being #12762 ) to solve #12680
2021-03-16 16:34:15 +01:00
wpedrak
4b21e38381
refactored l-read loop in v3_server.go
2021-03-16 11:03:45 +01:00
Piotr Tabor
1e7c1805d8
Unify logic of building raft-loggers for etcd.
...
1. We had the same code copied 3 times.
2. For no good reason the code was not reusing existing logger if this one is given.
2021-03-14 16:02:50 +01:00
Piotr Tabor
44bd22307e
Merge get_logger() & Logger() method.
2021-03-14 14:05:17 +01:00
Piotr Tabor
fd7fed1511
Move config (ServerConfig) out of etcdserver package.
...
Motivation:
- ServerConfig is part of 'embed' public API, while etcdserver is more 'internal'
- EtcdServer is already too big and config is pretty wide-spread leaf
if we were to split etcdserver (e.g. into pre & post-apply part).
2021-03-11 20:56:22 +01:00
Gyuho Lee
3ead91ca3e
Merge pull request #12739 from LeoYang90/optimization_watch_prevkv
...
create event do not need prevkv range
2021-03-10 09:48:42 -08:00
Piotr Tabor
fb1d48e98e
Integration tests: Use BeforeTest(t) instead of defer AfterTest().
...
Thanks to this change, a single method BeforeTest(t) can handle
before-test logic as well as registration of cleanup code
(t.Cleanup(func)).
2021-03-09 18:19:51 +01:00
Gyuho Lee
94a371acd7
Merge pull request #12750 from ptabor/20210306-mlock
...
--experimental-memory-mlock support
2021-03-09 09:13:40 -08:00
Gyuho Lee
6fd85af641
Merge pull request #12702 from hexfusion/add-so
...
*: add support for socket options
2021-03-09 09:02:24 -08:00
Sam Batschelet
5b49fb41c8
fixup: add ListenerOptions
...
Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
2021-03-08 11:27:03 -05:00
Piotr Tabor
a46a358577
--experimental-memory-mlock support
...
The flag protects etcd memory from being swapped out to disk.
This can happen in memory constrained systems where mmaped bbolt
area is natural condidate for swapping out.
This flag should provide better tail latency on the cost of higher RSS
ram usage. If the experiment is successful, the logic should get moved
into bbolt layer, where we can protect specific bbolt instances
(e.g. avoid protecting both during defragmentation).
2021-03-07 12:32:57 +01:00
leoyang.yl
d70f35f8d1
create event do not need prevkv range
2021-03-02 17:43:24 +08:00
Gyuho Lee
3d7aac948b
Merge pull request #12196 from ironcladlou/metrics-watch-error-fix
...
etcdserver: fix incorrect metrics generated when clients cancel watches
2021-02-19 12:46:49 -08:00
Sam Batschelet
49078c683b
*: add support for socket options
...
Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
2021-02-19 13:31:23 -05:00
Maksim Buldukyan
7e38cfcc8d
raft: makes 'ConnReadTimeout/ConnWriteTimeout' customizable
2021-02-10 10:36:50 +07:00
Chao Chen
2ae3e82f07
etcdserver/api/etcdhttp: log successful etcd server side health check in debug level
...
When we have an external component that checks /health periodically, the
etcd server logs can be quite verbose (e.g., DDOS-ing against insure
etcd health check can lead to disk space full due to large log files).
This change was introduced in #11704 .
While we keep the warning logs for etcd health check failures, the
success (or OK) log level should be set to DEBUG.
Fixes #12676
2021-02-08 17:15:43 -08:00
Yanhao Mo
6d82778a4e
etcdserver: export method EtcdServer.leaderChangedNotify ( #12378 )
2021-02-02 18:13:32 +08:00
Piotr Tabor
d6d03beaea
Merge pull request #12538 from lzhfromustc/12_9_GoroutineLeak
...
test: change channel operations to avoid potential goroutine leaks
2021-02-01 21:16:43 +01:00
Sahdev Zala
69e99e80fa
Merge pull request #12465 from spacewander/fdoc
...
chore: update the documentation link in the comment
2021-01-14 00:39:25 -05:00
