Piotr Tabor
e90504fe62
Unify shared code (and constants) with cindex package.
2021-04-29 11:51:25 +02:00
Piotr Tabor
f53b70facb
Embed: In case KVStoreHash verification fails, close the backend.
...
In case of failed verification, the server used to keep opened backend
(so the file was locked on OS level).
2021-04-29 11:51:25 +02:00
Piotr Tabor
911204cd76
Fix ETCDCTL_API=2 etcdctl backup --with-v3 consistent index consistency
...
Prior to this CL, `ETCDCTL_API=2 etcdctl backup --with-v3` was readacting WAL log
(by removal of some entries), but was NOT updating consistent_index in the backend.
Also the WAL editing logic was buggy, as it didn't took in consideration the fact
that when TERM changes, there can be entries with duplicated indexes in
the log. So its NOT sufficient to subtract number of removed entries to
get accurate log indexes.
The PR replaces removing and shifting of WAL entries with replacing them with an no-op entries.
Thanks to this consistent-index references are staying up to date.
The PR also:
- updates 'verification' logic to check whether consistent_index does not lag befor last snapshot
- env-gated execution of verification framework in `etcdctl backup`.
Tested with:
```
(./build.sh && cd tests && EXPECT_DEBUG=TRUE 'env' 'go' 'test' '-timeout=300m' 'go.etcd.io/etcd/tests/v3/e2e' -run=TestCtlV2Backup --count=1000 2>&1 | tee TestCtlV2BackupV3.log)
```
2021-04-29 11:51:24 +02:00
Piotr Tabor
c4b13a5c83
Integrate verification framework
...
Verification framework is integrated with:
- integration tests (by default)
- `ETCD_VERIFY=all etcdctl snapshot restore` command
- etcd shutdown when running with `ETCD_VERIFY=all` env.
2021-04-28 07:56:16 +02:00
Piotr Tabor
47b28b600a
Verification package: Verified given data-dir.
...
For now verifies whete Backend.cindex is consistent with WAL log,
but should get expanded to cover memberships & revisions.
2021-04-28 07:56:15 +02:00
Piotr Tabor
6f8f506cf4
Create 'datadir' package responsible for paths.
2021-04-28 07:56:13 +02:00
Piotr Tabor
067521981e
v2 etcdctl backup: producing consistent state of membership
2021-04-27 19:34:34 +02:00
Piotr Tabor
a70386a1a4
Simplify membership interface: Does not pass the 'unused' token.
2021-04-27 17:17:31 +02:00
Piotr Tabor
7ae3d25f91
Membership: Add additional methods to trim/manage membership data in backend.
2021-04-27 17:17:31 +02:00
Piotr Tabor
768da490ed
sever: v2store deprecation: Fix etcdctl snapshot restore to restore
...
correct 'backend' (bbolt) context in aspect of membership.
Prior to this change the 'restored' backend used to still contain:
- old memberid (mvcc deletion used, why the membership is in bolt
bucket, but not mvcc part):
```
mvs := mvcc.NewStore(s.lg, be, lessor, ci, mvcc.StoreConfig{CompactionBatchLimit: math.MaxInt32})
defer mvs.Close()
txn := mvs.Write(traceutil.TODO())
btx := be.BatchTx()
del := func(k, v []byte) error {
txn.DeleteRange(k, nil)
return nil
}
// delete stored members from old cluster since using new members
btx.UnsafeForEach([]byte("members"), del)
```
- didn't get new members added.
2021-04-27 17:17:30 +02:00
Makdon
ddcb463822
etcdserver/mvcc: update trace.Step condition
2021-04-25 23:07:45 +08:00
Piotr Tabor
9a4b2bdccc
Errors: context cancelled or context deadline exceeded are exposed as codes.Canceled, codes.DeadlineExceeded instead of 'codes.Unknown'
2021-04-22 14:35:24 +02:00
Piotr Tabor
d7d110b5a8
mvcc/backend tests: Refactor: Do not mix testing&prod code.
2021-04-21 09:43:13 +02:00
Piotr Tabor
ea287dd9f8
Merge pull request #12854 from ptabor/20210410-shouldApplyV3
...
(no)StoreV2 (Part 3): Applying consistency fix: ClusterVersionSet (and co) might get not applied on v2store
2021-04-21 09:31:38 +02:00
Sam Batschelet
0f2c940f64
Merge pull request #12880 from chaochn47/exclude_alarms_from_health_check
...
etcdhttp/metrics.go: exclude alarms from health check conditionally with `?exclude=NOSPACE`
2021-04-20 21:18:15 -04:00
Chao Chen
140ea4fa29
etcdhttp/metrics.go: exclude alarms from health check conditionally with ?exclude=NOSPACE
2021-04-20 13:17:09 -07:00
Piotr Tabor
11249fdee9
Merge pull request #12874 from ptabor/20210417-go1.16.3
...
Update go for 3.5: 1.15.x -> 1.16.3
2021-04-19 16:51:49 +02:00
Piotr Tabor
3423a949c0
Update go for 3.5: 1.15 -> 1.16.(3).
...
https://github.com/etcd-io/etcd/issues/12732
2021-04-19 16:50:54 +02:00
Piotr Tabor
06ba0fc5a2
Merge pull request #12846 from pyiyun/fix-snaptmpfile-bug
...
etcdserver: remove temp files in snap dir when etcdserver starting
2021-04-17 12:58:46 +02:00
chao
80586c5b47
etcdserver/util.go: reduce memory when logging range requests ( #12871 )
...
* etcdserver/util.go: reduce memory when logging range requests
Fixes #12835
* Update CHANGELOG-3.5.md
2021-04-16 16:39:34 -07:00
Piotr Tabor
17b982382e
Fix TestSnapshotV3RestoreMultiMemberAdd flakes (leaks)
...
- most important: unix's socket transport should not keep idle
connections. For top-level Transport we close them using:
f3c518025e/server/etcdserver/api/rafthttp/transport.go (L226)
2021-04-16 20:17:28 +02:00
pyiyun
28a490b09c
etcdserver: remove temp files in snap dir when etcdServer starting
...
When etcd exits abnormally, tmp files will remain in snap dir, so clean up tmp files in snap dir when etcdserver starting.
Fixes #12837
2021-04-16 20:30:04 +08:00
Piotr Tabor
b47c5fcc12
Address review comments a.d. logging.
2021-04-15 17:54:37 +02:00
Piotr Tabor
d72f7ef5cc
Give control to Embedded servers whether they override global loggers
...
So far each instance of embed server was overriding the grpc loggers and zap.global loggers.
It's counter intutitive that last created Embedded server was 'wining' and more-over it was breaking grpc expectation to change it "only" before the grpc stack is being used.
This PR introduces explicit call: `embed.Config::SetupGlobalLoggers()`, that changes the loggers where requested. The call is used by etcd main binary.
The immediate benefit from this change is reduction of test flakiness, as there were flakes due to not a proper logger being used across tests.
2021-04-14 12:47:38 +02:00
Piotr Tabor
eafbc8c57e
Update zap logging dependency.
...
In particular bring up zapgrpc V2 code:
89e382035dhttps://pkg.go.dev/google.golang.org/grpc/grpclog#LoggerV2
2021-04-14 12:15:48 +02:00
Piotr Tabor
d69e46ea47
Make ShouldApplyV3 an enum - not bool
2021-04-13 23:01:03 +02:00
Piotr Tabor
b1c04ce043
Applying consistency fix: ClusterVersionSet (and co) might get no applied on v2store
...
ClusterVersionSet, ClusterMemberAttrSet, DowngradeInfoSet functions are
writing both to V2store and backend. Prior this CL there were
in a branch not executed if shouldApplyV3 was false,
e.g. during restore when Backend is up-to-date (has high
consistency-index) while v2store requires replay from WAL log.
The most serious consequence of this bug was that v2store after restore
could have different index (revision) than the same exact store before restore,
so potentially different content between replicas.
Also this change is supressing double-applying of Membership
(ClusterConfig) changes on Backend (store v3) - that lackilly are not
part of MVCC/KeyValue store, so they didn't caused Revisions to be
bumped.
Inspired by jingyih@ comment:
https://github.com/etcd-io/etcd/pull/12820#issuecomment-815299406
2021-04-12 09:43:48 +02:00
wpedrak
3991a8c9fa
etcdserver: replace forceVersionC with FirstCommitInTermNotify
2021-04-09 11:30:42 +02:00
wpedrak
3d485faac5
etcdserver: resend ReadIndex request on empty apply request
...
Empty apply indicates first commit in current term. It is first time when follower is sure, that it's ReadIndex request can be processed.
2021-04-09 11:30:42 +02:00
Piotr Tabor
63c25bf378
Merge pull request #12804 from ptabor/20210326-v3-publish
...
server: v2store deprecation: Prepare to use publishV3 instead of publish V2.
2021-04-08 09:22:22 +02:00
Piotr Tabor
e776efbb2a
Merge pull request #12828 from ptabor/20210404-embed-etcd
...
embed: etcd.Close() is closing Errc() channel as well.
2021-04-08 01:20:07 +02:00
Piotr Tabor
5da9cac193
embed: etcd.Close() is closing Errc() channel as well.
...
Inspired by https://github.com/etcd-io/etcd/pull/9612 by purpleidea@.
2021-04-08 01:19:13 +02:00
Piotr Tabor
cc0f812f51
server_test.go: Use context.Background() instead of TODO in tests.
...
Suggected in: https://golang.org/pkg/context/#Background
2021-04-08 01:15:16 +02:00
Piotr Tabor
931af493cf
Merge pull request #12830 from ptabor/20210405-split-pkg
...
Split client/pkg as dedicated low-dependencies module for client
2021-04-08 01:12:17 +02:00
Piotr Tabor
3bb7acc8cf
Migrate dependencies pkg/foo -> client/pkg/foo
2021-04-07 00:38:47 +02:00
Sam Batschelet
a40a6e9ad8
Merge pull request #12805 from ptabor/20210326-minor-test-fixes
...
tests: logging & temp-dir fixes
2021-03-28 19:14:13 -04:00
Piotr Tabor
f290ab2e60
Update dependecies:
...
github.com/grpc-ecosystem/grpc-gateway v1.14.6 -> grpc-gateway v1.16.0
golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e->v0.0.0-20210220033141-f8bda1e9f3ba
2021-03-27 20:48:33 +01:00
Piotr Tabor
c49807f59e
Update cmux to 1.5.0.
...
Executed using:
```
./scripts/update_dep.sh github.com/soheilhy/cmux v0.1.5
```
2021-03-27 11:18:13 +01:00
Piotr Tabor
55ccbe62a2
membership/cluster_test: Use zaptest logger.
2021-03-26 13:54:59 +01:00
Piotr Tabor
03f55eeb2c
Make NewTmpBackend use testing tmp location (so cleanup).
2021-03-26 13:54:55 +01:00
Piotr Tabor
4d4c84e014
server: Written Snapshot's to WAL contains populated ConfState.
...
This will (among others) allow etcd-3.6 to not depend on store_v2 .snap files at all,
as WAL + db file will be self-sufficient.
2021-03-25 00:31:44 +01:00
Piotr Tabor
fce0c192eb
Regenerate protos.
2021-03-25 00:31:44 +01:00
Piotr Tabor
9180e8e56e
wal: Update WAL Snapshot format to contain ConfState.
2021-03-25 00:31:44 +01:00
Piotr Tabor
a60676686b
Update dep: grpc: 1.32.0 -> 1.36.0.
2021-03-24 22:27:55 +01:00
Piotr Tabor
45fb7b41d6
Update dep: github.com/golang/protobuf v1.3.5 -> v1.5.1
...
Thanks to https://go-review.googlesource.com/c/protobuf/+/300869/ its
feasible now.
2021-03-24 22:27:12 +01:00
Piotr Tabor
7d7a9c6f23
codec.go: should use google runtime
...
golang-proto runtime can deal with both: gogo & golang generated
protobufs. It does not work vice-versa with protobuf-1.5.1.
2021-03-24 22:06:47 +01:00
Joel Smith
19f7c6ef3e
*: Update gogo/protobuf to v1.3.2, rerun ./scripts/genproto.sh
...
While it appears that etcd is not vulnerable to CVE-2021-3121,
it is a good idea to update to the new generator so that new
vulnerable code isn't generated in any future APIs. Also, this
lays the issue to rest of whether there is any issue with
etcd and CVE-2021-3121.
2021-03-23 11:48:06 -06:00
Piotr Tabor
30ce6067da
Merge pull request #12780 from wpedrak/read_index_retry
...
Read index retry
2021-03-23 13:47:41 +01:00
Gábor Lipták
49d19efd82
Correct ineffassign
2021-03-22 14:11:30 -04:00
Piotr Tabor
8469108548
Merge pull request #12782 from ptabor/20210316-fixes
...
Integration: Test flakiness fixes
2021-03-19 20:10:11 +01:00
