varvara dvortsova
c3303d94a4
embed: Added a check for an nil value of the request in ServeHTTP method. Fixed #12821
2021-05-09 20:45:43 +03:00
Piotr Tabor
aeb9b5fc73
Merge pull request #12855 from ptabor/20210409-backend-hooks
...
(no)StoreV2 (Part 4): Backend hooks: precommit updates consistency_index
2021-05-08 09:34:31 +02:00
Sam Batschelet
6decbe15db
Merge pull request #12774 from hexfusion/add-log-rotate
...
server: add support for log rotation
2021-05-07 12:18:10 -04:00
Sam Batschelet
82b2d5c67d
server: add support for log rotation
...
Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
2021-05-07 08:39:51 -04:00
Gyuho Lee
344c9f3930
Merge pull request #12896 from wilsonwang371/profiling-txn2
...
server: make applier use ReadTx() in Txn() instead of ConcurrentReadTx()
2021-05-06 01:59:14 -07:00
tangcong
b32bc914ff
learner support snapshot RPC ( #12890 )
...
* learner support snapshot RPC
* CHANGELOG: update for 12890
2021-05-04 13:26:33 -07:00
Wilson Wang
98083ea914
server: add experimental flag for using shared buffer in transacton write
2021-05-04 11:59:08 -07:00
Piotr Tabor
a78d072b9a
Simplify KVstore dependency on cindex.
2021-05-04 18:21:24 +02:00
Piotr Tabor
2fb6f0a74b
Simplify lease management after cindex update is moved to 'hooks'.
2021-05-04 18:21:23 +02:00
Piotr Tabor
2dbecea5b2
Simplify KVStore interaction with cindex thanks to hooks.
2021-05-04 18:21:23 +02:00
Piotr Tabor
fe3254aee3
Remove explicit authStore->ConsistencyIndex updates, as they are taken care by hook.
2021-05-04 15:38:23 +02:00
Piotr Tabor
50051675f9
Integrate backend::hooks with consistent_index.
...
Every transaction committed to backend is writing most recent consistent_index.
Makes sure that even automatically trigger commits of batch-transactions
stays "really" consistent a.d. the most recent WAL log index applied.
2021-05-04 15:38:23 +02:00
Piotr Tabor
d53d2db1e2
Tests: Backend hooks support.
2021-05-04 15:38:22 +02:00
Piotr Tabor
9f11b16b2d
backend: Hooks interface & implementation.
2021-05-04 15:38:22 +02:00
Piotr Tabor
a3fc535a5a
Merge pull request #12914 from ptabor/20210430-read-membership
...
No-storeV2: Read membership information from the backend (Part5)
2021-05-04 10:15:01 +02:00
Wilson Wang
56154216b7
update variable declaration location
2021-05-03 10:30:15 -07:00
wpedrak
6623c008ee
server: reapply Mlock flag after defrag
2021-05-03 11:01:02 +02:00
Piotr Tabor
451f65d661
Merge pull request #12908 from ptabor/20210429-client-retry-logging
...
Clientv3 (retry interceptor) logs should use the configured logger
2021-04-29 19:25:04 +02:00
Piotr Tabor
cedbea6c81
Merge pull request #12904 from wpedrak/limit_mlocked_memory
...
server: replace mlockall with `Mlock` in `--experimental-memory-mlock`
2021-04-29 18:21:24 +02:00
Piotr Tabor
ffea1537d4
ClientV3 tests use integration.NewClient that configures proper logger.
2021-04-29 18:18:34 +02:00
Piotr Tabor
205a1a442a
Read membership information from the backend.
2021-04-29 13:45:45 +02:00
Piotr Tabor
bc8d3f6639
Clientv3 (retry) logs should use the configured logger.
...
clientv3 logs (especially tests) were poluted with unattributed to testing.T log lines:
```
{"level":"warn","ts":"2021-04-29T12:42:11.055+0200","logger":"etcd-client","caller":"v3/retry_interceptor.go:64","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000fafc0/#initially=[unix://localhost:m10]","attempt":0,"error":"rpc error: code = ResourceExhausted desc = etcdserver: mvcc: database space exceeded"}
```
The reasons were 2 fold:
- Interceptors were copying logger before "WithLogger" could modify it.
- We were not propagating the loggers in a few testing contexts.
2021-04-29 12:57:09 +02:00
wpedrak
1145c57601
server: replace mlockall with Mlock in --experimental-memory-mlock
...
Implementation of `--experimental-memory-mlock` backed by `mlockall` syscall is replaced by `Mlock` flag (backed by mlock syscall) of bboltDB.
2021-04-29 12:08:52 +02:00
wpedrak
927b3a3152
server: replace mlockall with Mlock in --experimental-memory-mlock
...
Implementation of `--experimental-memory-mlock` backed by `mlockall` syscall is replaced by `Mlock` flag (backed by mlock syscall) of bboltDB.
2021-04-29 12:08:20 +02:00
Piotr Tabor
e90504fe62
Unify shared code (and constants) with cindex package.
2021-04-29 11:51:25 +02:00
Piotr Tabor
f53b70facb
Embed: In case KVStoreHash verification fails, close the backend.
...
In case of failed verification, the server used to keep opened backend
(so the file was locked on OS level).
2021-04-29 11:51:25 +02:00
Piotr Tabor
911204cd76
Fix ETCDCTL_API=2 etcdctl backup --with-v3 consistent index consistency
...
Prior to this CL, `ETCDCTL_API=2 etcdctl backup --with-v3` was readacting WAL log
(by removal of some entries), but was NOT updating consistent_index in the backend.
Also the WAL editing logic was buggy, as it didn't took in consideration the fact
that when TERM changes, there can be entries with duplicated indexes in
the log. So its NOT sufficient to subtract number of removed entries to
get accurate log indexes.
The PR replaces removing and shifting of WAL entries with replacing them with an no-op entries.
Thanks to this consistent-index references are staying up to date.
The PR also:
- updates 'verification' logic to check whether consistent_index does not lag befor last snapshot
- env-gated execution of verification framework in `etcdctl backup`.
Tested with:
```
(./build.sh && cd tests && EXPECT_DEBUG=TRUE 'env' 'go' 'test' '-timeout=300m' 'go.etcd.io/etcd/tests/v3/e2e' -run=TestCtlV2Backup --count=1000 2>&1 | tee TestCtlV2BackupV3.log)
```
2021-04-29 11:51:24 +02:00
Wilson Wang
8d8d0377a2
server: applier uses ReadTx instead of ConcurrentTx
2021-04-28 11:06:24 -07:00
Piotr Tabor
c4b13a5c83
Integrate verification framework
...
Verification framework is integrated with:
- integration tests (by default)
- `ETCD_VERIFY=all etcdctl snapshot restore` command
- etcd shutdown when running with `ETCD_VERIFY=all` env.
2021-04-28 07:56:16 +02:00
Piotr Tabor
47b28b600a
Verification package: Verified given data-dir.
...
For now verifies whete Backend.cindex is consistent with WAL log,
but should get expanded to cover memberships & revisions.
2021-04-28 07:56:15 +02:00
Piotr Tabor
6f8f506cf4
Create 'datadir' package responsible for paths.
2021-04-28 07:56:13 +02:00
Piotr Tabor
067521981e
v2 etcdctl backup: producing consistent state of membership
2021-04-27 19:34:34 +02:00
Piotr Tabor
a70386a1a4
Simplify membership interface: Does not pass the 'unused' token.
2021-04-27 17:17:31 +02:00
Piotr Tabor
7ae3d25f91
Membership: Add additional methods to trim/manage membership data in backend.
2021-04-27 17:17:31 +02:00
Piotr Tabor
768da490ed
sever: v2store deprecation: Fix etcdctl snapshot restore to restore
...
correct 'backend' (bbolt) context in aspect of membership.
Prior to this change the 'restored' backend used to still contain:
- old memberid (mvcc deletion used, why the membership is in bolt
bucket, but not mvcc part):
```
mvs := mvcc.NewStore(s.lg, be, lessor, ci, mvcc.StoreConfig{CompactionBatchLimit: math.MaxInt32})
defer mvs.Close()
txn := mvs.Write(traceutil.TODO())
btx := be.BatchTx()
del := func(k, v []byte) error {
txn.DeleteRange(k, nil)
return nil
}
// delete stored members from old cluster since using new members
btx.UnsafeForEach([]byte("members"), del)
```
- didn't get new members added.
2021-04-27 17:17:30 +02:00
Makdon
ddcb463822
etcdserver/mvcc: update trace.Step condition
2021-04-25 23:07:45 +08:00
Piotr Tabor
9a4b2bdccc
Errors: context cancelled or context deadline exceeded are exposed as codes.Canceled, codes.DeadlineExceeded instead of 'codes.Unknown'
2021-04-22 14:35:24 +02:00
Piotr Tabor
d7d110b5a8
mvcc/backend tests: Refactor: Do not mix testing&prod code.
2021-04-21 09:43:13 +02:00
Piotr Tabor
ea287dd9f8
Merge pull request #12854 from ptabor/20210410-shouldApplyV3
...
(no)StoreV2 (Part 3): Applying consistency fix: ClusterVersionSet (and co) might get not applied on v2store
2021-04-21 09:31:38 +02:00
Sam Batschelet
0f2c940f64
Merge pull request #12880 from chaochn47/exclude_alarms_from_health_check
...
etcdhttp/metrics.go: exclude alarms from health check conditionally with `?exclude=NOSPACE`
2021-04-20 21:18:15 -04:00
Chao Chen
140ea4fa29
etcdhttp/metrics.go: exclude alarms from health check conditionally with ?exclude=NOSPACE
2021-04-20 13:17:09 -07:00
Piotr Tabor
11249fdee9
Merge pull request #12874 from ptabor/20210417-go1.16.3
...
Update go for 3.5: 1.15.x -> 1.16.3
2021-04-19 16:51:49 +02:00
Piotr Tabor
3423a949c0
Update go for 3.5: 1.15 -> 1.16.(3).
...
https://github.com/etcd-io/etcd/issues/12732
2021-04-19 16:50:54 +02:00
Piotr Tabor
06ba0fc5a2
Merge pull request #12846 from pyiyun/fix-snaptmpfile-bug
...
etcdserver: remove temp files in snap dir when etcdserver starting
2021-04-17 12:58:46 +02:00
chao
80586c5b47
etcdserver/util.go: reduce memory when logging range requests ( #12871 )
...
* etcdserver/util.go: reduce memory when logging range requests
Fixes #12835
* Update CHANGELOG-3.5.md
2021-04-16 16:39:34 -07:00
Piotr Tabor
17b982382e
Fix TestSnapshotV3RestoreMultiMemberAdd flakes (leaks)
...
- most important: unix's socket transport should not keep idle
connections. For top-level Transport we close them using:
f3c518025e/server/etcdserver/api/rafthttp/transport.go (L226)
2021-04-16 20:17:28 +02:00
pyiyun
28a490b09c
etcdserver: remove temp files in snap dir when etcdServer starting
...
When etcd exits abnormally, tmp files will remain in snap dir, so clean up tmp files in snap dir when etcdserver starting.
Fixes #12837
2021-04-16 20:30:04 +08:00
Piotr Tabor
b47c5fcc12
Address review comments a.d. logging.
2021-04-15 17:54:37 +02:00
Piotr Tabor
d72f7ef5cc
Give control to Embedded servers whether they override global loggers
...
So far each instance of embed server was overriding the grpc loggers and zap.global loggers.
It's counter intutitive that last created Embedded server was 'wining' and more-over it was breaking grpc expectation to change it "only" before the grpc stack is being used.
This PR introduces explicit call: `embed.Config::SetupGlobalLoggers()`, that changes the loggers where requested. The call is used by etcd main binary.
The immediate benefit from this change is reduction of test flakiness, as there were flakes due to not a proper logger being used across tests.
2021-04-14 12:47:38 +02:00
Piotr Tabor
eafbc8c57e
Update zap logging dependency.
...
In particular bring up zapgrpc V2 code:
89e382035dhttps://pkg.go.dev/google.golang.org/grpc/grpclog#LoggerV2
2021-04-14 12:15:48 +02:00
