19073 Commits

Author SHA1 Message Date
Marek Siarkowicz
ca8baeb308
Merge pull request #14685 from serathius/linearizability-revision
Revision inconsistency caused by panic during defrag
2022-11-14 12:49:22 +01:00
Marek Siarkowicz
77cd6a660f
Merge pull request #14742 from fuweid/deflake-issue-14694
tests/e2e: disable testShellCompletion if enable cov flag
2022-11-14 12:41:40 +01:00
Marek Siarkowicz
4cdcb91fac
Merge pull request #14672 from tjungblu/etcd-14638
Expect exit code enhancement
2022-11-14 12:34:43 +01:00
Marek Siarkowicz
ff6c93f630 tests: Add revision to etcd linearizability model
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-11-14 12:20:55 +01:00
Tobias Grieger
e64d644989
Merge pull request #14624 from pavelkalinnikov/limit_inflight_bytes 2022-11-14 12:10:13 +01:00
Benjamin Wang
ae7e7241b3
Merge pull request #14734 from ahrtr/changelog_rev_inconsistency
changelog: add item for 3.5 to cover the revision inconsistency issue
2022-11-14 18:28:51 +08:00
Thomas Jungblut
5b163aa507 Expect exit code enhancement
ExpectProcess and ExpectFunc now take the exit code of the process into
account, not just the matching of the tty output.

This also refactors the many tests that were previously succeeding on
matching an output from a failing cmd execution.

Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
2022-11-14 11:19:41 +01:00
Wei Fu
c2f27a0070 tests/e2e: disable testShellCompletion if enable cov flag
The etcdctl and etcdutl built with `-tags cov` mode will show go-test result
after each execution, like

```
...

PASS
coverage: 0.0% of statements in ./...
```

Since the PASS is not real command, the `source completion` command will
fail with command-not-found error. And there is no easy way to disable
the (*testing.M).Run's output. Therefore, this patch uses build tag !cov
to disable cases when enable coverage.

Fixes: #14694

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2022-11-14 17:58:18 +08:00
Benjamin Wang
3232c04138
Merge pull request #14746 from ahrtr/add_gofailpoint_OnPreCommitUnsafe
etcdserver: Add gofailpoints for pre commit hook
2022-11-14 17:54:04 +08:00
Benjamin Wang
d3c64ed94f
Merge pull request #14743 from fuweid/test-downgrade-with-latest-point-release
e2e: test DowngradeVersion with latest point release
2022-11-14 15:23:50 +08:00
Benjamin Wang
2532ca84d1 test: add failpoints into linearizablity test for commitBeforePreCommitHook and commitAfterPreCommitHook
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-14 11:11:48 +08:00
Benjamin Wang
3f18816e7d etcdserver: add gofail points before and after OnPreCommitUnsafe
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-14 11:02:18 +08:00
Benjamin Wang
3cf2e79e16
Merge pull request #14537 from ahrtr/add_hash_rev
Add a new field hash_revision into HashKVResponse
2022-11-14 09:20:44 +08:00
Benjamin Wang
f77b8a735f etcdserver: populate HashRevision when responding to leader or client's HashKV request
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-14 08:33:44 +08:00
Benjamin Wang
228f493c76 rpc.proto: regenerate rpc.proto related files
Run
1. ./script/genproto.sh
2. ./scripts/update_proto_annotations.sh

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-14 08:33:44 +08:00
Benjamin Wang
cd746de14e rpc.proto: added a new field hash_revision into HashKVResponse
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-14 08:33:44 +08:00
Tobias Grieger
4c8244622e
Merge pull request #14717 from falser101/fix/testify 2022-11-13 23:08:30 +01:00
Pavel Kalinnikov
0ef5df11a6 raft: update changelog
Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
2022-11-13 23:05:16 +01:00
Pavel Kalinnikov
68af01ca6e raft: add MaxInflightBytes to Config
This commit introduces the max inflight bytes setting at the Config level, and
tests that raft flow control honours it.

Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
2022-11-13 23:05:16 +01:00
Pavel Kalinnikov
8c9c557d85 raft: factor out payloadsSize helper
Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
2022-11-13 23:05:16 +01:00
Pavel Kalinnikov
7bda0d7773 raft/tracker: add MaxInflightBytes to ProgressTracker
This commit plumbs the max total byte size of the Inflights type higher up the
stack to the ProgressTracker.

Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
2022-11-13 23:05:16 +01:00
Pavel Kalinnikov
bfb7b16f4f raft/tracker: add byte size limit to Inflights type
The Inflights type has limits on the message size and the number of inflight
messages. However, a single large entry that exceeds the size limit can still
be sent. In combination with the max messages count limit, many large messages
can be sent in a row and overflow the receiver. In effect, the "max" values act
as "target" rather than hard limits.

This commit adds an additional soft limit on the total size of inflight
messages, which catches such situations and prevents the receiver overflow.

Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
2022-11-13 23:05:16 +01:00
Tobias Grieger
de97f6aa3d raft: tidy up the unit tests some more
Use `t.Run` for each test case, and make some tests more idiomatic.

Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
2022-11-13 22:34:47 +01:00
Clark
3510680c32 tests: refactor EtcdProcessClusterConfig using Functional Options Pattern
replace some initialisations with `DefaultConfig`, `NewConfig` and `EPClusterOption`

Signed-off-by: Clark <fwyongxing@gmail.com>
2022-11-14 03:07:51 +08:00
jianfei.zhang
1f4f70723f feat: use testify packages in tests
Signed-off-by: jianfei.zhang <jianfei.zhang@daocloud.io>
2022-11-13 23:38:57 +08:00
Marek Siarkowicz
d5a6d2518d tests: Optimize checking failed writes
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-11-13 13:36:21 +01:00
Marek Siarkowicz
0e4bf4ac4e
Merge pull request #14730 from ahrtr/rev_inconsistency_20221111
etcdserver: call the OnPreCommitUnsafe in unsafeCommit
2022-11-13 13:34:39 +01:00
Marek Siarkowicz
2a6e57d4fc
Merge pull request #14741 from ahrtr/update_readme_20221112
README: remove Tobias Grieger from the Emeritus Maintainers list
2022-11-13 12:21:52 +01:00
Wei Fu
71d45461ff e2e: test DowngradeVersion with latest point release
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2022-11-13 17:21:13 +08:00
Benjamin Wang
e8097fe030 README: remove Tobias Grieger from the Emeritus Maintainers list
Tobias Grieger is actively working on etcd/raft, and has already
been added back as a raft maintainer, so no reason to keep him
in the Emeritus Maintainers list.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-13 13:15:49 +08:00
Benjamin Wang
62cba5b4f1
Merge pull request #14725 from clarkfw/functional-options-pattern-EtcdProcessClusterConfig-2022-11-11
tests: refactor `EtcdProcessClusterConfig` using Functional Options Pattern
2022-11-13 07:00:10 +08:00
Clark
caaeee2920 tests: refactor EtcdProcessClusterConfig with Functional Options Pattern
replace some initialisations with `DefaultConfig`, `NewConfig` and `EPClusterOption`

Signed-off-by: Clark <fwyongxing@gmail.com>
2022-11-13 00:27:41 +08:00
Nathan VanBenschoten
539a8410f4 raft: don't apply entries when applying snapshot
This commit removes the ability to apply log entries at the same time as
applying a snapshot. Doing so it possible, but it leads to complex code and
raises questions about what should be applied first. It also raises additional
complexity when we start allowing concurrent, asynchronous log appends and log
application. It's easiest to just disallow this.

Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
2022-11-11 13:58:09 -05:00
Nathan VanBenschoten
95c5fed3cf raft: remove IsEmptySnap check from raftLog.hasPendingSnapshot
unstable.snapshot is never an empty snapshot. This check made it look
like it could be.

Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
2022-11-11 13:57:40 -05:00
Benjamin Wang
bdd5347313
Merge pull request #14719 from nvanbenschoten/nvanbenschoten/nextCommittedEnts
raft: rename raftLog.nextEnts to raftLog.nextCommittedEnts
2022-11-12 02:51:30 +08:00
Nathan VanBenschoten
3711fde822 raft: rename raftLog.nextEnts to raftLog.nextCommittedEnts
Also rename hasNextEnts to hasNextCommittedEnts.
Also rename maxNextEntsSize to maxNextCommittedEntsSize.

Pure refactor.

Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
2022-11-11 13:21:20 -05:00
Tobias Grieger
22d930b3d5
Merge pull request #14722 from nvanbenschoten/nvanbenschoten/unusedReadyContainsUpdates 2022-11-11 16:25:46 +01:00
Tobias Grieger
910b81b153
Merge pull request #14720 from nvanbenschoten/nvanbenschoten/clarifyUnstableStableTo 2022-11-11 16:25:37 +01:00
Benjamin Wang
da619c3567
Merge pull request #14718 from ahrtr/fix_e2e_datadir_20221110
test: Create a subdirectory for each member when DataDirPath is specified
2022-11-11 19:15:06 +08:00
Benjamin Wang
dbc7204610 test: Create a subdirectory for each member when DataDirPath is specified
When e2e test cases specify the DataDirPath and there are more than
one member in the cluster, we need to create a subdirectory for each
member. Otherwise all members share the same directory and accordingly
lead to conflict.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-11 18:48:42 +08:00
Benjamin Wang
3d5f293f3e changelog: add item for 3.5 to cover the revision inconsistency issue
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-11 17:40:20 +08:00
Ramil Mirhasanov
44d8e732cf client/v3: replace deprecated grpc.WithInsecure()
Deprecated: use WithTransportCredentials and insecure.NewCredentials() instead. Will be supported throughout 1.x.

Signed-off-by: Ramil Mirhasanov <ramil600@yahoo.com>
2022-11-11 12:08:50 +03:00
Benjamin Wang
5a3ef953eb etcdserver: call the OnPreCommitUnsafe in unsafeCommit
`unsafeCommit` is called by both `(*batchTxBuffered) commit` and
`(*backend) defrag`. When users perform the defragmentation
operation, etcd doesn't update the consistent index. If etcd
crashes(e.g. panicking) in the process for whatever reason, then
etcd replays the WAL entries starting from the latest snapshot,
accordingly it may re-apply entries which might have already been
applied, eventually the revision isn't consistent with other members.

Refer to discussion in https://github.com/etcd-io/etcd/pull/14685

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-11 10:57:15 +08:00
Marek Siarkowicz
2f558ca0db
Merge pull request #14714 from serathius/linearizability-data-dir
tests: Preserve data dir from failed linearizability tests
2022-11-10 13:58:23 +01:00
Marek Siarkowicz
04caec5960 tests: Preserve data dir from failed linearizability tests
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-11-10 11:05:59 +01:00
Nathan VanBenschoten
0ea6fa542a raft: clean up IsLocalMsg and IsResponseMsg logic
Use array indexing to clean up the code and make it constant time.

Also, add a test for IsResponseMsg.

Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
2022-11-10 03:31:31 +00:00
Nathan VanBenschoten
e0beef6830 raft: delete unused Ready.containsUpdates method
Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
2022-11-09 22:25:34 -05:00
Nathan VanBenschoten
c18d79df37 raft: clarify conditions in unstable.stableTo
No change in behavior, but clarify interaction with unstable snapshot.

Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
2022-11-09 22:12:49 -05:00
Benjamin Wang
0bff3ade4d
Merge pull request #14704 from chenyahui/mutex_unlock_state
etcdclient: check mutex state in Unlock method of concurrency.Mutex
2022-11-10 10:38:03 +08:00
Benjamin Wang
ccec27be62
Merge pull request #14592 from nvanbenschoten/nvanbenschoten/nilSnapMsg
raft: make Message.Snapshot nullable, halve struct size
2022-11-10 05:47:21 +08:00