18769 Commits

Author SHA1 Message Date
Marek Siarkowicz
2fc1485f29 tests: Make linearizability traffic extendable
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-11-14 23:14:10 +01:00
Marek Siarkowicz
ca8baeb308
Merge pull request #14685 from serathius/linearizability-revision
Revision inconsistency caused by panic during defrag
2022-11-14 12:49:22 +01:00
Marek Siarkowicz
77cd6a660f
Merge pull request #14742 from fuweid/deflake-issue-14694
tests/e2e: disable testShellCompletion if enable cov flag
2022-11-14 12:41:40 +01:00
Marek Siarkowicz
4cdcb91fac
Merge pull request #14672 from tjungblu/etcd-14638
Expect exit code enhancement
2022-11-14 12:34:43 +01:00
Marek Siarkowicz
ff6c93f630 tests: Add revision to etcd linearizability model
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-11-14 12:20:55 +01:00
Tobias Grieger
e64d644989
Merge pull request #14624 from pavelkalinnikov/limit_inflight_bytes 2022-11-14 12:10:13 +01:00
Benjamin Wang
ae7e7241b3
Merge pull request #14734 from ahrtr/changelog_rev_inconsistency
changelog: add item for 3.5 to cover the revision inconsistency issue
2022-11-14 18:28:51 +08:00
Thomas Jungblut
5b163aa507 Expect exit code enhancement
ExpectProcess and ExpectFunc now take the exit code of the process into
account, not just the matching of the tty output.

This also refactors the many tests that were previously succeeding on
matching an output from a failing cmd execution.

Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
2022-11-14 11:19:41 +01:00
Wei Fu
c2f27a0070 tests/e2e: disable testShellCompletion if enable cov flag
The etcdctl and etcdutl built with `-tags cov` mode will show go-test result
after each execution, like

```
...

PASS
coverage: 0.0% of statements in ./...
```

Since the PASS is not real command, the `source completion` command will
fail with command-not-found error. And there is no easy way to disable
the (*testing.M).Run's output. Therefore, this patch uses build tag !cov
to disable cases when enable coverage.

Fixes: #14694

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2022-11-14 17:58:18 +08:00
Benjamin Wang
3232c04138
Merge pull request #14746 from ahrtr/add_gofailpoint_OnPreCommitUnsafe
etcdserver: Add gofailpoints for pre commit hook
2022-11-14 17:54:04 +08:00
Benjamin Wang
d3c64ed94f
Merge pull request #14743 from fuweid/test-downgrade-with-latest-point-release
e2e: test DowngradeVersion with latest point release
2022-11-14 15:23:50 +08:00
Benjamin Wang
2532ca84d1 test: add failpoints into linearizablity test for commitBeforePreCommitHook and commitAfterPreCommitHook
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-14 11:11:48 +08:00
Benjamin Wang
3f18816e7d etcdserver: add gofail points before and after OnPreCommitUnsafe
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-14 11:02:18 +08:00
Benjamin Wang
3cf2e79e16
Merge pull request #14537 from ahrtr/add_hash_rev
Add a new field hash_revision into HashKVResponse
2022-11-14 09:20:44 +08:00
Benjamin Wang
f77b8a735f etcdserver: populate HashRevision when responding to leader or client's HashKV request
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-14 08:33:44 +08:00
Benjamin Wang
228f493c76 rpc.proto: regenerate rpc.proto related files
Run
1. ./script/genproto.sh
2. ./scripts/update_proto_annotations.sh

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-14 08:33:44 +08:00
Benjamin Wang
cd746de14e rpc.proto: added a new field hash_revision into HashKVResponse
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-14 08:33:44 +08:00
Tobias Grieger
4c8244622e
Merge pull request #14717 from falser101/fix/testify 2022-11-13 23:08:30 +01:00
Pavel Kalinnikov
0ef5df11a6 raft: update changelog
Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
2022-11-13 23:05:16 +01:00
Pavel Kalinnikov
68af01ca6e raft: add MaxInflightBytes to Config
This commit introduces the max inflight bytes setting at the Config level, and
tests that raft flow control honours it.

Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
2022-11-13 23:05:16 +01:00
Pavel Kalinnikov
8c9c557d85 raft: factor out payloadsSize helper
Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
2022-11-13 23:05:16 +01:00
Pavel Kalinnikov
7bda0d7773 raft/tracker: add MaxInflightBytes to ProgressTracker
This commit plumbs the max total byte size of the Inflights type higher up the
stack to the ProgressTracker.

Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
2022-11-13 23:05:16 +01:00
Pavel Kalinnikov
bfb7b16f4f raft/tracker: add byte size limit to Inflights type
The Inflights type has limits on the message size and the number of inflight
messages. However, a single large entry that exceeds the size limit can still
be sent. In combination with the max messages count limit, many large messages
can be sent in a row and overflow the receiver. In effect, the "max" values act
as "target" rather than hard limits.

This commit adds an additional soft limit on the total size of inflight
messages, which catches such situations and prevents the receiver overflow.

Signed-off-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
2022-11-13 23:05:16 +01:00
Tobias Grieger
de97f6aa3d raft: tidy up the unit tests some more
Use `t.Run` for each test case, and make some tests more idiomatic.

Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
2022-11-13 22:34:47 +01:00
jianfei.zhang
1f4f70723f feat: use testify packages in tests
Signed-off-by: jianfei.zhang <jianfei.zhang@daocloud.io>
2022-11-13 23:38:57 +08:00
Marek Siarkowicz
d5a6d2518d tests: Optimize checking failed writes
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-11-13 13:36:21 +01:00
Marek Siarkowicz
0e4bf4ac4e
Merge pull request #14730 from ahrtr/rev_inconsistency_20221111
etcdserver: call the OnPreCommitUnsafe in unsafeCommit
2022-11-13 13:34:39 +01:00
Marek Siarkowicz
2a6e57d4fc
Merge pull request #14741 from ahrtr/update_readme_20221112
README: remove Tobias Grieger from the Emeritus Maintainers list
2022-11-13 12:21:52 +01:00
Wei Fu
71d45461ff e2e: test DowngradeVersion with latest point release
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2022-11-13 17:21:13 +08:00
Benjamin Wang
e8097fe030 README: remove Tobias Grieger from the Emeritus Maintainers list
Tobias Grieger is actively working on etcd/raft, and has already
been added back as a raft maintainer, so no reason to keep him
in the Emeritus Maintainers list.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-13 13:15:49 +08:00
Benjamin Wang
62cba5b4f1
Merge pull request #14725 from clarkfw/functional-options-pattern-EtcdProcessClusterConfig-2022-11-11
tests: refactor `EtcdProcessClusterConfig` using Functional Options Pattern
2022-11-13 07:00:10 +08:00
Clark
caaeee2920 tests: refactor EtcdProcessClusterConfig with Functional Options Pattern
replace some initialisations with `DefaultConfig`, `NewConfig` and `EPClusterOption`

Signed-off-by: Clark <fwyongxing@gmail.com>
2022-11-13 00:27:41 +08:00
Benjamin Wang
bdd5347313
Merge pull request #14719 from nvanbenschoten/nvanbenschoten/nextCommittedEnts
raft: rename raftLog.nextEnts to raftLog.nextCommittedEnts
2022-11-12 02:51:30 +08:00
Nathan VanBenschoten
3711fde822 raft: rename raftLog.nextEnts to raftLog.nextCommittedEnts
Also rename hasNextEnts to hasNextCommittedEnts.
Also rename maxNextEntsSize to maxNextCommittedEntsSize.

Pure refactor.

Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
2022-11-11 13:21:20 -05:00
Tobias Grieger
22d930b3d5
Merge pull request #14722 from nvanbenschoten/nvanbenschoten/unusedReadyContainsUpdates 2022-11-11 16:25:46 +01:00
Tobias Grieger
910b81b153
Merge pull request #14720 from nvanbenschoten/nvanbenschoten/clarifyUnstableStableTo 2022-11-11 16:25:37 +01:00
Benjamin Wang
da619c3567
Merge pull request #14718 from ahrtr/fix_e2e_datadir_20221110
test: Create a subdirectory for each member when DataDirPath is specified
2022-11-11 19:15:06 +08:00
Benjamin Wang
dbc7204610 test: Create a subdirectory for each member when DataDirPath is specified
When e2e test cases specify the DataDirPath and there are more than
one member in the cluster, we need to create a subdirectory for each
member. Otherwise all members share the same directory and accordingly
lead to conflict.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-11 18:48:42 +08:00
Benjamin Wang
3d5f293f3e changelog: add item for 3.5 to cover the revision inconsistency issue
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-11 17:40:20 +08:00
Benjamin Wang
5a3ef953eb etcdserver: call the OnPreCommitUnsafe in unsafeCommit
`unsafeCommit` is called by both `(*batchTxBuffered) commit` and
`(*backend) defrag`. When users perform the defragmentation
operation, etcd doesn't update the consistent index. If etcd
crashes(e.g. panicking) in the process for whatever reason, then
etcd replays the WAL entries starting from the latest snapshot,
accordingly it may re-apply entries which might have already been
applied, eventually the revision isn't consistent with other members.

Refer to discussion in https://github.com/etcd-io/etcd/pull/14685

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-11 10:57:15 +08:00
Marek Siarkowicz
2f558ca0db
Merge pull request #14714 from serathius/linearizability-data-dir
tests: Preserve data dir from failed linearizability tests
2022-11-10 13:58:23 +01:00
Marek Siarkowicz
04caec5960 tests: Preserve data dir from failed linearizability tests
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-11-10 11:05:59 +01:00
Nathan VanBenschoten
e0beef6830 raft: delete unused Ready.containsUpdates method
Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
2022-11-09 22:25:34 -05:00
Nathan VanBenschoten
c18d79df37 raft: clarify conditions in unstable.stableTo
No change in behavior, but clarify interaction with unstable snapshot.

Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
2022-11-09 22:12:49 -05:00
Benjamin Wang
0bff3ade4d
Merge pull request #14704 from chenyahui/mutex_unlock_state
etcdclient: check mutex state in Unlock method of concurrency.Mutex
2022-11-10 10:38:03 +08:00
Benjamin Wang
ccec27be62
Merge pull request #14592 from nvanbenschoten/nvanbenschoten/nilSnapMsg
raft: make Message.Snapshot nullable, halve struct size
2022-11-10 05:47:21 +08:00
Nathan VanBenschoten
0f9d7a4f95 raft: make Message.Snapshot nullable, halve struct size
This commit makes the rarely used `raftpb.Message.Snapshot` field nullable.
In doing so, it reduces the memory size of a `raftpb.Message` message from
264 bytes to 128 bytes — a 52% reduction in size.

While this commit does not change the protobuf encoding, it does change
how that encoding is used. `(gogoproto.nullable) = false` instruct the
generated proto marshaling logic to always encode a value for the field,
even if that value is empty. `(gogoproto.nullable) = true` instructs the
generated proto marshaling logic to omit an encoded value for the field
if the field is nil.

This raises compatibility concerns in both directions. Messages encoded
by new binary versions without a `Snapshot` field will be decoded as an
empty field by old binary versions. In other words, old binary versions
can't tell the difference. However, messages encoded by old binary versions
with an empty Snapshot field will be decoded as a non-nil, empty field by
new binary versions. As a result, new binary versions need to be prepared
to handle such messages.

While Message.Snapshot is not intentionally part of the external interface
of this library, it was possible for users of the library to access it and
manipulate it. As such, this change may be considered a breaking change.

Signed-off-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
2022-11-09 17:35:52 +00:00
Tobias Grieger
00820f0fc9
Merge pull request #14708 from pavelkalinnikov/testify_demo 2022-11-09 15:26:26 +01:00
Marek Siarkowicz
0d4a516d11
Merge pull request #14710 from tjungblu/downgrade_binary
Take binary version in Downgrade/Upgrade Test
2022-11-09 13:44:15 +01:00
Thomas Jungblut
0daf153b4f Take binary version in Downgrade/Upgrade Test
This fix avoids the assumption of knowing the current version of the
binary. We can query the binary with the version flag to get the actual
version of the given binary we upgrade and downgrade to. The
respectively reported versions should match what is returned by the
version endpoint.

Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
2022-11-09 10:41:07 +01:00