16832 Commits

Author SHA1 Message Date
qupeng
6828517965 raft: implement fast log rejection
Signed-off-by: qupeng <qupeng@pingcap.com>
2021-02-10 15:48:32 -05:00
Nathan VanBenschoten
b757e1bc87 raft: create new probe_and_replicate.txt interactive test
This commit creates a new probe_and_replicate.txt interactive test. The
test creates a complete Raft log configuration and demonstrates how a
leader probes and replicates to each of its followers. The log
configuration constructed is identical to the one present in Figure 7 of
the raft paper (https://raft.github.io/raft.pdf), which looks like:

```
     1  2  3  4  5  6  7  8  9  10 11 12
n1: [1][1][1][4][4][5][5][6][6][6]
n2: [1][1][1][4][4][5][5][6][6]
n3: [1][1][1][4]
n4: [1][1][1][4][4][5][5][6][6][6][6]
n5: [1][1][1][4][4][5][5][6][7][7][7][7]
n6: [1][1][1][4][4][4][4]
n7: [1][1][1][2][2][2][3][3][3][3][3]
```

Once in this state, we then elect node 1 as the leader and stabilize the
entire raft group. This demonstrates how a newly elected leader probes
for matching indexes, overwrites conflicting entries, and catches up all
followers.

This will be useful to demonstrate the impact of more efficient probing
behavior.
2021-02-10 15:02:36 -05:00
Nate W
2afa5ff45f removing Documentation from etcd-io/etcd
Leaving docs, files, etc that are required for generated documentation

fixes https://github.com/etcd-io/website/issues/97

Signed-off-by: Nate W <4453979+nate-double-u@users.noreply.github.com>
2021-02-10 11:14:40 -08:00
Piotr Tabor
e8ba375032
Merge pull request #11889 from mrkm4ntr/example-recover-from-snap
raftexample: Fix recovery from snapshot
2021-02-10 12:18:30 +01:00
Maksim Buldukyan
7e38cfcc8d raft: makes 'ConnReadTimeout/ConnWriteTimeout' customizable 2021-02-10 10:36:50 +07:00
Piotr Tabor
a836a8045b Get rid of legacy client/v3/naming API.
Update grpcproxy to use the new abstractions.
2021-02-09 11:56:28 +01:00
Gyuho Lee
44c889a8ce
Merge pull request #12677 from chaochn47/reduce_verbosity_of_etcd_server_side_success_health_check_logs
etcdserver/api/etcdhttp: log successful etcd server side health check in debug level
2021-02-08 21:15:25 -08:00
Chao Chen
028d351a46 CHANGELOG-3.5.md: update 2021-02-08 19:46:43 -08:00
Chao Chen
2ae3e82f07 etcdserver/api/etcdhttp: log successful etcd server side health check in debug level
When we have an external component that checks /health periodically, the
etcd server logs can be quite verbose (e.g., DDOS-ing against insure
etcd health check can lead to disk space full due to large log files).

This change was introduced in #11704.

While we keep the warning logs for etcd health check failures, the
success (or OK) log level should be set to DEBUG.

Fixes #12676
2021-02-08 17:15:43 -08:00
Gyuho Lee
96a9860b60
Merge pull request #12674 from ptabor/20210208-skip-386-tls-test
Fix pkg/tlsutil (test) to not fail on 386.
2021-02-08 16:46:47 -08:00
Piotr Tabor
60e44286fa Fix pkg/tlsutil (test) to not fail on 386.
In fact this commit rewrites the functionality to use upstream list of
ciphers instead of checking whether the lists are in sync using ast
analysis.
2021-02-08 23:03:43 +01:00
Piotr Tabor
0b75fede64 Replace client/v3/balancer with standard components: resolver + round_robin LB
This commit significantly reduces volume of custom code
in etcd client v3, while preserving full existing functionality.
2021-02-08 18:50:31 +01:00
Piotr Tabor
3fddea9669
Merge pull request #12669 from lemonlinger/implement-endpoint-manager-and-resolver
Implement endpoint watch and resolver
2021-02-08 14:31:48 +01:00
limeng01
8feb55f65c *: implement Endpoint Watch and new Resolver 2021-02-08 20:05:45 +08:00
Shintaro Murakami
be2167ebab Wait until all committed entries are applied
To take a snapshot
2021-02-05 19:05:41 +09:00
Shintaro Murakami
cb14cdd774 raftexample: Fix recovery from snapshot
* If there is a snapshot, HTTP server won't start.
* Resotring form snapshot occurs after replaying WAL.
* When taking a snapshot, the last change is not applied to the state machine yet.
2021-02-05 09:34:34 +09:00
Piotr Tabor
dae29bb719
Merge pull request #12667 from lemonlinger/implement-endpoint-manager-and-resolver
Implement Update method for EndpointManager
2021-02-04 17:25:51 +01:00
limeng01
571ed502d4 endpoints: implement Update method for EndpointManager.
- Add integration test for endpoints and resolver.
2021-02-04 23:30:22 +08:00
Sahdev Zala
4d9f1a96da
Merge pull request #12664 from spzala/rpctypeserrors
Doc: fix inconsistency and broken path
2021-02-03 15:06:54 -05:00
Sahdev Zala
66777cf1e8 Doc: fix inconsistency and broken path
1) Fix the path for rpc errors
2) The README says 2 types of client errors vs doc mentioning 3
types of client errors. grpc erros should be considered single time
as covered in the README.
2021-02-03 12:02:49 -05:00
Piotr Tabor
4accc34c98
Merge pull request #11776 from brandond/fix_srv_11321
Fix cluster peer HTTP SRV discovery when no HTTPS records exist
2021-02-03 13:17:55 +01:00
Brad Davidson
603d975599 Fix cluster peer HTTP SRV discovery
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2021-02-03 03:08:13 -08:00
Piotr Tabor
6881ea828e
Merge pull request #12658 from dhermes/pr-12636-balancer
Prepare `balancer` interfaces for `>=google.golang.org/grpc@1.30.0` upgrade.
2021-02-02 23:03:56 +01:00
Danny Hermes
ea34f8dbc6 Prepare balancer interfaces for >=google.golang.org/grpc@1.30.0 upgrade. 2021-02-02 15:54:33 -06:00
Piotr Tabor
50ca440c49
Merge pull request #12410 from Yanhao/master
etcdserver: export method EtcdServer.leaderChangedNotify (#12378)
2021-02-02 12:23:11 +01:00
Yanhao Mo
6d82778a4e etcdserver: export method EtcdServer.leaderChangedNotify (#12378) 2021-02-02 18:13:32 +08:00
Sahdev Zala
e897daaebc
Merge pull request #12659 from spzala/genprotodoc
Clarify the root directory comment
2021-02-01 17:32:03 -05:00
Sahdev Zala
6bbef10721 Clarify the root directory comment
A contributor had an issue when ran script changing the directory
name after clone. This was discussed on the etcd slack channel.
2021-02-01 15:41:36 -05:00
Piotr Tabor
d6d03beaea
Merge pull request #12538 from lzhfromustc/12_9_GoroutineLeak
test: change channel operations to avoid potential goroutine leaks
2021-02-01 21:16:43 +01:00
Piotr Tabor
b5d11723d1
Merge pull request #12393 from viviyww/contrib-doc
contrib: del systemd/etcd2-backup-coreos in docs
2021-02-01 15:33:44 +01:00
Piotr Tabor
855eeb75c5
Merge pull request #12654 from mengjiao-liu/debian-1.4.0
update etcd base image to debian-base 1.4.0 which is now a sane multi-architecture image
2021-02-01 08:08:03 +01:00
mengjiao.liu
b5d4ea13ff update etcd base image to debian-base 1.4.0 2021-02-01 13:53:58 +08:00
Piotr Tabor
4af159a30a
Merge pull request #12259 from alvistack/master-aio_graceful_reboot
`etcd.service`: Define explicit dependencies of systemd etcd service
2021-01-31 23:58:12 +01:00
Piotr Tabor
a18cf56d8d
Merge pull request #12507 from etcd-io/add-codeql
Add CodeQL Security Scanning
2021-01-31 23:53:05 +01:00
Piotr Tabor
4d0a35a0fb
Merge pull request #11518 from nlamirault/patch-1
Add tag "etcd-mixin" into the Grafana dashboard
2021-01-31 23:27:11 +01:00
Piotr Tabor
958f6f9878
Merge pull request #12481 from kalexmills/fix-defer-log
fix: pass argument url in defer to avoid loopclosure
2021-01-31 23:20:32 +01:00
Piotr Tabor
2c8de24afc
Merge pull request #12495 from VertexC/fix/testErrorMsg
raft/raft_test: minor fix on inconsistent format of test case's error message
2021-01-31 22:25:23 +01:00
Piotr Tabor
6bb26ef008
Merge pull request #12533 from lucab/ups/contrib-sysusers
contrib/systemd: add a sysusers entry
2021-01-31 11:40:25 +01:00
Piotr Tabor
6db49d403c
Merge pull request #12600 from ravitri/remove_v2_alerts
Documentation: Removing etcd v2 specific alerts
2021-01-31 11:19:22 +01:00
Piotr Tabor
2fd6787506
Merge pull request #12653 from etcd-io/20201111-fix-flakes-go15
Tests:Use temp-directory that is covered by framework level cleanup
2021-01-31 11:05:13 +01:00
Piotr Tabor
0d7a671c75 Tests: Fix vet warnings about Fatal in sub-goroutines.
% (cd tests && go vet ./...)
stderr: # go.etcd.io/etcd/tests/v3/integration/clientv3/concurrency_test
stderr: integration/clientv3/concurrency/election_test.go:74:6: call to (*T).Fatal from a non-test goroutine
stderr: integration/clientv3/concurrency/mutex_test.go:57:4: call to (*T).Fatal from a non-test goroutine
2021-01-31 00:00:26 +01:00
Piotr Tabor
73ef0cde01 Update travis to use 1.15.7 go version. 2021-01-30 23:47:54 +01:00
Piotr Tabor
f0ecad00e3 Use temp-directory that is covered by framework level cleanup
Prior to this PR, the e2e tests where creating dirs like:
```
/tmp/testname1.etcd030299846
/tmp/testname0.etcd039445123
/tmp/testname0.etcd206372065
```
and not cleaning them, that led to disk-space-exceeded flakes.

After the PR, the testing.TB tempdir mechanism is used and the names are
being cleaned and are more miningful:

```
../../bin/etcd --name test-TestCtlV3EndpointHashKV-2 --listen-client-urls http://localhost:20010 --advertise-client-urls http://localhost:20010 --listen-peer-urls https://localhost:20011 --initial-advertise-peer-urls https://localhost:20011 --initial-cluster-token new --data-dir /tmp/TestCtlV3EndpointHashKV429176179/003 --snapshot-count 100000 --experimental-initial-corrupt-check --peer-auto-tls --initial-cluster test-TestCtlV3EndpointHashKV-0=https://localhost:20001,test-TestCtlV3EndpointHashKV-1=https://localhost:20006,test-TestCtlV3EndpointHashKV-2=https://localhost:20011
```
2021-01-30 13:25:55 +01:00
Piotr Tabor
cf2153d44b
Merge pull request #12614 from ptabor/prepare-to-remove-grpc-naming-package
Replacement API for client/v3/naming package to be compatible with new GRPC1.30+ resolver API.
2021-01-30 12:51:03 +01:00
Piotr Tabor
5d7c1db3a9 Introduce grpc-1.30+ compatible client/v3/naming API.
This is not yet implementation, just API and tests to be filled
with implementation in next CLs,
tracked by: https://github.com/etcd-io/etcd/issues/12652

We propose here 3 packages:
 - clientv3/naming/endpoints ->
    That is abstraction layer over etcd that allows to write, read &
    watch Endpoints information. It's independent from GRPC API. It hides
    the storage details.

 - clientv3/naming/endpoints/internal ->
    That contains the grpc's compatible Update class to preserve the
    internal JSON mashalling format.

 - clientv3/naming/resolver ->
   That implements the GRPC resolver API, such that etcd can be
   used for connection.Dial in grpc.

Please see the grpc_naming.md document changes & grpcproxy/cluster.go
new integration, to see how the new abstractions work.
2021-01-30 12:32:19 +01:00
Piotr Tabor
90d1b838ad Fix shellcheck warnings: Escaping.
```
build:3:65: note: Backslash is literal in "\n". Prefer explicit escaping: "\\n". [SC1117]
test:3:64: note: Backslash is literal in "\n". Prefer explicit escaping: "\\n". [SC1117]
scripts/fix.sh:23:17: note: Backslash is literal in "\n". Prefer explicit escaping: "\\n". [SC1117]
scripts/fix.sh:23:46: note: Backslash is literal in "\n". Prefer explicit escaping: "\\n". [SC1117]
scripts/fix.sh:30:17: note: Backslash is literal in "\n". Prefer explicit escaping: "\\n". [SC1117]
scripts/genproto.sh:38:17: note: Backslash is literal in "\n". Prefer explicit escaping: "\\n". [SC1117]
scripts/genproto.sh:53:17: note: Backslash is literal in "\n". Prefer explicit escaping: "\\n". [SC1117]
scripts/genproto.sh:73:23: note: Backslash is literal in "\(". Prefer explicit escaping: "\\(". [SC1117]
scripts/genproto.sh:73:33: note: Backslash is literal in "\(". Prefer explicit escaping: "\\(". [SC1117]
scripts/genproto.sh:74:69: note: Backslash is literal in "\1". Prefer explicit escaping: "\\1". [SC1117]
scripts/genproto.sh:74:78: note: Backslash is literal in "\2". Prefer explicit escaping: "\\2". [SC1117]
scripts/genproto.sh:75:40: note: Backslash is literal in "\(". Prefer explicit escaping: "\\(". [SC1117]
scripts/genproto.sh:75:52: note: Backslash is literal in "\1". Prefer explicit escaping: "\\1". [SC1117]
scripts/genproto.sh:91:17: note: Backslash is literal in "\n". Prefer explicit escaping: "\\n". [SC1117]
scripts/genproto.sh:115:17: note: Backslash is literal in "\n". Prefer explicit escaping: "\\n". [SC1117]
./test.sh:71:29: warning: Quote to prevent word splitting, or split robustly with mapfile or read -a. [SC2206]
./test.sh:400:28: note: Backslash is literal in "\t". Prefer explicit escaping: "\\t". [SC1117]
./test.sh:406:33: note: Backslash is literal in "\t". Prefer explicit escaping: "\\t". [SC1117]
./test.sh:415:36: note: Backslash is literal in "\*". Prefer explicit escaping: "\\*". [SC1117]
./test.sh:415:176: note: Backslash is literal in "\'". Prefer explicit escaping: "\\'". [SC1117]
./test.sh:493:56: note: Backslash is literal in "\.". Prefer explicit escaping: "\\.". [SC1117]
./test.sh:601:57: note: Backslash is literal in "\(". Prefer explicit escaping: "\\(". [SC1117]
./test.sh:601:64: note: Backslash is literal in "\)". Prefer explicit escaping: "\\)". [SC1117]
./test.sh:601:67: note: Backslash is literal in "\(". Prefer explicit escaping: "\\(". [SC1117]
./test.sh:601:74: note: Backslash is literal in "\)". Prefer explicit escaping: "\\)". [SC1117]
./test.sh:601:77: note: Backslash is literal in "\(". Prefer explicit escaping: "\\(". [SC1117]
./test.sh:601:84: note: Backslash is literal in "\)". Prefer explicit escaping: "\\)". [SC1117]
./test.sh:601:91: note: Backslash is literal in "\1". Prefer explicit escaping: "\\1". [SC1117]
./test.sh:601:94: note: Backslash is literal in "\2". Prefer explicit escaping: "\\2". [SC1117]
./test.sh:601:103: note: Backslash is literal in "\3". Prefer explicit escaping: "\\3". [SC1117]
```
2021-01-29 22:32:43 +00:00
Piotr Tabor
9600bbd5fe Update root image for docker-test-image to ubuntu:20.10 2021-01-29 22:16:51 +00:00
Piotr Tabor
73bfa8cfb1 wqMakefile: Docker generation should not override the src file
Also fix /tmp directory during: 'make docker-test' to make it executable.
2021-01-29 22:08:58 +00:00
Piotr Tabor
a8c935eab0 test.sh: Stop requiring/defaulting CPU flag to 4 2021-01-28 19:15:04 +00:00
Piotr Tabor
a4fac14353
Merge pull request #12648 from pacoxu/debian-1.3.0
update etcd base image to debian-base 1.3.0 for CVEs
2021-01-27 09:15:12 +01:00