16448 Commits

Author SHA1 Message Date
Gyuho Lee
8fc5ef4a03
Merge pull request #12418 from ptabor/20201023-fix-flaky
./pkg/testutil: wait for: (*watchGrpcStream).sendCloseSubstream(...) goroutines.
2020-10-24 11:53:10 -07:00
Ankur Gargi
8866d55b9b
command: Enhance health command to check if there are any active alarms (#12150) 2020-10-22 15:55:15 -07:00
Piotr Tabor
f2ee15a1e1 ./pkg/testutil: wait for: (*watchGrpcStream).sendCloseSubstream(...) goroutines.
Should solve the problem of flakes documented here:
https://github.com/etcd-io/etcd/issues/12372#issuecomment-706337969

```
% (cd tests && env go test -short -timeout=3m -cpu=4 --race=true ./... -p=2)

Unexpected goroutines running after all test(s).
1 instances of:
go.etcd.io/etcd/v3/clientv3.(*watchGrpcStream).sendCloseSubstream(...)
	/go/src/go.etcd.io/etcd/clientv3/watch.go:464 +0x204
created by go.etcd.io/etcd/v3/clientv3.(*watchGrpcStream).closeSubstream
	/go/src/go.etcd.io/etcd/clientv3/watch.go:480 +0x21f
FAIL	go.etcd.io/etcd/tests/v3/integration/clientv3/examples	2.111s
```

The goroutine finishes automatically with timeout of 250ms.The change
makes the test wait for it - if it still exists.

Examples:
  https://travis-ci.com/github/etcd-io/etcd/jobs/397449189
  https://travis-ci.com/github/etcd-io/etcd/jobs/397532784
  https://travis-ci.com/github/etcd-io/etcd/jobs/397696506
  https://travis-ci.com/github/etcd-io/etcd/jobs/403603526
2020-10-22 14:23:08 +02:00
Jingyi Hu
97354af44b
Merge pull request #12411 from ptabor/20201021-move-contrib-recipies
Modularization: Move contrib/recipies to clientv3/experimental/recipies/...
2020-10-22 17:40:05 +08:00
Piotr Tabor
45b007b8b4 contrib,clientv3: Move contrib/recipies to clientv3/experimental/recipies/...
Recipies is set of patterns / primitives implementation on top of clientv3.
It's used by integration tests. It shouldn't be considered "server" code.
2020-10-22 11:10:07 +02:00
Jingyi Hu
41557d9330
Merge pull request #12404 from ptabor/20201020-etcdctl-module
Modularization: etcdctl as a module
2020-10-21 21:48:38 +08:00
Piotr Tabor
09679d29ad etcdctl: Rename of imports after making etcdctl a module.
```
find -name '*.go' | xargs sed -i --follow-symlinks 's|etcd/v3/etcdctl|etcd/etcdctl/v3|g'
```
2020-10-21 11:15:35 +02:00
Gyuho Lee
ae962f7cda
Merge pull request #12406 from tangcong/silence-stream-log
embed: only log stream error with debug level
2020-10-20 11:03:37 -07:00
Gyuho Lee
d5363e3560
Merge pull request #12407 from tangcong/fix-networkpartition-flaky-test
clientv3test: fix network partition flaky test
2020-10-20 10:18:56 -07:00
tangcong
0a830c871e clientv3test: fix network partition flaky test 2020-10-20 23:13:23 +08:00
tangcong
27d308e173 embed: only log stream error with debug level 2020-10-20 21:56:06 +08:00
Piotr Tabor
df48e499ea etcdctl: Make etcdctl a module (go.mod, LICENSE) 2020-10-20 12:07:27 +02:00
Jingyi Hu
7c6c2c142b
Merge pull request #12400 from ptabor/20201015-clientv2
Modularization: Migrate clientv3 to go.etcd.io/etcd/client/v3
2020-10-20 18:03:04 +08:00
Piotr Tabor
e33c6dd9df client/v3: Rename of imports 2020-10-20 10:13:06 +02:00
Piotr Tabor
cf795c0ae6 client/v3: Create and update go.mod 2020-10-20 10:09:12 +02:00
Piotr Tabor
8a7f15511c clientv3: Move to client/v3 (just file move)
Mechanical:
% git mv clientv3 client/v3
% git mv client/mock client/v3/
2020-10-20 10:08:35 +02:00
Jingyi Hu
5d930e5dc1
Merge pull request #12395 from ptabor/20201007-raft
Modularization: Make /raft a module
2020-10-20 16:00:08 +08:00
Vitaliy Filippov
15f507f6b5
etcdserver: Fix 64 KB websocket notification message limit (#12403)
This fixes etcd being unable to send any message longer than 64 KB as
a notification over the websocket. This was because the older version
of grpc-websocket-proxy was used and WithMaxRespBodyBufferSize option
wasn't set.

Co-authored-by: Vitaliy Filippov <vitalif@yourcmc.ru>
2020-10-19 14:36:06 -07:00
Piotr Tabor
e62417297d *: Rename of imports of raft (as its now a module)
% find -name '*.go' -o -name '*.md' -o -name '*.sh' | xargs sed -i --follow-symlinks 's|etcd/v3/raft|etcd/raft/v3|g'
2020-10-16 13:58:18 +02:00
Piotr Tabor
bea5f379bc raft: Make raft a module
Establishing a module: Creating a go.mod files
2020-10-16 09:15:48 +02:00
Piotr Tabor
b7f0f52a16
*: Refresh of dependencies (#12399)
This PR focuses on dependencies that have new stable versions.
2020-10-15 15:32:00 -07:00
Gyuho Lee
e42127658c
Merge pull request #12391 from ptabor/20201013-module-client
Modularization: Move /client/ -> /client/v2 and make it a module
2020-10-15 10:26:44 -07:00
Piotr Tabor
45e4306508 client: Move client to client/v2 as a module.
We make v2 client code a module go.etcd.io/etcd/client/v2.

Pretty mechanical change that can be summarized as:

  mkdir client/v2
  cd client/v2 && git mod init go.etcd.io/etcd/client/v2

  git mv client/*.go client/v2/
  find -name '*.go' | xargs sed -i --follow-symlinks 's|/v3/client["]|/client/v2\"|g'

  + fixing changelog, bom, go.mod etc.
2020-10-15 14:39:43 +02:00
Gyuho Lee
d0d927f0db
Merge pull request #12397 from ptabor/20201015-update-protos2
tools: Update tools & go.mod for protos (gogo/protobuf 1.3.1, grpc 1.29.1, grpc-gateway 1.14.6)
2020-10-14 10:31:00 -07:00
Piotr Tabor
371ddf0b69 tests: Update diagnostic update in tests after change of proto version. 2020-10-14 18:46:38 +02:00
Piotr Tabor
cc7b4fab9f *: Execution of ./scripts/genproto.sh after versions change 2020-10-14 18:46:38 +02:00
Piotr Tabor
da5ca2c0d5 tools: Update of go.mod for protos (gogo/protobuf 1.3.1, grpc 1.29.1, grpc-gateway 1.14.6)
This brings consistency between proto-generation code and actual versions of libraries being used in runtime:

github.com/gogo/protobuf                        v1.2.1,v1.0.0 -> v1.3.1
github.com/golang/protobuf                             v1.3.2 -> v1.3.5
github.com/grpc-ecosystem/grpc-gateway  v1.9.5,v1.4.1,v1.15.2 -> v1.14.6
google.golang.org/grpc                                v1.26.0 -> v1.29.1

Moved as far as possible, without bumping on grpc 1.30.0 "naming" decomissioning.
Please also notice that gogo/protobuf is likely to reach EOL: https://github.com/gogo/protobuf/issues/691
2020-10-14 18:46:38 +02:00
Jingyi Hu
b08a46d37d
Merge pull request #12396 from ptabor/20201014-deps
test: Adding dependencies consistency check
2020-10-14 20:44:29 +08:00
Piotr Tabor
f7f17dd6a1 test: Adding dependencies consistency check
Examplar output for artificially inconsistent dependencies:

```
% PASSES="dep" ./test
% 'env' 'GO111MODULE=off' 'go' 'get' 'github.com/myitcv/gobin'
% 'env' 'GO111MODULE=off' 'go' 'get' 'github.com/myitcv/gobin'
% 'gofail' 'disable' 'etcdserver/' 'mvcc/backend/'
Running with -cpu=4 --race

'dep' started at Wed 14 Oct 2020 09:56:34 AM CEST
% (cd api && 'go' 'list' '-m')
% (cd api && 'go' 'list' '-f' '{{if not .Indirect}}{{if .Version}}{{.Path}},{{.Version}},go.etcd.io/etcd/api/v3{{end}}{{end}}' '-m' 'all')
% (cd pkg && 'go' 'list' '-m')
% (cd pkg && 'go' 'list' '-f' '{{if not .Indirect}}{{if .Version}}{{.Path}},{{.Version}},go.etcd.io/etcd/pkg/v3{{end}}{{end}}' '-m' 'all')
% 'go' 'list' '-m'
% 'go' 'list' '-f' '{{if not .Indirect}}{{if .Version}}{{.Path}},{{.Version}},go.etcd.io/etcd/v3{{end}}{{end}}' '-m' 'all'
% (cd tests && 'go' 'list' '-m')
% (cd tests && 'go' 'list' '-f' '{{if not .Indirect}}{{if .Version}}{{.Path}},{{.Version}},go.etcd.io/etcd/tests/v3{{end}}{{end}}' '-m' 'all')
FAIL: inconsistent versions for depencency: github.com/coreos/go-semver
github.com/coreos/go-semver:
  - v0.2.0 from: go.etcd.io/etcd/api/v3
  - v0.3.0 from: go.etcd.io/etcd/v3
FAIL: inconsistent dependencies
FAIL: 'dep' failed at Wed 14 Oct 2020 09:56:35 AM CEST
```
2020-10-14 09:57:27 +02:00
Joe Betz
afc5a717c9
Merge pull request #12390 from ptabor/20201013-scripts-fixes
tools: Migrate remaining tools to gobin
2020-10-13 13:58:51 -04:00
Piotr Tabor
bc9e433ca2 tools: Migrate remaining tools to gobin
Replace ./scripts/install_tool.sh with `gobin`, such that we have
consistent handling for all tools needed for build and consistent
versioning within ./tools/mod/go.mod.

Side changes:
  - Expose /scripts/fix.sh that fixes formatting and bom across modules
  - Expose *.sh variants of scripts like build and ./test (first step
towards replacement).
  - Make stderr output of commands explicit and make commands use
different color than callouts.
2020-10-13 19:33:01 +02:00
Joe Betz
061d8e8ead
Merge pull request #12354 from ptabor/202009027-module-pkgs
Modularization: Make pkg a module (and distribute not pkg code)
2020-10-13 12:56:21 -04:00
vivian
667ed96957
wal: remove redundant war tmp dirs if create wal failed (#12388)
If we create 000000000-0000000.war failed in war tmp dir(XXX.tmp) and we should
remove the tmp dirs. If create war successfully in tmp dir and it will
be renamed to common dir and remove tmp dir(XXX.tmp) will also be ok.

Co-authored-by: yangweiwei <yangweiwei@cmss.chinamobile.com>
2020-10-13 08:11:52 -07:00
Piotr Tabor
f408939aa1 bom: Update bill-of-materials generation logic (and refresh bom). 2020-10-13 07:50:08 +02:00
Piotr Tabor
04f63b6571 go.mod,tests: Update go.mod for tests and . to reflect new dependencies. 2020-10-13 00:09:27 +02:00
Piotr Tabor
de55bb6331 pkg: Rename imports after making 'pkg' a module
find -name '*.go' | xargs sed --follow-symlinks -i 's|go.etcd.io/etcd/v3/pkg/|go.etcd.io/etcd/pkg/v3/|g'
go fmt ./...
2020-10-13 00:09:27 +02:00
Piotr Tabor
bfa91d48af pkg: Estabilish pkg as a module 2020-10-12 23:58:09 +02:00
Piotr Tabor
1983c89e27 pkg/logutil, etcdserver: Move zap-raft logger to etcdserver.
This package depends on raft and is solelly used by etcdserver/raft.go.
So it does not fullfills conditions of:
```
pkg/ is a collection of utility packages used by etcd without being specific to etcd itself. A package belongs here
only if it could possibly be moved out into its own repository in the future.
```
2020-10-12 23:58:09 +02:00
Piotr Tabor
5ba46f625f pkg/mock: Update imports to point on new files locations.
find ./ -name '*.go' | xargs sed -i 's|go.etcd.io/etcd/v3/pkg/mock/mockserver|go.etcd.io/etcd/v3/client/mock/mockserver|g'
find ./ -name '*.go' | xargs sed -i 's|go.etcd.io/etcd/v3/pkg/mock|go.etcd.io/etcd/v3/server/mock|g'
go fmt ./...
2020-10-12 23:58:09 +02:00
Piotr Tabor
28036db6f0 Mechanical: Move mock packages out of ./pkg
Mechanical execution of:

```
git mv ./pkg/mock/mockserver ./client/pkg/mock
git mv ./pkg/mock/{mockstorage,mockstore,mockwait} ./server/pkg/mock
```

The packages depend on etcd API / protos - so they are NOT etcd-dependencies.
In such situation thay should be placed in 'pkg' folder.
2020-10-12 23:58:09 +02:00
Gyuho Lee
0b95e8cef1
Merge pull request #12374 from ptabor/20201009-new-genproto
script/genproto.sh: Refactor to be explicit about versions.
2020-10-12 14:25:43 -07:00
Gyuho Lee
fa83da446b
Merge pull request #12369 from ptabor/20201007-snapshot-out
Modularization: clientv3/snapshot -> etcdctl/snapshot
2020-10-12 14:24:45 -07:00
Sahdev Zala
891c69a5cf
Merge pull request #12386 from 2hamed/patch-1
README: Fixed the link to `functional` test dir
2020-10-12 14:55:48 -04:00
Hamed Momeni
386bf316a4 README: Fixed the link to test dir 2020-10-11 20:40:34 +02:00
Piotr Tabor
30811a06aa etcdctl, clientv3/snapshot: snapshot (Restore,Status) code out of client
"snapshot" Restore/Status code was the only remaining dependency of client on 'server'
code. The code is solelly used by etcdctl. Long-term the snapshot code
should be migrated to 'etcdadm' style of tool such that we can
distinguish tool solelly depending on networking API vs. tools that
operation on etcd files directly.

We left snapshot.Save() code in clientv3.snapshot package, such that
clients can benefits from automated download&safe to file snapshot
functionality over the wire.
2020-10-10 14:14:57 +02:00
Piotr Tabor
00e49d0c10
build: Fix updating 'go.etcd.io/etcd/api/v3/version.GitSHA' + test (#12382)
During move of code to 'api' (0aab02e7b5ab94ef666392f7a56e0780e7cb27f5),
I overlooked that the ./build script is setting the version on the 'symbol'.

I added a code to ./build script that checks whether the symbol's are in sync.
2020-10-09 17:32:15 -07:00
Sam Batschelet
e1bf097928
Merge pull request #12266 from chaitanyaenr/etcd_fsync_alert
Add alerts for etcd fsync duration
2020-10-09 14:42:12 -04:00
Joe Betz
bc0d8dad1e
Merge pull request #12380 from ptabor/20201009-move-single-integration
[important] tests/integration: Move misplaced integration test integration/v3_kv_test.go
2020-10-09 13:14:15 -04:00
Piotr Tabor
485030052a tests/integration: Move misplaces integration tests.
There was undetected 'conflict' between
  11ba1a610939218c6779a960f764b2bcfdd7fb83
and 2c66612e0ecb42abde6a4761cd708f5d285e0635.

Moving the file to proper location.
2020-10-09 09:39:55 +02:00
Josh Bleecher Snyder
24724af7f1
etcdserver: prevent crash using expvars when server is not running (#12376)
The raft.status expvar is added at init time.
This change ensures that evaluating that expvar variable
doesn't panic during evaluation, even when there is
no server running.
2020-10-08 17:37:25 -07:00