16562 Commits

Author SHA1 Message Date
retroflexer
3d2bb0797b CHANGELOG: Update for cherry picks for CVE 2021-01-21 15:26:40 -05:00
Piotr Tabor
7d444ee327
Merge pull request #12629 from ptabor/20210118-release-attempt
Modernize release script
2021-01-21 20:33:30 +01:00
Piotr Tabor
598ca6caab Modernize release script:
- making sure the DRY_RUN mode can finish e2e, so e.g. commits to
local copy of repository are OK in dry-run (while git pushes are NOT).
  - better interaction with ./test_lib.sh script.
  - more consistent logging
  - bringing back s390x architecture that on go 1.14.3 seems to work as
expected.
2021-01-21 17:47:36 +01:00
Piotr Tabor
5dcd459ae9
Merge pull request #12564 from dhermes/patch-1
Adding `clientv3` import alias to match usage in `register_test.go`.
2021-01-19 23:00:19 +01:00
Gyuho Lee
1d73d2e85e
Merge pull request #12632 from ptabor/20210119-fix-coverage-report
Fix code coverage measurement
2021-01-19 10:19:39 -08:00
Gyuho Lee
50352affd4
Merge pull request #12630 from anguslees/manifestlist
Generate and push multi-arch docker manifest list
2021-01-19 10:18:48 -08:00
Piotr Tabor
5472b3336b
Merge pull request #12525 from sakateka/remove_raft.peers
raft tests: Remove Config.peers and Config.learners
2021-01-19 16:00:54 +01:00
Piotr Tabor
486687843b Fix code coverage measurement. 2021-01-19 15:41:36 +01:00
Angus Lees
2529fee958 Use /build instead of deprecated /build.sh 2021-01-19 12:46:56 +11:00
Angus Lees
7f79ffb409 Generate and push multi-arch docker manifest list
Change amd64 docker image to use a `-amd64` suffix, consistent with
other architectures.  Introduce a multiarch docker manifest list with
the original (no-arch) docker tag.

Fixes #10380
2021-01-19 12:46:56 +11:00
Piotr Tabor
bb71f706a6
Merge pull request #12580 from dlorenc/update
Switch from dgrijalva/jwt-go to form3tech-oss/jwt-go.
2021-01-16 22:15:09 +01:00
Piotr Tabor
1a890a4659
Merge branch 'master' into update 2021-01-16 09:35:05 +01:00
Sergey Kacheev
ccfd00f687
raft: specify voters and learners via snapshot 2021-01-16 13:03:47 +07:00
Gyuho Lee
c0ab7cd6a1
Merge pull request #12624 from gyuho/maintainers
MAINTAINERS: add wenjiaswe@ and ptabor@
2021-01-15 12:34:05 -08:00
Jingyi Hu
84f270f88d
Merge pull request #12588 from ptabor/20201230-raft-docs
Raft: Expand raft documentation, in particular point on the godocs
2021-01-15 22:49:40 +08:00
Jingyi Hu
daa5bd11c0
Merge pull request #12497 from ptabor/20201127-release
Modularized release script integration
2021-01-15 22:46:41 +08:00
Piotr Tabor
bf6f173d5e Document Raft.send method.
The change makes it explicit that sending messages does not happen
immidietely and is subject to proper persist & then send protocol
on the application side. See:

https://github.com/etcd-io/etcd/issues/12589#issuecomment-752867024

for more context.
2021-01-15 12:35:58 +01:00
Piotr Tabor
58f78df1de Raft: Expand raft documentation, in particular point on godocs. 2021-01-15 12:34:02 +01:00
Piotr Tabor
577c898fee scripts: Integrate ./scripts/release with new code for tagging modules.
Changes:
  - signing tags.
  - allows to override BRANCH and REPOSITORY using env variables.

Tested by a release in my private fork:
  BRANCH="20201126-ptabor-release" REPOSITORY="git@github.com:ptabor/etcd.git" ./scripts/release 3.5.0-alpha.20
2021-01-15 12:31:44 +01:00
Gyuho Lee
c4c7d5d848 MAINTAINERS: add wenjiaswe@ and ptabor@
Both have shown great initiatives and dedication to the project's long-term health and community engagement.

I strongly support for their recognition and continued work on the project.

Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2021-01-14 15:16:38 -08:00
Sahdev Zala
69e99e80fa
Merge pull request #12465 from spacewander/fdoc
chore: update the documentation link in the comment
2021-01-14 00:39:25 -05:00
Sahdev Zala
e9e0cb1077
Merge pull request #12545 from sid597/dev_guide-api_concurrency-path-update
Documentation (dev_guide--api_concurrency_reference_v3) : Path to v3lock and v3election updated
2021-01-14 00:07:24 -05:00
Sahdev Zala
2c11bd5e19
Merge pull request #12586 from viviyww/clientconfigerror
transport: catch new cert error
2021-01-13 23:47:36 -05:00
Jingyi Hu
c95d9434f1
Merge pull request #12602 from ptabor/20210107-scripts-on-osx
Make ./script/fix.sh & test.sh OSX/BSD compatible
2021-01-13 00:30:58 +08:00
Piotr Tabor
84b5b87fb2 Make test.sh scripts OSX/BSD compatible:
- build & test scripts deprecated. Call *.sh variants.
  This will avoid delete the symlinks and get rid of
  subtle dependency on 'sed --follow-symlinks' on OsX/BSD sed.

- Fix parameters to mktemp
2021-01-12 16:29:00 +01:00
Jingyi Hu
bfc6e2ff30
Merge pull request #12611 from ptabor/20210111-fix-flakes
e2e tests flakes & leaks fixes: In particular TestIssue6361
2021-01-12 21:26:54 +08:00
Jingyi Hu
b760bb60e5
Merge pull request #12604 from ptabor/20210107-proto-post-modules
Refresh proto generation script after moving modules files.
2021-01-12 21:08:51 +08:00
Piotr Tabor
0d9cfc11c8 Fix usage of reflect.SliceHeader: reported by vet on tip golang
Example:
https://travis-ci.com/github/etcd-io/etcd/jobs/470404938

```
% (cd server && go vet ./...)
stderr: # go.etcd.io/etcd/server/v3/etcdserver/api/v2store
stderr: etcdserver/api/v2store/node_extern_test.go:107:9: possible misuse of reflect.SliceHeader
stderr: etcdserver/api/v2store/node_extern_test.go:107:16: possible misuse of reflect.SliceHeader
```
2021-01-12 00:14:51 +01:00
Piotr Tabor
70b5ef1d3a Fix tests flakiness: in particular TestIssue6361.
The root reason of flakes, was that server was considered as ready to
early.
In particular:
```
../../bin/etcd-2456648: {"level":"info","ts":"2021-01-11T09:56:44.474+0100","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"ed5f620d34a8e61b","remote-peer-id":"ca50e9357181d758"}
../../bin/etcd-2456648: {"level":"warn","ts":"2021-01-11T09:56:49.040+0100","caller":"etcdserver/server.go:1942","msg":"failed to publish local member to cluster through raft","local-member-id":"ed5f620d34a8e61b","local-member-attributes":"{Name:infra2 ClientURLs:[http://localhost:20030]}","request-path":"/0/members/ed5f620d34a8e61b/attributes","publish-timeout":"7s","error":"etcdserver: request timed out, possibly due to connection lost"}
../../bin/etcd-2456648: {"level":"info","ts":"2021-01-11T09:56:49.049+0100","caller":"etcdserver/server.go:1921","msg":"published local member to cluster through raft","local-member-id":"ed5f620d34a8e61b","local-member-attributes":"{Name:infra2 ClientURLs:[http://localhost:20030]}","request-path":"/0/members/ed5f620d34a8e61b/attributes","cluster-id":"34f27e83b3bc2ff","publish-timeout":"7s"}
```
was taking 5s.   If this was happening concurrently with etcdctl, the
etcdctl could timeout.

The fix, requires servers to report 'ready to serve client requests' to consider them up.

Fixed also some whitelisted 'goroutines'.
2021-01-12 00:14:51 +01:00
Piotr Tabor
74274f4417 e2e: Adding better diagnostic and location for temporary files to Snapshot tests. 2021-01-12 00:14:51 +01:00
Piotr Tabor
26f9b4be8f e2e tests were leaking 'defunc' etcdctl processes.
The commit ensures that spawned etcdctl processes are "closed",
so they perform proper os wait processing.
This might have contributed to file-descriptor/open-files limit being
exceeded.
2021-01-11 11:55:30 +01:00
Piotr Tabor
e2a65bee6e Avoid 'interactive prompt' for root password in etcd tests.
Before:
```
{"level":"info","ts":1610273495.3791487,"caller":"agent/handler.go:668","msg":"cleaning up page cache"}
{"level":"info","ts":1610273495.3793094,"caller":"agent/handler.go:94","msg":"created etcd log file","path":"/tmp/etcd-functional-2/etcd.log"}
{"level":"info","ts":1610273495.379328,"caller":"agent/handler.go:668","msg":"cleaning up page cache"}
[sudo] password for ptab:
pam_glogin: invalid password
Sorry, try again.
[sudo] password for ptab:
```

Now the caches are dropped if the current users is in sudoers, bot not
in the other cases.
To be honest I don't see the purpose for dropping the caches at all in
the test.
2021-01-11 10:06:31 +01:00
Piotr Tabor
8ccd4e1146 Fix flaky tests reported due to data race on grpc logging registration.
Example:
```
    ==================
    WARNING: DATA RACE
    Write at 0x000002178320 by goroutine 575:
      google.golang.org/grpc/grpclog.SetLoggerV2()
          /home/ptab/private/golang/pkg/mod/google.golang.org/grpc@v1.29.1/grpclog/loggerv2.go:70 +0x444
      go.etcd.io/etcd/server/v3/embed.(*Config).setupLogging.func1.1()
          /home/ptab/corp/etcd/server/embed/config_logging.go:119 +0x345
      sync.(*Once).doSlow()
          /usr/lib/google-golang/src/sync/once.go:66 +0x109
      sync.(*Once).Do()
          /usr/lib/google-golang/src/sync/once.go:57 +0x68
      go.etcd.io/etcd/server/v3/embed.(*Config).setupLogging.func1()
          /home/ptab/corp/etcd/server/embed/config_logging.go:109 +0x3b1
      go.etcd.io/etcd/server/v3/embed.(*Config).setupLogging()
          /home/ptab/corp/etcd/server/embed/config_logging.go:174 +0x6af
      go.etcd.io/etcd/server/v3/embed.(*Config).Validate()
          /home/ptab/corp/etcd/server/embed/config.go:553 +0x55
      go.etcd.io/etcd/server/v3/embed.StartEtcd()
          /home/ptab/corp/etcd/server/embed/etcd.go:93 +0x84
      go.etcd.io/etcd/tests/v3/integration.TestKVWithEmptyValue()
          /home/ptab/corp/etcd/tests/integration/v3_kv_test.go:33 +0x18c
      testing.tRunner()
          /usr/lib/google-golang/src/testing/testing.go:1123 +0x202

    Previous read at 0x000002178320 by goroutine 956:
      [failed to restore the stack]

    Goroutine 575 (running) created at:
      testing.(*T).Run()
          /usr/lib/google-golang/src/testing/testing.go:1168 +0x5bb
      testing.runTests.func1()
          /usr/lib/google-golang/src/testing/testing.go:1441 +0xa6
      testing.tRunner()
          /usr/lib/google-golang/src/testing/testing.go:1123 +0x202
      testing.runTests()
          /usr/lib/google-golang/src/testing/testing.go:1439 +0x612
      testing.(*M).Run()
          /usr/lib/google-golang/src/testing/testing.go:1347 +0x3c4
      go.etcd.io/etcd/pkg/v3/testutil.MustTestMainWithLeakDetection()
          /home/ptab/corp/etcd/pkg/testutil/leak.go:150 +0x38
      go.etcd.io/etcd/tests/v3/integration.TestMain()
          /home/ptab/corp/etcd/tests/integration/main_test.go:14 +0x272
      main.main()
          _testmain.go:349 +0x269

    Goroutine 956 (finished) created at:
      google.golang.org/grpc/internal/transport.newHTTP2Server()
          /home/ptab/private/golang/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_server.go:288 +0x18a4
      google.golang.org/grpc/internal/transport.NewServerTransport()
          /home/ptab/private/golang/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:534 +0x2f5
      google.golang.org/grpc.(*Server).newHTTP2Transport()
          /home/ptab/private/golang/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:726 +0x2ca
      google.golang.org/grpc.(*Server).handleRawConn()
          /home/ptab/private/golang/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:693 +0x60f
      google.golang.org/grpc.(*Server).Serve.func3()
          /home/ptab/private/golang/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:663 +0x4c
    ==================
    ...
    {"level":"info","ts":"2021-01-09T22:21:04.550+0100","caller":"embed/etcd.go:330","msg":"closed etcd server","name":"default","data-dir":"/tmp/etcd-017337431","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
    --- FAIL: TestKVWithEmptyValue (1.08s)
        v3_kv_test.go:62: my-namespace/foobar = data
        v3_kv_test.go:62: my-namespace/foobar1 = data
        v3_kv_test.go:62: namespace/foobar1 = data
        v3_kv_test.go:72: foobar = data
        v3_kv_test.go:72: foobar1 = data
        v3_kv_test.go:87: delete keys:2
        testing.go:1038: race detected during execution of test
```
2021-01-11 10:06:31 +01:00
Dan Lorenc
5b90402082 Switch from dgrijalva/jwt-go to form3tech-oss/jwt-go.
dgrijalva/jwt-go has been abandoned and contains several serious
security issues. Most projects are now switching to the form3tech fork.

See https://snyk.io/vuln/SNYK-GOLANG-GITHUBCOMDGRIJALVAJWTGO-596515 for
info on the issues.

Signed-off-by: Dan Lorenc <dlorenc@google.com>
2021-01-10 08:04:20 -06:00
Piotr Tabor
23340bb62a Refresh proto generation script after moving modules files.
With modulatiozation server protos get moved into ./server directory,
but it was not reflected in scripts/genproto.sh.
2021-01-08 16:33:12 +01:00
Sahdev Zala
c632042bb9
Merge pull request #12489 from zhijianli88/cleanup-tmpfiles
Cleanup tmpfiles
2021-01-07 12:26:04 -05:00
Danny Hermes
91bdc7eb49 Putting etcd/v3 as a FORBIDDEN_DEPENDENCY for etcdctl. 2021-01-07 08:45:24 -06:00
Sahdev Zala
a4570a60e7
Merge pull request #12549 from bbiao/bugfix/refresh_token
client/v3: clear auth token when encounter ErrInvalidAuthToken
2020-12-29 22:08:14 -05:00
yangweiwei
826573586f transport: catch new cert error
in pkg/transport, we should catch NewCert error.
2020-12-30 09:33:42 +08:00
bbiao
af4ef4ec04
client/v3: clear auth token when encounter ErrInvalidAuthToken
Old etcdserver which have not apply pr of #12165 will check auth token
even if the request is an Authenticate request.
If the client has a invalid auth token, it will not able to update it's
token, since the Authenticate has a invalid auth token.
This fix clear the auth token when encounter an ErrInvalidAuthToken to
talk with old version etcd servers.

Fix #12385 with #12165 and #12264
2020-12-27 11:08:22 +08:00
Sahdev Zala
a1ff0d5373
Merge pull request #12328 from viviyww/tests-e2e-panic-case
tests: fix test case panic error
2020-12-24 14:54:33 -05:00
Gyuho Lee
a88877ec7b
Merge pull request #12579 from ianwoolf/fix-clientv3-readme
docs: update client v3's godoc link in readme
2020-12-23 14:08:30 -08:00
Sahdev Zala
ad5b30297a
Merge pull request #12575 from nate-double-u/81-docs-toc-compilation
Documentation Content: TOC — Compilation PR (weights & descriptions)
2020-12-23 15:27:49 -05:00
Nate W
6c08437ec3 Adding weights and descriptions to Documentation files' frontmatter.
Updating for new TOC as per issue https://github.com/etcd-io/website/issues/81

Signed-off-by: Nate W <4453979+nate-double-u@users.noreply.github.com>
2020-12-23 11:13:04 -08:00
ianwoolf
fc5b5fa31e fix godoc link in clientv3 readme 2020-12-23 10:59:13 +08:00
Sahdev Zala
4726911ea5
Merge pull request #12570 from juzhiyuan/patch-1
docs: update client v3's README
2020-12-21 12:03:50 -05:00
琚致远
bd97953de3
Update README.md 2020-12-21 17:40:40 +08:00
Sahdev Zala
0539bcb81f
Merge pull request #12509 from nate-double-u/81-docs-toc-sections
Documentation Content: TOC — Sections Order
2020-12-19 18:17:35 -05:00
琚致远
549bff5980
docs: update client v3's README 2020-12-20 00:06:10 +08:00
Danny Hermes
47dbeeb65a
Removing errant commented out replace line 2020-12-15 12:05:26 -06:00