17042 Commits

Author SHA1 Message Date
Piotr Tabor
8ccd4e1146 Fix flaky tests reported due to data race on grpc logging registration.
Example:
```
    ==================
    WARNING: DATA RACE
    Write at 0x000002178320 by goroutine 575:
      google.golang.org/grpc/grpclog.SetLoggerV2()
          /home/ptab/private/golang/pkg/mod/google.golang.org/grpc@v1.29.1/grpclog/loggerv2.go:70 +0x444
      go.etcd.io/etcd/server/v3/embed.(*Config).setupLogging.func1.1()
          /home/ptab/corp/etcd/server/embed/config_logging.go:119 +0x345
      sync.(*Once).doSlow()
          /usr/lib/google-golang/src/sync/once.go:66 +0x109
      sync.(*Once).Do()
          /usr/lib/google-golang/src/sync/once.go:57 +0x68
      go.etcd.io/etcd/server/v3/embed.(*Config).setupLogging.func1()
          /home/ptab/corp/etcd/server/embed/config_logging.go:109 +0x3b1
      go.etcd.io/etcd/server/v3/embed.(*Config).setupLogging()
          /home/ptab/corp/etcd/server/embed/config_logging.go:174 +0x6af
      go.etcd.io/etcd/server/v3/embed.(*Config).Validate()
          /home/ptab/corp/etcd/server/embed/config.go:553 +0x55
      go.etcd.io/etcd/server/v3/embed.StartEtcd()
          /home/ptab/corp/etcd/server/embed/etcd.go:93 +0x84
      go.etcd.io/etcd/tests/v3/integration.TestKVWithEmptyValue()
          /home/ptab/corp/etcd/tests/integration/v3_kv_test.go:33 +0x18c
      testing.tRunner()
          /usr/lib/google-golang/src/testing/testing.go:1123 +0x202

    Previous read at 0x000002178320 by goroutine 956:
      [failed to restore the stack]

    Goroutine 575 (running) created at:
      testing.(*T).Run()
          /usr/lib/google-golang/src/testing/testing.go:1168 +0x5bb
      testing.runTests.func1()
          /usr/lib/google-golang/src/testing/testing.go:1441 +0xa6
      testing.tRunner()
          /usr/lib/google-golang/src/testing/testing.go:1123 +0x202
      testing.runTests()
          /usr/lib/google-golang/src/testing/testing.go:1439 +0x612
      testing.(*M).Run()
          /usr/lib/google-golang/src/testing/testing.go:1347 +0x3c4
      go.etcd.io/etcd/pkg/v3/testutil.MustTestMainWithLeakDetection()
          /home/ptab/corp/etcd/pkg/testutil/leak.go:150 +0x38
      go.etcd.io/etcd/tests/v3/integration.TestMain()
          /home/ptab/corp/etcd/tests/integration/main_test.go:14 +0x272
      main.main()
          _testmain.go:349 +0x269

    Goroutine 956 (finished) created at:
      google.golang.org/grpc/internal/transport.newHTTP2Server()
          /home/ptab/private/golang/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_server.go:288 +0x18a4
      google.golang.org/grpc/internal/transport.NewServerTransport()
          /home/ptab/private/golang/pkg/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:534 +0x2f5
      google.golang.org/grpc.(*Server).newHTTP2Transport()
          /home/ptab/private/golang/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:726 +0x2ca
      google.golang.org/grpc.(*Server).handleRawConn()
          /home/ptab/private/golang/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:693 +0x60f
      google.golang.org/grpc.(*Server).Serve.func3()
          /home/ptab/private/golang/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:663 +0x4c
    ==================
    ...
    {"level":"info","ts":"2021-01-09T22:21:04.550+0100","caller":"embed/etcd.go:330","msg":"closed etcd server","name":"default","data-dir":"/tmp/etcd-017337431","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"]}
    --- FAIL: TestKVWithEmptyValue (1.08s)
        v3_kv_test.go:62: my-namespace/foobar = data
        v3_kv_test.go:62: my-namespace/foobar1 = data
        v3_kv_test.go:62: namespace/foobar1 = data
        v3_kv_test.go:72: foobar = data
        v3_kv_test.go:72: foobar1 = data
        v3_kv_test.go:87: delete keys:2
        testing.go:1038: race detected during execution of test
```
2021-01-11 10:06:31 +01:00
Dan Lorenc
5b90402082 Switch from dgrijalva/jwt-go to form3tech-oss/jwt-go.
dgrijalva/jwt-go has been abandoned and contains several serious
security issues. Most projects are now switching to the form3tech fork.

See https://snyk.io/vuln/SNYK-GOLANG-GITHUBCOMDGRIJALVAJWTGO-596515 for
info on the issues.

Signed-off-by: Dan Lorenc <dlorenc@google.com>
2021-01-10 08:04:20 -06:00
Piotr Tabor
23340bb62a Refresh proto generation script after moving modules files.
With modulatiozation server protos get moved into ./server directory,
but it was not reflected in scripts/genproto.sh.
2021-01-08 16:33:12 +01:00
Sahdev Zala
c632042bb9
Merge pull request #12489 from zhijianli88/cleanup-tmpfiles
Cleanup tmpfiles
2021-01-07 12:26:04 -05:00
Danny Hermes
91bdc7eb49 Putting etcd/v3 as a FORBIDDEN_DEPENDENCY for etcdctl. 2021-01-07 08:45:24 -06:00
Ravi Trivedi
0abc7515b9 Documentation: Removing etcd v2 specific etcd_http* alerts 2021-01-05 22:28:30 +05:30
mlmhl
ebf461a7de backend: fix buffer range bug 2020-12-30 16:05:42 +08:00
Sahdev Zala
a4570a60e7
Merge pull request #12549 from bbiao/bugfix/refresh_token
client/v3: clear auth token when encounter ErrInvalidAuthToken
2020-12-29 22:08:14 -05:00
yangweiwei
826573586f transport: catch new cert error
in pkg/transport, we should catch NewCert error.
2020-12-30 09:33:42 +08:00
bbiao
af4ef4ec04
client/v3: clear auth token when encounter ErrInvalidAuthToken
Old etcdserver which have not apply pr of #12165 will check auth token
even if the request is an Authenticate request.
If the client has a invalid auth token, it will not able to update it's
token, since the Authenticate has a invalid auth token.
This fix clear the auth token when encounter an ErrInvalidAuthToken to
talk with old version etcd servers.

Fix #12385 with #12165 and #12264
2020-12-27 11:08:22 +08:00
Sahdev Zala
a1ff0d5373
Merge pull request #12328 from viviyww/tests-e2e-panic-case
tests: fix test case panic error
2020-12-24 14:54:33 -05:00
Gyuho Lee
a88877ec7b
Merge pull request #12579 from ianwoolf/fix-clientv3-readme
docs: update client v3's godoc link in readme
2020-12-23 14:08:30 -08:00
Sahdev Zala
ad5b30297a
Merge pull request #12575 from nate-double-u/81-docs-toc-compilation
Documentation Content: TOC — Compilation PR (weights & descriptions)
2020-12-23 15:27:49 -05:00
Nate W
6c08437ec3 Adding weights and descriptions to Documentation files' frontmatter.
Updating for new TOC as per issue https://github.com/etcd-io/website/issues/81

Signed-off-by: Nate W <4453979+nate-double-u@users.noreply.github.com>
2020-12-23 11:13:04 -08:00
ianwoolf
fc5b5fa31e fix godoc link in clientv3 readme 2020-12-23 10:59:13 +08:00
Sahdev Zala
4726911ea5
Merge pull request #12570 from juzhiyuan/patch-1
docs: update client v3's README
2020-12-21 12:03:50 -05:00
琚致远
bd97953de3
Update README.md 2020-12-21 17:40:40 +08:00
Sahdev Zala
0539bcb81f
Merge pull request #12509 from nate-double-u/81-docs-toc-sections
Documentation Content: TOC — Sections Order
2020-12-19 18:17:35 -05:00
琚致远
549bff5980
docs: update client v3's README 2020-12-20 00:06:10 +08:00
Danny Hermes
47dbeeb65a
Removing errant commented out replace line 2020-12-15 12:05:26 -06:00
Danny Hermes
c899316fe8
Adding clientv3 import alias to match usage in register_test.go. 2020-12-15 11:54:03 -06:00
Sahdev Zala
a3174d0f8e
Merge pull request #12544 from sid597/learning-authDesign-link-update
[3.4.0] Documentation (learning-design_auth_v3) : Path to client.go updated
2020-12-13 20:12:46 -05:00
Sahdev Zala
cb6951c9f0
Merge pull request #12540 from sid597/learning-why-links-update
[3.4.0] Documentation (Learning-why) : Broken links updated
2020-12-12 21:49:30 -05:00
sid597
67da5e9722 Documentation (dev_guide--api_concurrency_reference_v3) : Path to v3lock
and v3election updated

Current path for v3lock and v3election are set to :
etcdserver/api/v3lock/v3lockpb... and etcdserver/api/v3election/v3electionpb... respectively
changed them to server/etcdserver...
2020-12-11 18:21:17 +05:30
sid597
07fbd24f3d [3.4.0] Documentation (learning-design_auth_v3) : Path to client.go
outdated
2020-12-11 18:11:26 +05:30
Siddharth Yadav
fe4833730d
recipes link updated 2020-12-10 23:20:14 +05:30
sid597
2e873e6a9f Documentation (Learning-why) : Broken links updated 2020-12-10 22:42:38 +05:30
lzhfromustc
f2a912a4e6 test: change channel operations to avoid potential goroutine leaks
In these unit tests, goroutines may leak if certain branches are chosen. This commit edits channel operations and buffer sizes, so no matter what branch is chosen, the test will end correctly. This commit doesn't change the semantics of unit tests.
2020-12-09 22:23:21 -05:00
Luca BRUNO
b0e2c70c71
contrib/systemd: add a sysusers entry
This adds a sysusers.d file, in order to create a system user/group
which matches the one used by the service unit.

Ref: https://www.freedesktop.org/software/systemd/man/sysusers.d.html
2020-12-09 13:59:46 +00:00
Nate W
bedad020eb Adding weights to main section files
Updating for new TOC as per issue https://github.com/etcd-io/website/issues/81

Signed-off-by: Nate W <4453979+nate-double-u@users.noreply.github.com>
2020-12-08 16:22:29 -08:00
Sahdev Zala
ca866c0242
Merge pull request #12511 from randomizedcoder/failures.md
failures.md explicit cluster members note
2020-12-03 20:02:09 -05:00
dseddon
0c6357469d failures.md explicit cluster members note 2020-12-03 15:47:07 -08:00
Sahdev Zala
bfc97c1123
Merge pull request #12499 from ptabor/20201127-modules-docs
Documentation: Developer's docs about etcd-3.5.x modules
2020-12-02 17:20:44 -05:00
Chris Aniszczyk
c0d44ec4b7
Add CodeQL Security Scanning
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
2020-12-02 10:37:09 -08:00
Piotr Tabor
4d173231d1 Documentation: Developer's docs about etcd-3.5.x modules 2020-12-02 10:00:09 +01:00
VertexC
082b9a815d minor fix on inconsistent format of test case's error message 2020-11-30 12:18:12 +08:00
Gyuho Lee
28d1af294e CHANGELOG-3.4: update release dates
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2020-11-25 12:21:32 -08:00
Gyuho Lee
64c409e124
Merge pull request #12491 from gyuho/better-exec
*: validate exec command args, remove unused "iptables" wrapper
2020-11-25 11:26:58 -08:00
Gyuho Lee
b40b46f473 CHANGELOG: updates
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2020-11-25 10:40:42 -08:00
Gyuho Lee
85b61c0f4d pkg/netutil: remove unused "iptables" wrapper
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2020-11-25 10:38:42 -08:00
Gyuho Lee
3eea37cf31 tools/etcd-dump-metrics: validate exec cmd args
To prevent arbitrary command invocations.

Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2020-11-25 10:37:52 -08:00
Li Zhijian
fb3f91c141 test: cleanup temporary file for TestConfigFromFile
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
2020-11-25 17:40:43 +08:00
Li Zhijian
63c284211d test: cleanup temporary files for TestZeroToEnd
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
2020-11-25 17:33:43 +08:00
K. Alex Mills
3f6e0ec94b fix: pass argument url in defer to avoid loopclosure
Because of the well-known range loop closure issue, the value of u may
have changed by the time the anonymous function mentioned in the defer
is run. To address this, the simplest fix is to pass the url used in the
loop as an argument to the function run in defer.
2020-11-19 15:29:26 -06:00
Gyuho Lee
b5cefb5b3d
Merge pull request #12392 from ironcladlou/fixture-mutations
tests: prevent cross-test contamination via shared state
2020-11-19 10:05:42 -08:00
Dan Mace
9571325fe8 etcdserver: fix incorrect metrics generated when clients cancel watches
Before this patch, a client which cancels the context for a watch results in the
server generating a `rpctypes.ErrGRPCNoLeader` error that leads the recording of
a gRPC `Unavailable` metric in association with the client watch cancellation.
The metric looks like this:

    grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}

So, the watch server has misidentified the error as a server error and then
propagates the mistake to metrics, leading to a false indicator that the leader
has been lost. This false signal then leads to false alerting.

The commit 9c103dd0dedfc723cd4f33b6a5e81343d8a6bae7 introduced an interceptor which wraps
watch streams requiring a leader, causing those streams to be actively canceled
when leader loss is detected.

However, the error handling code assumes all stream context cancellations are
from the interceptor. This assumption is broken when the context was canceled
because of a client stream cancelation.

The core challenge is lack of information conveyed via `context.Context` which
is shared by both the send and receive sides of the stream handling and is
subject to cancellation by all paths (including the gRPC library itself). If any
piece of the system cancels the shared context, there's no way for a context
consumer to understand who cancelled the context or why.

To solve the ambiguity of the stream interceptor code specifically, this patch
introduces a custom context struct which the interceptor uses to expose a custom
error through the context when the interceptor decides to actively cancel a
stream. Now the consuming side can more safely assume a generic context
cancellation can be propagated as a cancellation, and the server generated
leader error is preserved and propagated normally without any special inference.

When a client cancels the stream, there remains a race in the error handling
code between the send and receive goroutines whereby the underlying gRPC error
is lost in the case where the send path returns and is handled first, but this
issue can be taken separately as no matter which paths wins, we can detect a
generic cancellation.

This is a replacement of https://github.com/etcd-io/etcd/pull/11375.

Fixes #10289, #9725, #9576, #9166
2020-11-18 17:02:09 -05:00
Jingyi Hu
c11ddc65ce
Merge pull request #12448 from agargi/introduce_config_parameter
server: Added config parameter experimental-apply-warning-duration
2020-11-19 02:29:08 +08:00
Ankur Gargi
c1c681adc3 server: Added config parameter experimental-warning-apply-duration 2020-11-17 17:33:19 -05:00
Sam Batschelet
06e48f0486
Merge pull request #12476 from hexfusion/mixin-typo
Documentation/etcd-mixin: fix typo
2020-11-15 19:19:35 -05:00
Sam Batschelet
07c15890ab Documentation/etcd-mixin: fix typo
Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
2020-11-15 12:26:18 -05:00