Mirroristas/etcd
2
0
Fork 0
mirror of https://github.com/etcd-io/etcd.git synced 2024-09-27 06:25:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
16,843 Commits 15 Branches 491 Tags 148 MiB
Commit Graph

16843 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
chao
80586c5b47
etcdserver/util.go: reduce memory when logging range requests (#12871) 
* etcdserver/util.go: reduce memory when logging range requests

Fixes #12835

* Update CHANGELOG-3.5.md
 2021-04-16 16:39:34 -07:00
Piotr Tabor
5744cdf199
Merge pull request #12870 from ptabor/20210416-fix-flake-TestSnapshotV3RestoreMultiMemberAdd-master 
Fix TestSnapshotV3RestoreMultiMemberAdd flakes (leaks)
 2021-04-16 21:49:46 +02:00
Piotr Tabor
17b982382e Fix TestSnapshotV3RestoreMultiMemberAdd flakes (leaks) 
- most important: unix's socket transport should not keep idle
connections. For top-level Transport we close them using:
    f3c518025e/server/etcdserver/api/rafthttp/transport.go (L226)
    but currently we don't have access to close them witing the nest (unix) transport. Short idle deadline is good enough.

  - Use dialContext (instead of dial) to make sure context is passed down the stack
  - Make sure Context is cancelled as soon as the operation is done in pipeline
  - nit: use dedicated method to yeld goroutines.

Tested with:
```
d=$(date +"%Y%m%d_%H%M")
(cd tests && go test --timeout=60m ./integration/snapshot -run TestSnapshotV3RestoreMultiMemberAdd -v --count=180 2>&1 | tee log_${d}.log)
```

There were transports & cmux leaked:

```
   leak.go:118: Test appears to have leaked a Transport:
        internal/poll.runtime_pollWait(0x7f6c5c3784c8, 0x72, 0xffffffffffffffff)
        	/usr/lib/google-golang/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc003296298, 0x72, 0x0, 0x18, 0xffffffffffffffff)
        	/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:87 +0x45
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc003296280, 0xc0031f60a8, 0x18, 0x18, 0x0, 0x0, 0x0)
        	/usr/lib/google-golang/src/internal/poll/fd_unix.go:166 +0x1d5
        net.(*netFD).Read(0xc003296280, 0xc0031f60a8, 0x18, 0x18, 0x18, 0xc0009056e2, 0x203000)
        	/usr/lib/google-golang/src/net/fd_posix.go:55 +0x4f
        net.(*conn).Read(0xc000010258, 0xc0031f60a8, 0x18, 0x18, 0x0, 0x0, 0x0)
        	/usr/lib/google-golang/src/net/net.go:183 +0x91
        github.com/soheilhy/cmux.(*bufferedReader).Read(0xc0003d24e0, 0xc0031f60a8, 0x18, 0x18, 0xc0003d24d0, 0xc0009056e2, 0xc000278400)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/buffer.go:53 +0x12d
        github.com/soheilhy/cmux.hasHTTP2Preface(0x1367e20, 0xc0003d24e0, 0x7f6c5c699f40)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/matchers.go:195 +0x8a
        github.com/soheilhy/cmux.matchersToMatchWriters.func1(0x7f6c5c699f40, 0xc000010258, 0x1367e20, 0xc0003d24e0, 0xc000010258)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:128 +0x39
        github.com/soheilhy/cmux.(*cMux).serve(0xc003228690, 0x138c410, 0xc000010258, 0xc00327f740, 0xc0059ba860)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:192 +0x1e7
        created by github.com/soheilhy/cmux.(*cMux).Serve
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:179 +0x191

        internal/poll.runtime_pollWait(0x7f6c5c60f3f0, 0x72, 0xffffffffffffffff)
        	/usr/lib/google-golang/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc000d53018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
        	/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:87 +0x45
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc000d53000, 0xc000cfd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        	/usr/lib/google-golang/src/internal/poll/fd_unix.go:166 +0x1d5
        net.(*netFD).Read(0xc000d53000, 0xc000cfd000, 0x1000, 0x1000, 0x3, 0x3, 0x1000000000001)
        	/usr/lib/google-golang/src/net/fd_posix.go:55 +0x4f
        net.(*conn).Read(0xc00031a570, 0xc000cfd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        	/usr/lib/google-golang/src/net/net.go:183 +0x91
        net/http.(*persistConn).Read(0xc00093b320, 0xc000cfd000, 0x1000, 0x1000, 0x577750, 0x60, 0x0)
        	/usr/lib/google-golang/src/net/http/transport.go:1933 +0x77
        bufio.(*Reader).fill(0xc005702fc0)
        	/usr/lib/google-golang/src/bufio/bufio.go:101 +0x108
        bufio.(*Reader).Peek(0xc005702fc0, 0x1, 0xc00077c660, 0xc003b082a0, 0xc000d08de0, 0x5ae586, 0x11dd6c0)
        	/usr/lib/google-golang/src/bufio/bufio.go:139 +0x4f
        net/http.(*persistConn).readLoop(0xc00093b320)
        	/usr/lib/google-golang/src/net/http/transport.go:2094 +0x1a8
        created by net/http.(*Transport).dialConn
        	/usr/lib/google-golang/src/net/http/transport.go:1754 +0xdaa

        net/http.(*persistConn).writeLoop(0xc00093b320)
        	/usr/lib/google-golang/src/net/http/transport.go:2393 +0xf7
        created by net/http.(*Transport).dialConn
        	/usr/lib/google-golang/src/net/http/transport.go:1755 +0xdcf

        sync.runtime_Semacquire(0xc0059ba868)
        	/usr/lib/google-golang/src/runtime/sema.go:56 +0x45
        sync.(*WaitGroup).Wait(0xc0059ba860)
        	/usr/lib/google-golang/src/sync/waitgroup.go:130 +0x65
        github.com/soheilhy/cmux.(*cMux).Serve.func1(0xc003228690, 0xc0059ba860)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:158 +0x56
        github.com/soheilhy/cmux.(*cMux).Serve(0xc003228690, 0x13698c0, 0xc00377a0f0)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:173 +0x115
        go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers.func1(0xc0007cc360, 0x122b75f)
        	/home/ptab/corp/etcd/server/embed/etcd.go:518 +0x2b9
        go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers.func3(0xc00036d080, 0xc0059330a0)
        	/home/ptab/corp/etcd/server/embed/etcd.go:549 +0x182
        created by go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers
        	/home/ptab/corp/etcd/server/embed/etcd.go:543 +0x73a
--- FAIL: TestSnapshotV3RestoreMultiMemberAdd (17.74s)
```
 2021-04-16 20:17:28 +02:00
Piotr Tabor
f3c518025e
Merge pull request #12861 from ptabor/20210413-test-logging-fixes 
Embedded server should not mess global loggers (by default)
 2021-04-16 00:07:35 +02:00
Piotr Tabor
b47c5fcc12 Address review comments a.d. logging. 2021-04-15 17:54:37 +02:00
Piotr Tabor
c73da740fa
Merge pull request #12862 from lilic/enable-race 
.travis.yaml: Enables race in the tests
 2021-04-15 17:37:03 +02:00
Lili Cosic
2843fded06 .travis.yaml: Enables race in the tests 
As the TestV3WatchRestoreSnapshotUnsync flakiness was fixed, this
removes the TODO and enables the race again.
 2021-04-15 14:32:52 +02:00
Piotr Tabor
fad6391745 Configure proper logging for grpc integration tests. 2021-04-14 12:47:38 +02:00
Piotr Tabor
d72f7ef5cc Give control to Embedded servers whether they override global loggers 
So far each instance of embed server was overriding the grpc loggers and zap.global loggers.
It's counter intutitive that last created Embedded server was 'wining' and more-over it was breaking grpc expectation to change it "only" before the grpc stack is being used.

This PR introduces explicit call: `embed.Config::SetupGlobalLoggers()`, that changes the loggers where requested. The call is used by etcd main binary.

The immediate benefit from this change is reduction of  test flakiness, as there were flakes due to not a proper logger being used across tests.
 2021-04-14 12:47:38 +02:00
Piotr Tabor
eafbc8c57e Update zap logging dependency. 
In particular bring up zapgrpc V2 code:
89e382035d
https://pkg.go.dev/google.golang.org/grpc/grpclog#LoggerV2
 2021-04-14 12:15:48 +02:00
Piotr Tabor
57a092b45d
Merge pull request #12859 from tomwilkie/fix-mixin 
Fix the mixin.
 2021-04-13 23:05:52 +02:00
Tom Wilkie
562d645ac9
Fix the mixin. 
Signed-off-by: Tom Wilkie <tom@grafana.com>
 2021-04-13 19:38:55 +01:00
Piotr Tabor
4388bfc925
Merge pull request #12858 from hnlq715/patch-1 
client: fix doc typo
 2021-04-13 17:33:43 +02:00
大可
5db0070150
Update doc.go 2021-04-13 22:22:42 +08:00
大可
d7e971e8d8
client: fix doc typo 2021-04-13 16:33:24 +08:00
Piotr Tabor
7f97dfd45a
Merge pull request #12795 from wpedrak/resend-read-index-on-first-commit-in-term 
etcdserver: resend ReadIndex request on empty apply request
 2021-04-09 16:41:54 +02:00
wpedrak
08ea9cb756 etcdserver: integration test covering first commit in current term notification 2021-04-09 16:05:02 +02:00
wpedrak
3991a8c9fa etcdserver: replace forceVersionC with FirstCommitInTermNotify 2021-04-09 11:30:42 +02:00
wpedrak
3d485faac5 etcdserver: resend ReadIndex request on empty apply request 
Empty apply indicates first commit in current term. It is first time when follower is sure, that it's ReadIndex request can be processed.
 2021-04-09 11:30:42 +02:00
Piotr Tabor
bad0b4d513
Merge pull request #12823 from mtulio/chore/dash-var-refresh 
chore/dash-var-refresh: change default refresh to 2(time range)
 2021-04-08 15:14:53 +02:00
Marco Tulio R Braga
aeeecc06cf
fix/dash-var-refresh: add const and description 2021-04-08 10:12:41 -03:00
Piotr Tabor
63c25bf378
Merge pull request #12804 from ptabor/20210326-v3-publish 
server: v2store deprecation: Prepare to use publishV3 instead of publish V2.
 2021-04-08 09:22:22 +02:00
Piotr Tabor
e776efbb2a
Merge pull request #12828 from ptabor/20210404-embed-etcd 
embed: etcd.Close() is closing Errc() channel as well.
 2021-04-08 01:20:07 +02:00
Piotr Tabor
5da9cac193 embed: etcd.Close() is closing Errc() channel as well. 
Inspired by https://github.com/etcd-io/etcd/pull/9612 by purpleidea@.
 2021-04-08 01:19:13 +02:00
Piotr Tabor
cc0f812f51 server_test.go: Use context.Background() instead of TODO in tests. 
Suggected in: https://golang.org/pkg/context/#Background
 2021-04-08 01:15:16 +02:00
Piotr Tabor
d4a8093ea5 Switch release-test (upgrade test) to use etcd 3.4 (instead of 3.3) as upgrade-base. 2021-04-08 01:15:16 +02:00
Piotr Tabor
931af493cf Merge pull request #12830 from ptabor/20210405-split-pkg 
Split client/pkg as dedicated low-dependencies module for client
 2021-04-08 01:12:17 +02:00
Piotr Tabor
816d332d81
Merge pull request #12830 from ptabor/20210405-split-pkg 
Split client/pkg as dedicated low-dependencies module for client
 2021-04-08 00:48:41 +02:00
Piotr Tabor
24d2a9da70 test.sh: Turn off enforcement for minimal go.sum. 
It seems that generated go.sum differs between different go versions.
This was introducing noise in the tests.
 2021-04-07 20:26:10 +02:00
Piotr Tabor
e24e72c5c8
Merge pull request #12836 from chalin/chalin-contrib-lock-2021-04-06 
Contrib lock example
 2021-04-07 20:23:08 +02:00
Piotr Tabor
3bb7acc8cf Migrate dependencies pkg/foo -> client/pkg/foo 2021-04-07 00:38:47 +02:00
Piotr Tabor
7168409e6b
Merge pull request #12829 from garenchan/dev/etcdctl-lock 
etcdctl: lock return exit code of exec-command
 2021-04-06 23:46:20 +02:00
Patrice Chalin
2ba69de281 Contrib lock example 2021-04-06 15:21:01 -04:00
garenchan
c047ed593c etcdctl: lock return exit code of exec-command 
Sometimes we expect to get the exit code of the command being
executed.
 2021-04-06 14:34:31 +08:00
Piotr Tabor
0b81fdf418 Move pkg/* used by client to client/pkg. 2021-04-05 23:47:27 +02:00
Piotr Tabor
dfb03aba0b
Merge pull request #12826 from allenporter/reporting-bugs 
Fix "reporting bugs" link in documentation, which currently 404s
 2021-04-02 11:04:38 +02:00
Allen Porter
1906cfdc48 etcdserver: Fix "reporting bugs" link in documentation, which currently 404s 
Update the "reporting bugs" link to be the current correct url from the documentation, replacing a dash with an
underscore in the URL.
 2021-04-01 19:24:15 -07:00
Marco Braga
d2bc5343fb chore/dash-var-refresh: change default refresh to 2(time range) 2021-04-01 00:06:57 -03:00
Piotr Tabor
d18ac20080
Merge pull request #12766 from mrajashree/master 
Update cert generation Makefile to include configurable cert file names
 2021-03-30 12:59:41 +02:00
Piotr Tabor
a49c7ec4fd
Merge pull request #12817 from cwedgwood/3.4-changelog 
CHANGELOG-3.4 updates
 2021-03-30 12:53:02 +02:00
Chris Wedgwood
5d05be31aa CHANGELOG-3.4 updates 2021-03-29 11:38:25 -07:00
Sam Batschelet
7899f76dce
Merge pull request #12812 from hexfusion/bump-cl-3.2 
CHANGELOG: release 3.2
 2021-03-29 14:30:21 -04:00
Sam Batschelet
b04477a580 CHANGELOG: release 3.2 
Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
 2021-03-28 22:19:47 -04:00
Sam Batschelet
a40a6e9ad8
Merge pull request #12805 from ptabor/20210326-minor-test-fixes 
tests: logging & temp-dir fixes
 2021-03-28 19:14:13 -04:00
Sam Batschelet
52b81bb975
Merge pull request #12807 from ptabor/20210327-update-cmux-stable 
Update dependencies -> in particular cmux to stable 1.15.0
 2021-03-28 19:12:43 -04:00
Piotr Tabor
578ffd5783
Merge pull request #12790 from nvanbenschoten/reduce-raft-proto-size 
raft: disable XXX_NoUnkeyedLiteral, XXX_unrecognized, and XXX_sizecache fields in protos
 2021-03-27 21:08:50 +01:00
Piotr Tabor
f290ab2e60 Update dependecies: 
github.com/grpc-ecosystem/grpc-gateway v1.14.6 -> grpc-gateway v1.16.0
  golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e->v0.0.0-20210220033141-f8bda1e9f3ba
 2021-03-27 20:48:33 +01:00
Piotr Tabor
c49807f59e Update cmux to 1.5.0. 
Executed using:
```
./scripts/update_dep.sh github.com/soheilhy/cmux v0.1.5
```
 2021-03-27 11:18:13 +01:00
Piotr Tabor
7665bd764e scripts/update_dep.sh: Script to update selected versions of dependencies. 2021-03-27 11:17:56 +01:00
Piotr Tabor
55ccbe62a2 membership/cluster_test: Use zaptest logger. 2021-03-26 13:54:59 +01:00