Mirroristas/etcd
2
0
Fork 0
mirror of https://github.com/etcd-io/etcd.git synced 2024-09-27 06:25:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
17,268 Commits 15 Branches 491 Tags 148 MiB
Commit Graph

663 Commits

Author SHA1 Message Date
Marek Siarkowicz
86f68b9374 *: Add missing file licenses and Fix static analysis prevent skipping them in future 2021-06-29 12:52:02 +02:00
Lili Cosic
df696a7e76 go.mod: Bump etcd to 3.5.0 2021-06-22 12:25:39 +02:00
Piotr Tabor
404efd70f7 Upgrade of prometheous deps. 
For now we need to depend on unstable: prometheus/client_golang
 2021-06-03 16:07:01 +02:00
Piotr Tabor
edcfe575cf Update dependencies: 
- github.com/coreos/go-systemd/v22 v22.3.2
  - github.com/google/btree v1.0.1
  - github.com/json-iterator/go v1.1.11
  - github.com/mattn/go-runewidth v0.0.13
  - github.com/prometheus/client_golang v1.10.0
  - github.com/prometheus/common v0.26.0
  - github.com/sirupsen/logrus v1.8.1
 2021-06-03 13:03:01 +02:00
Piotr Tabor
b240625d21 Update bbolt to v1.3.6. 
`./scripts/update_dep.sh go.etcd.io/bbolt v1.3.6`
 2021-06-03 12:39:53 +02:00
Piotr Tabor
90cea7d4e7 Update zap to v1.17.0. 2021-06-03 12:37:43 +02:00
Ting Yuan
838be9c751 Omit the blank line 2021-05-21 21:25:49 +08:00
Ting Yuan
2d2a95c3fb Fix the format issue 2021-05-21 21:23:38 +08:00
Ting Yuan
1b700e5ed8 Fix the goroutine leak in StubServer 
---

(If this PR fixes a github issue, please add `Fixes #<xyz>`)

Fixes #13023

(or if this PR is one task of a github issue, please add `Master Issue: #<xyz>`
to link to the master issue)

Master Issue: #13023

*Motivation*

Currently, StubServer can stop itself (by Stop()) before it start its
grpc server (created by Start()). This race condition may lead to a
goroutine leak mentioned by #13023.

*Modifications*

This PR add a channel to force the Stop() started after Start()

*Verify this change*

Please pick either of following options.

- This change is already covered by existing tests, such as *(please describe tests)*.

Test/TestEtcdGrpcResolver
 2021-05-21 20:54:06 +08:00
Piotr Tabor
d8550deb7f Fix pkg/proxy tests such that they don't leek goroutines and do close transports. 2021-05-14 22:45:59 +02:00
Piotr Tabor
8981afb6f5 Fix unit tests logging config. 2021-05-14 22:45:57 +02:00
Piotr Tabor
00c6090110 Refactor common code from etcdctl (v2,v3) to pkg/corbautl. 
Preparation for etcdutl split.
 2021-05-14 14:16:53 +02:00
Lili Cosic
1a718a958e Add initial Tracing with OpenTelemetry 2021-05-10 10:44:40 +02:00
Piotr Tabor
2ad893b110 Integrate verification into e2e tests. 2021-04-29 11:51:24 +02:00
Piotr Tabor
3423a949c0 Update go for 3.5: 1.15 -> 1.16.(3). 
https://github.com/etcd-io/etcd/issues/12732
 2021-04-19 16:50:54 +02:00
Piotr Tabor
17b982382e Fix TestSnapshotV3RestoreMultiMemberAdd flakes (leaks) 
- most important: unix's socket transport should not keep idle
connections. For top-level Transport we close them using:
    f3c518025e/server/etcdserver/api/rafthttp/transport.go (L226)
    but currently we don't have access to close them witing the nest (unix) transport. Short idle deadline is good enough.

  - Use dialContext (instead of dial) to make sure context is passed down the stack
  - Make sure Context is cancelled as soon as the operation is done in pipeline
  - nit: use dedicated method to yeld goroutines.

Tested with:
```
d=$(date +"%Y%m%d_%H%M")
(cd tests && go test --timeout=60m ./integration/snapshot -run TestSnapshotV3RestoreMultiMemberAdd -v --count=180 2>&1 | tee log_${d}.log)
```

There were transports & cmux leaked:

```
   leak.go:118: Test appears to have leaked a Transport:
        internal/poll.runtime_pollWait(0x7f6c5c3784c8, 0x72, 0xffffffffffffffff)
        	/usr/lib/google-golang/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc003296298, 0x72, 0x0, 0x18, 0xffffffffffffffff)
        	/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:87 +0x45
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc003296280, 0xc0031f60a8, 0x18, 0x18, 0x0, 0x0, 0x0)
        	/usr/lib/google-golang/src/internal/poll/fd_unix.go:166 +0x1d5
        net.(*netFD).Read(0xc003296280, 0xc0031f60a8, 0x18, 0x18, 0x18, 0xc0009056e2, 0x203000)
        	/usr/lib/google-golang/src/net/fd_posix.go:55 +0x4f
        net.(*conn).Read(0xc000010258, 0xc0031f60a8, 0x18, 0x18, 0x0, 0x0, 0x0)
        	/usr/lib/google-golang/src/net/net.go:183 +0x91
        github.com/soheilhy/cmux.(*bufferedReader).Read(0xc0003d24e0, 0xc0031f60a8, 0x18, 0x18, 0xc0003d24d0, 0xc0009056e2, 0xc000278400)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/buffer.go:53 +0x12d
        github.com/soheilhy/cmux.hasHTTP2Preface(0x1367e20, 0xc0003d24e0, 0x7f6c5c699f40)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/matchers.go:195 +0x8a
        github.com/soheilhy/cmux.matchersToMatchWriters.func1(0x7f6c5c699f40, 0xc000010258, 0x1367e20, 0xc0003d24e0, 0xc000010258)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:128 +0x39
        github.com/soheilhy/cmux.(*cMux).serve(0xc003228690, 0x138c410, 0xc000010258, 0xc00327f740, 0xc0059ba860)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:192 +0x1e7
        created by github.com/soheilhy/cmux.(*cMux).Serve
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:179 +0x191

        internal/poll.runtime_pollWait(0x7f6c5c60f3f0, 0x72, 0xffffffffffffffff)
        	/usr/lib/google-golang/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc000d53018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
        	/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:87 +0x45
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc000d53000, 0xc000cfd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        	/usr/lib/google-golang/src/internal/poll/fd_unix.go:166 +0x1d5
        net.(*netFD).Read(0xc000d53000, 0xc000cfd000, 0x1000, 0x1000, 0x3, 0x3, 0x1000000000001)
        	/usr/lib/google-golang/src/net/fd_posix.go:55 +0x4f
        net.(*conn).Read(0xc00031a570, 0xc000cfd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        	/usr/lib/google-golang/src/net/net.go:183 +0x91
        net/http.(*persistConn).Read(0xc00093b320, 0xc000cfd000, 0x1000, 0x1000, 0x577750, 0x60, 0x0)
        	/usr/lib/google-golang/src/net/http/transport.go:1933 +0x77
        bufio.(*Reader).fill(0xc005702fc0)
        	/usr/lib/google-golang/src/bufio/bufio.go:101 +0x108
        bufio.(*Reader).Peek(0xc005702fc0, 0x1, 0xc00077c660, 0xc003b082a0, 0xc000d08de0, 0x5ae586, 0x11dd6c0)
        	/usr/lib/google-golang/src/bufio/bufio.go:139 +0x4f
        net/http.(*persistConn).readLoop(0xc00093b320)
        	/usr/lib/google-golang/src/net/http/transport.go:2094 +0x1a8
        created by net/http.(*Transport).dialConn
        	/usr/lib/google-golang/src/net/http/transport.go:1754 +0xdaa

        net/http.(*persistConn).writeLoop(0xc00093b320)
        	/usr/lib/google-golang/src/net/http/transport.go:2393 +0xf7
        created by net/http.(*Transport).dialConn
        	/usr/lib/google-golang/src/net/http/transport.go:1755 +0xdcf

        sync.runtime_Semacquire(0xc0059ba868)
        	/usr/lib/google-golang/src/runtime/sema.go:56 +0x45
        sync.(*WaitGroup).Wait(0xc0059ba860)
        	/usr/lib/google-golang/src/sync/waitgroup.go:130 +0x65
        github.com/soheilhy/cmux.(*cMux).Serve.func1(0xc003228690, 0xc0059ba860)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:158 +0x56
        github.com/soheilhy/cmux.(*cMux).Serve(0xc003228690, 0x13698c0, 0xc00377a0f0)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:173 +0x115
        go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers.func1(0xc0007cc360, 0x122b75f)
        	/home/ptab/corp/etcd/server/embed/etcd.go:518 +0x2b9
        go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers.func3(0xc00036d080, 0xc0059330a0)
        	/home/ptab/corp/etcd/server/embed/etcd.go:549 +0x182
        created by go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers
        	/home/ptab/corp/etcd/server/embed/etcd.go:543 +0x73a
--- FAIL: TestSnapshotV3RestoreMultiMemberAdd (17.74s)
```
 2021-04-16 20:17:28 +02:00
Piotr Tabor
eafbc8c57e Update zap logging dependency. 
In particular bring up zapgrpc V2 code:
89e382035d
https://pkg.go.dev/google.golang.org/grpc/grpclog#LoggerV2
 2021-04-14 12:15:48 +02:00
Piotr Tabor
3bb7acc8cf Migrate dependencies pkg/foo -> client/pkg/foo 2021-04-07 00:38:47 +02:00
Piotr Tabor
0b81fdf418 Move pkg/* used by client to client/pkg. 2021-04-05 23:47:27 +02:00
Piotr Tabor
d0962f10ae Adopt grpc_testing to grpc-1.36. 2021-03-24 23:31:03 +01:00
Piotr Tabor
a60676686b Update dep: grpc: 1.32.0 -> 1.36.0. 2021-03-24 22:27:55 +01:00
Piotr Tabor
45fb7b41d6 Update dep: github.com/golang/protobuf v1.3.5 ->  v1.5.1 
Thanks to https://go-review.googlesource.com/c/protobuf/+/300869/ its
feasible now.
 2021-03-24 22:27:12 +01:00
Piotr Tabor
18321a0c85
Merge pull request #12786 from ptabor/20210318-debug-env 
client: Bring back ETCD_CLIENT_DEBUG variable interpretation.
 2021-03-18 21:55:51 +01:00
Piotr Tabor
e2b5835e64 Fix go-tip: (1.17) fmt error: 
```
gofmt' started at Thu Mar 18 18:06:00 UTC 2021
transport/sockopt_unix.go
diff -u transport/sockopt_unix.go.orig transport/sockopt_unix.go
--- transport/sockopt_unix.go.orig	2021-03-18 18:06:01.667483834 +0000
+++ transport/sockopt_unix.go	2021-03-18 18:06:01.667483834 +0000
@@ -1,3 +1,4 @@
+//go:build !windows
 // +build !windows

 package transport
transport/sockopt_windows.go
diff -u transport/sockopt_windows.go.orig transport/sockopt_windows.go
--- transport/sockopt_windows.go.orig	2021-03-18 18:06:01.667483834 +0000
+++ transport/sockopt_windows.go	2021-03-18 18:06:01.667483834 +0000
@@ -1,3 +1,4 @@
+//go:build windows
 // +build windows
```
 2021-03-18 19:44:28 +01:00
Piotr Tabor
9312d1b077 clientv3: Bring back ETCD_CLIENT_DEBUG variable interpretation. 
env ETCD_CLIENT_DEBUG supports log levels (debug, info, warn, error, dpanic, panic, fatal).
Only when set, overrides application-wide grpc logging settings.
 2021-03-18 18:43:07 +01:00
wpedrak
dac6e37ea1 *: over 20 staticcheck fixes 2021-03-18 15:06:17 +01:00
Piotr Tabor
de67806175 mend 2021-03-14 13:35:47 +01:00
Piotr Tabor
67491a00ea e2e/expect: In case of sut process failure, print last 40lines of logs. 2021-03-13 23:41:29 +01:00
wpedrak
2c2456bf3d *: partial staticcheck fix 2021-03-10 14:13:38 +00:00
Piotr Tabor
b6c2e87a74 Testing: Integration tests does not check whether t==nil 2021-03-09 18:19:52 +01:00
Piotr Tabor
5ddabfdb24 tests: Make tests operate in /tmp director instead of src. 
Thanks to this, unix sockets should be not longer
created by integration tests in the the source code directory,
so potentially trigger IDE reloads and unnecessery load (and mess).
 2021-03-09 18:19:52 +01:00
Piotr Tabor
fb1d48e98e Integration tests: Use BeforeTest(t) instead of defer AfterTest(). 
Thanks to this change, a single method BeforeTest(t) can handle
before-test logic as well as registration of cleanup code
(t.Cleanup(func)).
 2021-03-09 18:19:51 +01:00
Piotr Tabor
efb584cc9b leak.go: Make the per-test AfterTest strictly wait for none of the unwanted rountines. 2021-03-09 18:19:51 +01:00
Gyuho Lee
94a371acd7
Merge pull request #12750 from ptabor/20210306-mlock 
--experimental-memory-mlock support
 2021-03-09 09:13:40 -08:00
Gyuho Lee
6fd85af641
Merge pull request #12702 from hexfusion/add-so 
*: add support for socket options
 2021-03-09 09:02:24 -08:00
Sam Batschelet
5b49fb41c8 fixup: add ListenerOptions 
Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
 2021-03-08 11:27:03 -05:00
Piotr Tabor
a46a358577 --experimental-memory-mlock support 
The flag protects etcd memory from being swapped out to disk.
This can happen in memory constrained systems where mmaped bbolt
area is natural condidate for swapping out.

This flag should provide better tail latency on the cost of higher RSS
ram usage. If the experiment is successful, the logic should get moved
into bbolt layer, where we can protect specific bbolt instances
(e.g. avoid protecting both during defragmentation).
 2021-03-07 12:32:57 +01:00
Piotr Tabor
792b7f57d3
Merge pull request #12747 from wilsonwang371/master 
pkg/wait: change list from single element to an array.
 2021-03-07 11:53:30 +01:00
Piotr Tabor
66cef61444 Detect leaked go-routines bases on pre-normalization syntax. 2021-03-04 22:28:44 +01:00
Wilson Wang
432fde88a9 pkg/wait: change list from single element to an array. 
We found wait lock contention when a large amount of write operations. Converting wait from single element to an array helps to improve the performance.

Fixes #12731

Signed-off-by: Wilson Wang <wilsonny371@gmail.com>
 2021-03-04 12:16:41 -08:00
Piotr Tabor
61fef348f8
Merge pull request #12742 from wilsonwang371/master 
debugutil: Remove extra space in trace handler route
 2021-03-04 08:55:44 +01:00
Wilson Wang
7fc447fb5c debugutil: Remove extra space in trace handler route 
debugutil: Remove extra space in trace handler route. To use trace, user needed to escape the extra space and the extra space needs to be removed.
 2021-03-03 16:01:04 -08:00
Piotr Tabor
102c198444
Merge pull request #12705 from astromechza/bm_etcd_peer_server_cert 
etcdmain: added peer-client-{client,key}-file parameters for supporting separate client and server certs when communicating between peers
 2021-03-02 09:03:35 +01:00
Piotr Tabor
5e10d12996 tests: Fixes a few recently spotted test-flakes 
```
Unexpected goroutines running after all test(s).
1 instances of:
syscall.Syscall(...)
	/usr/local/go/src/syscall/asm_linux_386.s:19 +0x5
syscall.Close(...)
	/usr/local/go/src/syscall/zsyscall_linux_386.go:285 +0x3d
internal/poll.(*FD).destroy(...)
	/usr/local/go/src/internal/poll/fd_unix.go:77 +0x30
internal/poll.(*FD).decref(...)
	/usr/local/go/src/internal/poll/fd_mutex.go:213 +0x38
internal/poll.(*FD).Close(...)
	/usr/local/go/src/internal/poll/fd_unix.go:99 +0x43
net.(*netFD).Close(...)
	/usr/local/go/src/net/fd_posix.go:37 +0x49
FAIL	go.etcd.io/etcd/tests/v3/integration/client	0.039s
```

```
--- FAIL: TestServer_TCP_Secure_DelayTx (0.20s)
    server_test.go:110: took 128.026085ms with no latency
    server_test.go:125: took 62.980988ms with latency 50ms��5ms
    server_test.go:133: expected took1 128.026085ms < took2 62.980988ms (with latency)
```

https://github.com/etcd-io/etcd/issues/12372
 2021-03-01 18:07:38 +01:00
Ben Meier
3d44f5bf80
*: added client-{client,key}-file parameters for supporting separate client and server certs when communicating between peers 
In some environments, the CA is not able to sign certificates with both
'client auth' and 'server auth' extended usage parameters and so an operator
needs to be able to set a seperate client certificate to use when making
requests which is different to the certificate used for accepting requests.
This applies to both proxy and etcd member mode and is available as both a CLI
 flag and config file field for peer TLS.

Signed-off-by: Ben Meier <ben.meier@oracle.com>
 2021-02-28 14:37:56 +00:00
Gyuho Lee
cb040801f9
Merge pull request #12730 from ptabor/20200208-client-no-logger 
clientv3: Cleaning up dead logger code
 2021-02-26 10:44:42 -08:00
Piotr Tabor
a7f340216d Reformat code according to 'gotip' rules. 
In practices adds annotations in the new syntax:
```
+//go:build !linux
 // +build !linux
```

Fixes failing gotip PASSES='fmt' check:
https://travis-ci.com/github/etcd-io/etcd/jobs/486453806
 2021-02-26 10:14:46 +01:00
Piotr Tabor
54b87505a3 Remove dead legacy logger code. 2021-02-26 09:13:09 +01:00
Mohammad Aziz
252dcc9bdb
Fix typo in DialJournal 2021-02-24 22:23:27 +05:30
Piotr Tabor
1a9c81abda Update grpc dependency to 1.32. 
Simplify grpc testing infrastructure to align with upstream changes.
 2021-02-23 11:31:50 +01:00