Mirroristas/etcd
2
0
Fork 0
mirror of https://github.com/etcd-io/etcd.git synced 2024-09-27 06:25:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
17,223 Commits 15 Branches 491 Tags 148 MiB
Commit Graph

66 Commits

Author SHA1 Message Date
Marek Siarkowicz
7358362c99 server: Extract hasher to separate interface 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2022-09-07 15:11:55 +02:00
Marek Siarkowicz
631107285a server: Remove duplicated compaction revision 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2022-09-07 15:11:55 +02:00
Marek Siarkowicz
a3f609d742 server: Return revision range that hash was calcualted for 
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
 2022-09-07 15:11:55 +02:00
Benjamin Wang
437f3778d0 Add flag --max-concurrent-streams to set the max concurrent stream each client can open at a time 
Also refer to https://github.com/etcd-io/etcd/pull/14169#discussion_r917154243

Signed-off-by: Benjamin Wang <wachao@vmware.com>
 2022-07-13 14:18:15 +08:00
ahrtr
bc5307de95 support linearizable renew lease 
When etcdserver receives a LeaseRenew request, it may be still in
progress of processing the LeaseGrantRequest on exact the same
leaseID. Accordingly it may return a TTL=0 to client due to the
leaseID not found error. So the leader should wait for the appliedID
to be available before processing client requests.
 2022-04-12 14:12:45 +08:00
ahrtr
66c7aab4d3 fix the data inconsistency issue by adding a txPostLockHook into the backend 
Previously the SetConsistentIndex() is called during the apply workflow,
but it's outside the db transaction. If a commit happens between SetConsistentIndex
and the following apply workflow, and etcd crashes for whatever reason right
after the commit, then etcd commits an incomplete transaction to db.
Eventually etcd runs into the data inconsistency issue.

In this commit, we move the SetConsistentIndex into a txPostLockHook, so
it will be executed inside the transaction lock.
 2022-04-08 20:37:34 +08:00
ahrtr
7db1051774 enhance health check endpoint to support serializable request 2022-02-17 15:03:22 +08:00
Hitoshi Mitake
dec6f72d68 *: implement a retry logic for auth old revision in the client 2021-11-15 00:09:16 +09:00
Chao Chen
7d44a7cd6e server/etcdserver/api/etcdhttp: exclude the same alarm type activated by multiple peers 2021-11-12 14:21:14 -08:00
Marek Siarkowicz
58d2b12a50 client: Add grpc authority header integration tests 2021-09-30 12:15:32 +02:00
Marek Siarkowicz
e68c7ab4bc server: Ensure that adding and removing members handle storev2 and backend out of sync 2021-09-15 14:36:41 +02:00
tangcong
dfd2fea4c5 fix health endpoint not usable when authentication is enabled 2021-07-30 07:53:40 +08:00
J. David Lowe
e27effa250 etcdserver: don't attempt to grant nil permission to a role 
Prevent etcd from crashing when given a bad grant payload, e.g.:

$ curl -d '{"name": "foo"}' http://localhost:2379/v3/auth/role/add
{"header":{"cluster_id":"14841639068965178418", ...
$ curl -d '{"name": "foo"}' http://localhost:2379/v3/auth/role/grant
curl: (52) Empty reply from server

Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
 2021-06-04 15:34:22 -07:00
Piotr Tabor
3f13d3a2d5 integration.BeforeTest can be run without leak-detection. 2021-05-28 10:01:36 +02:00
Piotr Tabor
e6baf6d751 Represent bucket as object instead of []byte name. 
Thanks to this change:
  - all the maps bucket -> buffer are indexed by int's instead of
string. No need to do: byte[] -> string -> hash conversion on each
access.
  - buckets are strongly typed in backend/mvcc API.
 2021-05-25 09:22:25 +02:00
Piotr Tabor
e44fb40be5
Merge pull request #12962 from ptabor/20210513-write-conf-state 
Save raftpb.ConfState in the backend.
 2021-05-13 19:22:28 +02:00
Gyuho Lee
e2d67f2e3b
Merge pull request #12956 from gyuho/rename-to-main 
*: rename "master" branch references to "main" in source code
 2021-05-13 08:26:33 -07:00
Piotr Tabor
865df75714 Save raftpb.ConfState in the backend. 
This makes (bbolt) backend a full feature snapshot in term of WAL/raft,
i.e. carries:
  - commit : (applied_index)
  - confState

Benefits:
  - Backend will be a sufficient point in time definition sufficient to
start replaying WAL. We have applied_index & confState in consistent
state.
  - In case of emergency a backend state can be used for recovery
 2021-05-13 14:29:36 +02:00
Piotr Tabor
3cb1ba4b2b
Merge pull request #12954 from serathius/logger-new-ctx-client 
client: Add logger argument to NewCtxClient
 2021-05-13 09:03:38 +02:00
Gyuho Lee
77c8033739 server: rename "master" branch references 
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
 2021-05-12 10:37:35 -07:00
Piotr Tabor
f3b4a3e578 Detecting whether v2store is "empty" (metadata only). 2021-05-12 18:09:34 +02:00
Marek Siarkowicz
1189ee3f3d client: Add logger argument to NewCtxClient 2021-05-12 16:40:55 +02:00
Lili Cosic
1a718a958e Add initial Tracing with OpenTelemetry 2021-05-10 10:44:40 +02:00
Piotr Tabor
aeb9b5fc73
Merge pull request #12855 from ptabor/20210409-backend-hooks 
(no)StoreV2 (Part 4): Backend hooks:  precommit updates consistency_index
 2021-05-08 09:34:31 +02:00
tangcong
b32bc914ff
learner support snapshot RPC (#12890) 
* learner support snapshot RPC

* CHANGELOG: update for 12890
 2021-05-04 13:26:33 -07:00
Piotr Tabor
2dbecea5b2 Simplify KVStore interaction with cindex thanks to hooks. 2021-05-04 18:21:23 +02:00
Piotr Tabor
a3fc535a5a
Merge pull request #12914 from ptabor/20210430-read-membership 
No-storeV2: Read membership information from the backend (Part5)
 2021-05-04 10:15:01 +02:00
Piotr Tabor
ffea1537d4 ClientV3 tests use integration.NewClient that configures proper logger. 2021-04-29 18:18:34 +02:00
Piotr Tabor
205a1a442a Read membership information from the backend. 2021-04-29 13:45:45 +02:00
Piotr Tabor
067521981e v2 etcdctl backup: producing consistent state of membership 2021-04-27 19:34:34 +02:00
Piotr Tabor
a70386a1a4 Simplify membership interface: Does not pass the 'unused' token. 2021-04-27 17:17:31 +02:00
Piotr Tabor
7ae3d25f91 Membership: Add additional methods to trim/manage membership data in backend. 2021-04-27 17:17:31 +02:00
Piotr Tabor
768da490ed sever: v2store deprecation: Fix etcdctl snapshot restore to restore 
correct 'backend' (bbolt) context in aspect of membership.

Prior to this change the 'restored' backend used to still contain:
  - old memberid (mvcc deletion used, why the membership is in bolt
bucket, but not mvcc part):
    ```
	mvs := mvcc.NewStore(s.lg, be, lessor, ci, mvcc.StoreConfig{CompactionBatchLimit: math.MaxInt32})
	defer mvs.Close()
	txn := mvs.Write(traceutil.TODO())
	btx := be.BatchTx()
	del := func(k, v []byte) error {
		txn.DeleteRange(k, nil)
		return nil
	}

	// delete stored members from old cluster since using new members
	btx.UnsafeForEach([]byte("members"), del)
    ```
  - didn't get new members added.
 2021-04-27 17:17:30 +02:00
Piotr Tabor
9a4b2bdccc Errors: context cancelled or context deadline exceeded are exposed as codes.Canceled, codes.DeadlineExceeded instead of 'codes.Unknown' 2021-04-22 14:35:24 +02:00
Piotr Tabor
ea287dd9f8
Merge pull request #12854 from ptabor/20210410-shouldApplyV3 
(no)StoreV2 (Part 3): Applying consistency fix: ClusterVersionSet (and co) might get not applied on v2store
 2021-04-21 09:31:38 +02:00
Sam Batschelet
0f2c940f64
Merge pull request #12880 from chaochn47/exclude_alarms_from_health_check 
etcdhttp/metrics.go: exclude alarms from health check conditionally with `?exclude=NOSPACE`
 2021-04-20 21:18:15 -04:00
Chao Chen
140ea4fa29 etcdhttp/metrics.go: exclude alarms from health check conditionally with ?exclude=NOSPACE 2021-04-20 13:17:09 -07:00
Piotr Tabor
17b982382e Fix TestSnapshotV3RestoreMultiMemberAdd flakes (leaks) 
- most important: unix's socket transport should not keep idle
connections. For top-level Transport we close them using:
    f3c518025e/server/etcdserver/api/rafthttp/transport.go (L226)
    but currently we don't have access to close them witing the nest (unix) transport. Short idle deadline is good enough.

  - Use dialContext (instead of dial) to make sure context is passed down the stack
  - Make sure Context is cancelled as soon as the operation is done in pipeline
  - nit: use dedicated method to yeld goroutines.

Tested with:
```
d=$(date +"%Y%m%d_%H%M")
(cd tests && go test --timeout=60m ./integration/snapshot -run TestSnapshotV3RestoreMultiMemberAdd -v --count=180 2>&1 | tee log_${d}.log)
```

There were transports & cmux leaked:

```
   leak.go:118: Test appears to have leaked a Transport:
        internal/poll.runtime_pollWait(0x7f6c5c3784c8, 0x72, 0xffffffffffffffff)
        	/usr/lib/google-golang/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc003296298, 0x72, 0x0, 0x18, 0xffffffffffffffff)
        	/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:87 +0x45
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc003296280, 0xc0031f60a8, 0x18, 0x18, 0x0, 0x0, 0x0)
        	/usr/lib/google-golang/src/internal/poll/fd_unix.go:166 +0x1d5
        net.(*netFD).Read(0xc003296280, 0xc0031f60a8, 0x18, 0x18, 0x18, 0xc0009056e2, 0x203000)
        	/usr/lib/google-golang/src/net/fd_posix.go:55 +0x4f
        net.(*conn).Read(0xc000010258, 0xc0031f60a8, 0x18, 0x18, 0x0, 0x0, 0x0)
        	/usr/lib/google-golang/src/net/net.go:183 +0x91
        github.com/soheilhy/cmux.(*bufferedReader).Read(0xc0003d24e0, 0xc0031f60a8, 0x18, 0x18, 0xc0003d24d0, 0xc0009056e2, 0xc000278400)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/buffer.go:53 +0x12d
        github.com/soheilhy/cmux.hasHTTP2Preface(0x1367e20, 0xc0003d24e0, 0x7f6c5c699f40)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/matchers.go:195 +0x8a
        github.com/soheilhy/cmux.matchersToMatchWriters.func1(0x7f6c5c699f40, 0xc000010258, 0x1367e20, 0xc0003d24e0, 0xc000010258)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:128 +0x39
        github.com/soheilhy/cmux.(*cMux).serve(0xc003228690, 0x138c410, 0xc000010258, 0xc00327f740, 0xc0059ba860)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:192 +0x1e7
        created by github.com/soheilhy/cmux.(*cMux).Serve
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:179 +0x191

        internal/poll.runtime_pollWait(0x7f6c5c60f3f0, 0x72, 0xffffffffffffffff)
        	/usr/lib/google-golang/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc000d53018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
        	/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:87 +0x45
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc000d53000, 0xc000cfd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        	/usr/lib/google-golang/src/internal/poll/fd_unix.go:166 +0x1d5
        net.(*netFD).Read(0xc000d53000, 0xc000cfd000, 0x1000, 0x1000, 0x3, 0x3, 0x1000000000001)
        	/usr/lib/google-golang/src/net/fd_posix.go:55 +0x4f
        net.(*conn).Read(0xc00031a570, 0xc000cfd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        	/usr/lib/google-golang/src/net/net.go:183 +0x91
        net/http.(*persistConn).Read(0xc00093b320, 0xc000cfd000, 0x1000, 0x1000, 0x577750, 0x60, 0x0)
        	/usr/lib/google-golang/src/net/http/transport.go:1933 +0x77
        bufio.(*Reader).fill(0xc005702fc0)
        	/usr/lib/google-golang/src/bufio/bufio.go:101 +0x108
        bufio.(*Reader).Peek(0xc005702fc0, 0x1, 0xc00077c660, 0xc003b082a0, 0xc000d08de0, 0x5ae586, 0x11dd6c0)
        	/usr/lib/google-golang/src/bufio/bufio.go:139 +0x4f
        net/http.(*persistConn).readLoop(0xc00093b320)
        	/usr/lib/google-golang/src/net/http/transport.go:2094 +0x1a8
        created by net/http.(*Transport).dialConn
        	/usr/lib/google-golang/src/net/http/transport.go:1754 +0xdaa

        net/http.(*persistConn).writeLoop(0xc00093b320)
        	/usr/lib/google-golang/src/net/http/transport.go:2393 +0xf7
        created by net/http.(*Transport).dialConn
        	/usr/lib/google-golang/src/net/http/transport.go:1755 +0xdcf

        sync.runtime_Semacquire(0xc0059ba868)
        	/usr/lib/google-golang/src/runtime/sema.go:56 +0x45
        sync.(*WaitGroup).Wait(0xc0059ba860)
        	/usr/lib/google-golang/src/sync/waitgroup.go:130 +0x65
        github.com/soheilhy/cmux.(*cMux).Serve.func1(0xc003228690, 0xc0059ba860)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:158 +0x56
        github.com/soheilhy/cmux.(*cMux).Serve(0xc003228690, 0x13698c0, 0xc00377a0f0)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:173 +0x115
        go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers.func1(0xc0007cc360, 0x122b75f)
        	/home/ptab/corp/etcd/server/embed/etcd.go:518 +0x2b9
        go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers.func3(0xc00036d080, 0xc0059330a0)
        	/home/ptab/corp/etcd/server/embed/etcd.go:549 +0x182
        created by go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers
        	/home/ptab/corp/etcd/server/embed/etcd.go:543 +0x73a
--- FAIL: TestSnapshotV3RestoreMultiMemberAdd (17.74s)
```
 2021-04-16 20:17:28 +02:00
Piotr Tabor
d69e46ea47 Make ShouldApplyV3 an enum - not bool 2021-04-13 23:01:03 +02:00
Piotr Tabor
b1c04ce043 Applying consistency fix: ClusterVersionSet (and co) might get no applied on v2store 
ClusterVersionSet, ClusterMemberAttrSet, DowngradeInfoSet functions are
writing both to V2store and backend. Prior this CL there were
in a branch not executed if shouldApplyV3 was false,
e.g. during restore when Backend is up-to-date (has high
consistency-index) while v2store requires replay from WAL log.

The most serious consequence of this bug was that v2store after restore
could have different index (revision) than the same exact store before restore,
so potentially different content between replicas.

Also this change is supressing double-applying of Membership
(ClusterConfig) changes on Backend (store v3) - that lackilly are not
part of MVCC/KeyValue store, so they didn't caused Revisions to be
bumped.

Inspired by jingyih@ comment:
https://github.com/etcd-io/etcd/pull/12820#issuecomment-815299406
 2021-04-12 09:43:48 +02:00
Piotr Tabor
3bb7acc8cf Migrate dependencies pkg/foo -> client/pkg/foo 2021-04-07 00:38:47 +02:00
Piotr Tabor
55ccbe62a2 membership/cluster_test: Use zaptest logger. 2021-03-26 13:54:59 +01:00
Piotr Tabor
7d7a9c6f23 codec.go: should use google runtime 
golang-proto runtime can deal with both: gogo & golang generated
protobufs. It does not work vice-versa with protobuf-1.5.1.
 2021-03-24 22:06:47 +01:00
Joel Smith
19f7c6ef3e *: Update gogo/protobuf to v1.3.2, rerun ./scripts/genproto.sh 
While it appears that etcd is not vulnerable to CVE-2021-3121,
it is a good idea to update to the new generator so that new
vulnerable code isn't generated in any future APIs. Also, this
lays the issue to rest of whether there is any issue with
etcd and CVE-2021-3121.
 2021-03-23 11:48:06 -06:00
Gyuho Lee
3ead91ca3e
Merge pull request #12739 from LeoYang90/optimization_watch_prevkv 
create event do not need prevkv range
 2021-03-10 09:48:42 -08:00
Piotr Tabor
fb1d48e98e Integration tests: Use BeforeTest(t) instead of defer AfterTest(). 
Thanks to this change, a single method BeforeTest(t) can handle
before-test logic as well as registration of cleanup code
(t.Cleanup(func)).
 2021-03-09 18:19:51 +01:00
Gyuho Lee
6fd85af641
Merge pull request #12702 from hexfusion/add-so 
*: add support for socket options
 2021-03-09 09:02:24 -08:00
Sam Batschelet
5b49fb41c8 fixup: add ListenerOptions 
Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
 2021-03-08 11:27:03 -05:00
leoyang.yl
d70f35f8d1 create event do not need prevkv range 2021-03-02 17:43:24 +08:00
Gyuho Lee
3d7aac948b
Merge pull request #12196 from ironcladlou/metrics-watch-error-fix 
etcdserver: fix incorrect metrics generated when clients cancel watches
 2021-02-19 12:46:49 -08:00