126 Commits

Author SHA1 Message Date
Piotr Tabor
28f2b07623 *: Update references to code moved to the api/ dir.
Follow up to file-moves done in the previous commit.

The commit contains purely mechanical consequences of execution (apart
of scripts/genproto.sh):

  % find ./ -name '*.go'  | xargs sed --follow-symlinks -i 's|v3/etcdserver/api/v3rpc/rpctypes|v3/api/v3rpc/rpctypes|g'
  % find ./ -name '*.go'  | xargs sed --follow-symlinks -i 's|v3/version|v3/api/version|g'
  % find ./ -name '*.go'  | xargs sed --follow-symlinks -i 's|v3/mvcc/mvccpb|v3/api/mvccpb|g'
  % find ./ -name '*.go'  | xargs sed --follow-symlinks -i 's|v3/etcdserver/etcdserverpb|v3/api/etcdserverpb|g'
  % find ./ -name '*.go'  | xargs sed --follow-symlinks -i 's|v3/etcdserver/api/membership/membershippb|v3/api/membershippb|g'
  % find ./ -name '*.go'  | xargs sed --follow-symlinks -i 's|v3/auth/authpb|v3/api/authpb|g'

  % find ./ -name '*.proto' -o -name '*.md'  | xargs -L 1 sed --follow-symlinks -i 's|/mvcc/mvccpb/kv.proto|/api/mvccpb/kv.proto|g'
  % find ./ -name '*.proto' -o -name '*.md'  | xargs -L 1 sed --follow-symlinks -i 's|/auth/authpb/auth.proto|/api/authpb/auth.proto|g'
  % find ./ -name '*.proto' -o -name '*.md'  | xargs -L 1 sed --follow-symlinks -i 's|/etcdserver/api/membership/membershippb/membership.proto|/api/membershippb/membership.proto|g'

  I also modified manually paths in scripts/genproto.sh.

  % go fmt ./...
2020-10-06 11:56:16 +02:00
Piotr Tabor
97820f1c6e integration: Fix flakes of TestV3WatchRestoreSnapshotUnsync
```
```

The flakes manifested as:
```
--- FAIL: TestV3WatchRestoreSnapshotUnsync (3.59s)
    v3_watch_restore_test.go:82: inflight snapshot sends expected 0 or 1, got ""
FAIL
coverage: 55.2% of statements
FAIL	go.etcd.io/etcd/v3/integration	3.646s
FAIL
```

The root reason is that all the SnapMsg processing happends on both ends
(leader, follower) assynchronously in goroutines, e.g. on Fifo schedule
within EtcdServer.run, so when we observe through metrics, we don't
know whether it finised (or even got started).

Idally we should have EtcdServer.Drain() call that exits when the server
processed or internal 'queues' and is idle.
2020-10-03 19:39:08 +02:00
Piotr Tabor
31426b0041 clientv3/ordering: Split mocked part of the test from integration-level test. 2020-09-26 08:44:58 +02:00
Piotr Tabor
3f449a8548 *: 'go test --short ./...' runs all UNIT tests.
Marked all 'integrational, e2e' as skipped in the --short mode.

Thanks to this we will be able to significantly simplify ./test script.

The run currently takes ~23s.
With (follow up) move of ~clientv3/snapshot to integration tests (as
part of modularization), we can expect this to fall to 5-10s.

```
% time go test --short ./... --count=1
ok  	go.etcd.io/etcd/v3	0.098s
?   	go.etcd.io/etcd/v3/Documentation/learning/lock/client	[no test files]
?   	go.etcd.io/etcd/v3/Documentation/learning/lock/storage	[no test files]
ok  	go.etcd.io/etcd/v3/auth	0.724s
?   	go.etcd.io/etcd/v3/auth/authpb	[no test files]
ok  	go.etcd.io/etcd/v3/client	0.166s
ok  	go.etcd.io/etcd/v3/client/integration	0.166s
ok  	go.etcd.io/etcd/v3/clientv3	3.219s
ok  	go.etcd.io/etcd/v3/clientv3/balancer	1.102s
?   	go.etcd.io/etcd/v3/clientv3/balancer/connectivity	[no test files]
?   	go.etcd.io/etcd/v3/clientv3/balancer/picker	[no test files]
?   	go.etcd.io/etcd/v3/clientv3/balancer/resolver/endpoint	[no test files]
ok  	go.etcd.io/etcd/v3/clientv3/clientv3util	0.096s [no tests to run]
ok  	go.etcd.io/etcd/v3/clientv3/concurrency	3.323s
?   	go.etcd.io/etcd/v3/clientv3/credentials	[no test files]
ok  	go.etcd.io/etcd/v3/clientv3/integration	0.131s
?   	go.etcd.io/etcd/v3/clientv3/leasing	[no test files]
?   	go.etcd.io/etcd/v3/clientv3/mirror	[no test files]
ok  	go.etcd.io/etcd/v3/clientv3/namespace	0.041s
ok  	go.etcd.io/etcd/v3/clientv3/naming	0.115s
ok  	go.etcd.io/etcd/v3/clientv3/ordering	0.121s
ok  	go.etcd.io/etcd/v3/clientv3/snapshot	19.325s
ok  	go.etcd.io/etcd/v3/clientv3/yaml	0.090s
ok  	go.etcd.io/etcd/v3/contrib/raftexample	7.572s
?   	go.etcd.io/etcd/v3/contrib/recipes	[no test files]
ok  	go.etcd.io/etcd/v3/embed	0.282s
ok  	go.etcd.io/etcd/v3/etcdctl	0.054s
?   	go.etcd.io/etcd/v3/etcdctl/ctlv2	[no test files]
ok  	go.etcd.io/etcd/v3/etcdctl/ctlv2/command	0.117s
?   	go.etcd.io/etcd/v3/etcdctl/ctlv3	[no test files]
ok  	go.etcd.io/etcd/v3/etcdctl/ctlv3/command	0.070s
ok  	go.etcd.io/etcd/v3/etcdmain	0.172s
ok  	go.etcd.io/etcd/v3/etcdserver	1.698s
?   	go.etcd.io/etcd/v3/etcdserver/api	[no test files]
ok  	go.etcd.io/etcd/v3/etcdserver/api/etcdhttp	0.075s
ok  	go.etcd.io/etcd/v3/etcdserver/api/membership	0.104s
?   	go.etcd.io/etcd/v3/etcdserver/api/membership/membershippb	[no test files]
ok  	go.etcd.io/etcd/v3/etcdserver/api/rafthttp	0.181s
ok  	go.etcd.io/etcd/v3/etcdserver/api/snap	0.078s
?   	go.etcd.io/etcd/v3/etcdserver/api/snap/snappb	[no test files]
ok  	go.etcd.io/etcd/v3/etcdserver/api/v2auth	0.142s
ok  	go.etcd.io/etcd/v3/etcdserver/api/v2discovery	0.035s
ok  	go.etcd.io/etcd/v3/etcdserver/api/v2error	0.043s
ok  	go.etcd.io/etcd/v3/etcdserver/api/v2http	0.070s
ok  	go.etcd.io/etcd/v3/etcdserver/api/v2http/httptypes	0.031s
?   	go.etcd.io/etcd/v3/etcdserver/api/v2stats	[no test files]
ok  	go.etcd.io/etcd/v3/etcdserver/api/v2store	0.645s
ok  	go.etcd.io/etcd/v3/etcdserver/api/v2v3	0.218s
?   	go.etcd.io/etcd/v3/etcdserver/api/v3alarm	[no test files]
?   	go.etcd.io/etcd/v3/etcdserver/api/v3client	[no test files]
ok  	go.etcd.io/etcd/v3/etcdserver/api/v3compactor	1.765s
?   	go.etcd.io/etcd/v3/etcdserver/api/v3election	[no test files]
?   	go.etcd.io/etcd/v3/etcdserver/api/v3election/v3electionpb	[no test files]
?   	go.etcd.io/etcd/v3/etcdserver/api/v3election/v3electionpb/gw	[no test files]
?   	go.etcd.io/etcd/v3/etcdserver/api/v3lock	[no test files]
?   	go.etcd.io/etcd/v3/etcdserver/api/v3lock/v3lockpb	[no test files]
?   	go.etcd.io/etcd/v3/etcdserver/api/v3lock/v3lockpb/gw	[no test files]
ok  	go.etcd.io/etcd/v3/etcdserver/api/v3rpc	0.091s
ok  	go.etcd.io/etcd/v3/etcdserver/api/v3rpc/rpctypes	0.012s
ok  	go.etcd.io/etcd/v3/etcdserver/cindex	0.054s
ok  	go.etcd.io/etcd/v3/etcdserver/etcdserverpb	0.039s
?   	go.etcd.io/etcd/v3/etcdserver/etcdserverpb/gw	[no test files]
ok  	go.etcd.io/etcd/v3/functional/agent	0.094s
?   	go.etcd.io/etcd/v3/functional/cmd/etcd-agent	[no test files]
?   	go.etcd.io/etcd/v3/functional/cmd/etcd-proxy	[no test files]
?   	go.etcd.io/etcd/v3/functional/cmd/etcd-runner	[no test files]
?   	go.etcd.io/etcd/v3/functional/cmd/etcd-tester	[no test files]
ok  	go.etcd.io/etcd/v3/functional/rpcpb	0.060s
?   	go.etcd.io/etcd/v3/functional/runner	[no test files]
ok  	go.etcd.io/etcd/v3/functional/tester	0.079s
ok  	go.etcd.io/etcd/v3/integration	0.684s
ok  	go.etcd.io/etcd/v3/integration/embed	0.101s
ok  	go.etcd.io/etcd/v3/lease	3.455s
ok  	go.etcd.io/etcd/v3/lease/leasehttp	2.185s
?   	go.etcd.io/etcd/v3/lease/leasepb	[no test files]
ok  	go.etcd.io/etcd/v3/mvcc	7.246s
ok  	go.etcd.io/etcd/v3/mvcc/backend	0.354s
?   	go.etcd.io/etcd/v3/mvcc/mvccpb	[no test files]
ok  	go.etcd.io/etcd/v3/pkg/adt	0.025s
?   	go.etcd.io/etcd/v3/pkg/contention	[no test files]
?   	go.etcd.io/etcd/v3/pkg/cpuutil	[no test files]
ok  	go.etcd.io/etcd/v3/pkg/crc	0.008s
?   	go.etcd.io/etcd/v3/pkg/debugutil	[no test files]
ok  	go.etcd.io/etcd/v3/pkg/expect	0.015s
ok  	go.etcd.io/etcd/v3/pkg/fileutil	0.268s
ok  	go.etcd.io/etcd/v3/pkg/flags	0.021s
ok  	go.etcd.io/etcd/v3/pkg/httputil	0.020s
ok  	go.etcd.io/etcd/v3/pkg/idutil	0.008s
ok  	go.etcd.io/etcd/v3/pkg/ioutil	0.025s
ok  	go.etcd.io/etcd/v3/pkg/logutil	0.047s
?   	go.etcd.io/etcd/v3/pkg/mock/mockserver	[no test files]
?   	go.etcd.io/etcd/v3/pkg/mock/mockstorage	[no test files]
?   	go.etcd.io/etcd/v3/pkg/mock/mockstore	[no test files]
?   	go.etcd.io/etcd/v3/pkg/mock/mockwait	[no test files]
ok  	go.etcd.io/etcd/v3/pkg/netutil	1.024s
ok  	go.etcd.io/etcd/v3/pkg/osutil	0.021s
ok  	go.etcd.io/etcd/v3/pkg/pathutil	0.008s
ok  	go.etcd.io/etcd/v3/pkg/pbutil	0.008s
ok  	go.etcd.io/etcd/v3/pkg/proxy	4.081s
ok  	go.etcd.io/etcd/v3/pkg/report	0.008s
?   	go.etcd.io/etcd/v3/pkg/runtime	[no test files]
ok  	go.etcd.io/etcd/v3/pkg/schedule	0.009s
ok  	go.etcd.io/etcd/v3/pkg/srv	0.019s
ok  	go.etcd.io/etcd/v3/pkg/stringutil	0.008s
?   	go.etcd.io/etcd/v3/pkg/systemd	[no test files]
ok  	go.etcd.io/etcd/v3/pkg/testutil	0.023s
ok  	go.etcd.io/etcd/v3/pkg/tlsutil	3.965s
ok  	go.etcd.io/etcd/v3/pkg/traceutil	0.034s
ok  	go.etcd.io/etcd/v3/pkg/transport	0.532s
ok  	go.etcd.io/etcd/v3/pkg/types	0.028s
ok  	go.etcd.io/etcd/v3/pkg/wait	0.023s
ok  	go.etcd.io/etcd/v3/proxy/grpcproxy	0.101s
?   	go.etcd.io/etcd/v3/proxy/grpcproxy/adapter	[no test files]
?   	go.etcd.io/etcd/v3/proxy/grpcproxy/cache	[no test files]
ok  	go.etcd.io/etcd/v3/proxy/httpproxy	0.044s
ok  	go.etcd.io/etcd/v3/proxy/tcpproxy	0.047s
ok  	go.etcd.io/etcd/v3/raft	0.312s
ok  	go.etcd.io/etcd/v3/raft/confchange	0.183s
ok  	go.etcd.io/etcd/v3/raft/quorum	0.316s
ok  	go.etcd.io/etcd/v3/raft/raftpb	0.024s
ok  	go.etcd.io/etcd/v3/raft/rafttest	0.640s
ok  	go.etcd.io/etcd/v3/raft/tracker	0.026s
ok  	go.etcd.io/etcd/v3/tests/e2e	0.077s
?   	go.etcd.io/etcd/v3/tools/benchmark	[no test files]
?   	go.etcd.io/etcd/v3/tools/benchmark/cmd	[no test files]
?   	go.etcd.io/etcd/v3/tools/etcd-dump-db	[no test files]
ok  	go.etcd.io/etcd/v3/tools/etcd-dump-logs	0.088s
?   	go.etcd.io/etcd/v3/tools/etcd-dump-metrics	[no test files]
?   	go.etcd.io/etcd/v3/tools/local-tester/bridge	[no test files]
?   	go.etcd.io/etcd/v3/version	[no test files]
ok  	go.etcd.io/etcd/v3/wal	1.517s
?   	go.etcd.io/etcd/v3/wal/walpb	[no test files]
go test --short ./... --count=1  76.12s user 12.57s system 375% cpu 23.635 total
```
2020-09-16 13:21:46 +02:00
jingyih
73817b53fd integration: add flag WatchProgressNotifyInterval in integration test 2020-09-07 08:32:54 -07:00
Brandon Philips
96cce208c2 go.mod: use go.etcd.io/etcd/v3 versioning
This change makes the etcd package compatible with the existing Go
ecosystem for module versioning.

Used this tool to update package imports:
  https://github.com/KSubedi/gomove
2020-04-28 00:57:35 +00:00
Gyuho Lee
33907477dd *: add "etcd_server_client_requests_total", tests
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2020-03-17 12:16:11 -07:00
Fansong Zeng
d70251835d integration: gracefully shut down gRPC server (#11437) 2019-12-11 13:07:16 -08:00
Jingyi Hu
e67b9829b6 *: enable lease checkpoint via experimental flag
Primary lessor persist lease remainingTTL only if experimental flag
"--experimental-enable-lease-checkpoint" is set.
2019-06-05 15:30:03 -07:00
Jingyi Hu
6bf609b96d integration: update TestMemberPromote test
Update TestMemberPromote to include both learner not-ready and learner
ready test cases.

Removed unit test TestPromoteMember, it requires underlying raft node to
be started and running. The member promote is covered by the integration
test.
2019-05-28 18:50:13 -07:00
Jingyi Hu
d0c1b3fa38 etcdserver: learner return Unavailable for unsupported RPC
Make learner return code.Unavailable when the request is not supported
by learner. Client balancer will retry a different endpoint.
2019-05-28 18:47:03 -07:00
Gyuho Lee
34bd797e67 *: revert module import paths
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2019-05-28 15:39:35 -07:00
Jingyi Hu
57a11eb1e1 integration: add TestKVForLearner
Adding TestKVForLearner. Also adding test utility functions for clientv3
integration tests.
2019-05-15 13:27:38 -07:00
shivaramr
9150bf52d6 go modules: Fix module path version to include version number 2019-04-26 15:29:50 -07:00
Gyuho Lee
dca0dec382 integration: use default log configuration
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2019-02-21 11:00:01 -08:00
mrIncompetent
2be5994f61 integration: Replace *testing.T with testing.TB
Use testing.TB for integration test helpers to enable usage in benchmarks
2018-09-04 22:34:40 +02:00
Gyuho Lee
07fcc26799 *: fix gofmt warnings with Go 1.11
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2018-08-28 21:45:24 -07:00
Gyuho Lee
d37f1521b7 *: update import paths to "go.etcd.io/etcd"
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2018-08-28 17:47:55 -07:00
Joe Betz
750b87d622
Merge pull request #9924 from jpbetz/persist-lease-deadline
lease: Persist remainingTTL to prevent indefinite auto-renewal of long lived leases
2018-07-24 09:39:57 -07:00
Joe Betz
d1de41efe1
lease: Add unit and integration tests for lease checkpointing 2018-07-23 16:12:42 -07:00
Gyuho Lee
42bef8460c *: clean up code format
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-07-21 16:34:56 -07:00
Joe Betz
05c57a0ea4 integration: Fix unit test failures from new grpc LB changes, fix bom 2018-06-15 13:41:28 -07:00
Joe Betz
1f6548b751 clientv3: Stop expecting retry in integration tests with new grpc balancer 2018-06-15 13:41:28 -07:00
Joe Betz
9304d1abd1 clientv3: Fix TLS test failures by returning DeadlineExceeded error from dial without any additional wrapping 2018-06-15 13:41:28 -07:00
Joe Betz
f84f554301 clientv3: Fix auth client to use endpoints instead of host when dialing, fix tests to block on dial when required. 2018-06-15 13:41:28 -07:00
Joe Betz
66e65cd660 clientv3: Avoid timeouts in ordering test 2018-06-15 13:41:28 -07:00
Gyuho Lee
a52f16d4a6 integration: test lease revoke routine with JWT token
https://github.com/coreos/etcd/pull/9698 wasn't really testing
the panic code path when leases are expiry.

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-05-24 11:38:31 -07:00
Gyuho Lee
7940113906 *: move internal "etcdserver/api/rafthttp"
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-05-21 10:31:16 -07:00
Gyuho Lee
aa99ed54b7 integration: rename "SnapshotCount"
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-05-18 14:41:30 -07:00
Gyuho Lee
b7443ad849 integration: remove unused parameters
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-05-09 15:42:45 -07:00
Jiang Xuan
bf432648ae *: make bcrypt-cost configurable 2018-05-03 11:43:32 -07:00
Gyuho Lee
1d18f45caf integration: silence http server logs
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-05-02 12:58:56 -07:00
Gyuho Lee
58b7ab97eb integration: use structured logging in peer handler
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-05-02 12:53:56 -07:00
Gyuho Lee
d2e1484762 integration: support structured logging in "v2http"
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-05-02 11:57:23 -07:00
Gyuho Lee
21d75570ee integration: use "zap" logger
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-05-01 16:15:09 -07:00
Gyuho Lee
3fe9030d34 integration: set InitialElectionTickAdvance to true by default
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-04-19 17:45:23 -07:00
Gyuho Lee
68802b333b integration: re-overwrite "httptest.Server" TLS.Certificates
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-04-13 14:19:34 -07:00
Gyuho Lee
b705f6f8f2 Revert "integration: fix httptest TLS server start"
This reverts commit 3d3ba8b9cb182e958d8d0c26ac13fff899424fc9.
2018-04-13 13:06:09 -07:00
Gyuho Lee
6ab9776601 integration: add "TestTLSReloadCopyIPOnly"
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-04-13 13:02:45 -07:00
Gyuho Lee
233af4b4ec integration: add IP only TLSInfos
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-04-13 12:03:55 -07:00
Gyuho Lee
3d3ba8b9cb integration: fix httptest TLS server start
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-04-13 11:16:54 -07:00
Gyuho Lee
2d13894a16 integration: fix "fixtures-expired" path
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-04-09 11:49:42 -07:00
Gyuho Lee
a60bb6104c *: regenerate "fixtures-expired"
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-04-09 10:59:52 -07:00
Sam Batschelet
d21fef2d41 integration: add constant RequestWaitTimeout. 2018-01-03 15:53:57 -05:00
Gyuho Lee
3d56045da0 integration: bump up wait leader timeout for slow CIs
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2017-12-20 10:58:05 -08:00
Gyuho Lee
f38593bbad clientv3/integration: test large KV requests
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2017-12-20 10:58:05 -08:00
Xiang
1c69cc5657 integration: always enable initial corruption check 2017-11-26 16:51:04 -08:00
Gyu-Ho Lee
a41f3b64aa integration: expose ElectionTimeout, multiply ticks to timeout
To be consistent with etcdserver

Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-10-30 09:22:40 -07:00
Gyu-Ho Lee
b6f770fc24 integration: use variadic parameters for *Partition
'member' type is not exported.
In network partition tests, we want do

InjectPartition(t, clus.Members[lead], clus.Members[lead+1])

Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-10-25 14:55:03 -07:00
Gyu-Ho Lee
ff2ed93b5c clientv3/integration: do not create v3 clients when not used
Add 'SkipCreatingClients' field to skip creating clients if not used.

Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-10-24 16:59:41 -07:00