
Updated scripts and documentation to not recommend vendoring. Implemented best practices for tools installation. Performed multiple tests to confirm its not breaking any workflows and has no negative performance impact. Rather see 3x speedup. 1. PASSES="fmt unit integration e2e functional" ./test 2. ./scripts/updatebom.sh 3. ./scripts/updatedep.sh 4. ./scripts/genproto.sh - works - ca be simplified - in follow up PR 5. Installation without explicit GOPATH: ``` % unset GOPATH % [sudo] rm -rf ~/go % git clone https://github.com/etcd-io/etcd.git % time ./build go: downloading google.golang.org/grpc v1.26.0 go: downloading github.com/jonboulle/clockwork v0.1.0 go: downloading github.com/prometheus/client_golang v1.0.0 go: downloading github.com/soheilhy/cmux v0.1.4 go: downloading github.com/gogo/protobuf v1.2.1 go: downloading sigs.k8s.io/yaml v1.1.0 go: downloading golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2 go: downloading github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903 go: downloading go.etcd.io/bbolt v1.3.5 go: downloading go.uber.org/zap v1.15.0 go: downloading golang.org/x/crypto v0.0.0-20191002192127-34f69633bfdc go: downloading github.com/golang/protobuf v1.3.2 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8 go: downloading github.com/beorn7/perks v1.0.0 go: downloading github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4 go: downloading github.com/coreos/go-systemd/v22 v22.0.0 go: downloading gopkg.in/yaml.v2 v2.2.2 go: downloading github.com/coreos/go-semver v0.2.0 go: downloading github.com/sirupsen/logrus v1.4.2 go: downloading golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/google/uuid v1.0.0 go: downloading github.com/modern-go/reflect2 v1.0.1 go: downloading github.com/prometheus/common v0.4.1 go: downloading github.com/spf13/cobra v0.0.3 go: downloading github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 go: downloading github.com/gorilla/websocket v0.0.0-20170926233335-4201258b820c go: downloading github.com/spf13/pflag v1.0.1 go: downloading github.com/json-iterator/go v1.1.7 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/google/btree v1.0.0 go: downloading go.uber.org/atomic v1.6.0 go: downloading github.com/prometheus/procfs v0.0.2 go: downloading go.uber.org/multierr v1.5.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.9.5 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4 go: downloading github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4 go: downloading google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55 go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1 go: downloading golang.org/x/text v0.3.3 go: downloading github.com/olekukonko/tablewriter v0.0.0-20170122224234-a0225b3f23b5 go: downloading github.com/bgentry/speakeasy v0.1.0 go: downloading gopkg.in/cheggaaa/pb.v1 v1.0.25 go: downloading github.com/urfave/cli v1.20.0 go: downloading github.com/mattn/go-runewidth v0.0.2 ./build 8.22s user 2.31s system 117% cpu 8.961 total ``` Before: ``` % git clone https://github.com/etcd-io/etcd.git && cd etcd && time ./build Cloning into 'etcd'... remote: Enumerating objects: 97872, done. remote: Total 97872 (delta 0), reused 0 (delta 0), pack-reused 97872 Receiving objects: 100% (97872/97872), 58.97 MiB | 19.85 MiB/s, done. Resolving deltas: 100% (63091/63091), done. ./build 34.97s user 4.15s system 236% cpu 16.555 total ``` 6. Rebuild without changes: ``` % time ./build ./build 1.43s user 0.83s system 168% cpu 1.336 total ``` 7. Instantation of vendor directory (assuming ./build loaded them to $GOPATH/pkg): ``` time go mod vendor go: downloading github.com/inconshreveable/mousetrap v1.0.0 go: downloading github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa go: downloading github.com/creack/pty v1.1.11 go: downloading github.com/etcd-io/gofail v0.0.0-20190801230047-ad7f989257ca go: downloading github.com/konsorten/go-windows-terminal-sequences v1.0.1 go mod vendor 0.51s user 0.44s system 110% cpu 0.861 total ``` 8. Fresh instantation of vendor: ``` % rm -rf vendor % [sudo] rm -rf ~/go % time go mod vendor go: downloading github.com/coreos/go-systemd/v22 v22.0.0 go: downloading github.com/spf13/cobra v0.0.3 go: downloading github.com/prometheus/client_golang v1.0.0 go: downloading golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7 go: downloading github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4 go: downloading github.com/gogo/protobuf v1.2.1 go: downloading sigs.k8s.io/yaml v1.1.0 go: downloading google.golang.org/grpc v1.26.0 go: downloading github.com/urfave/cli v1.20.0 go: downloading go.uber.org/zap v1.15.0 go: downloading github.com/spf13/pflag v1.0.1 go: downloading github.com/soheilhy/cmux v0.1.4 go: downloading github.com/json-iterator/go v1.1.7 go: downloading github.com/coreos/go-semver v0.2.0 go: downloading github.com/prometheus/common v0.4.1 go: downloading github.com/prometheus/procfs v0.0.2 go: downloading go.uber.org/atomic v1.6.0 go: downloading github.com/olekukonko/tablewriter v0.0.0-20170122224234-a0225b3f23b5 go: downloading github.com/golang/protobuf v1.3.2 go: downloading github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4 go: downloading github.com/modern-go/reflect2 v1.0.1 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading go.uber.org/multierr v1.5.0 go: downloading github.com/creack/pty v1.1.11 go: downloading github.com/mattn/go-runewidth v0.0.2 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading golang.org/x/crypto v0.0.0-20191002192127-34f69633bfdc go: downloading golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5 go: downloading github.com/jonboulle/clockwork v0.1.0 go: downloading gopkg.in/yaml.v2 v2.2.2 go: downloading github.com/etcd-io/gofail v0.0.0-20190801230047-ad7f989257ca go: downloading github.com/grpc-ecosystem/grpc-gateway v1.9.5 go: downloading github.com/google/btree v1.0.0 go: downloading google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55 go: downloading github.com/beorn7/perks v1.0.0 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/google/uuid v1.0.0 go: downloading golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8 go: downloading go.etcd.io/bbolt v1.3.5 go: downloading golang.org/x/text v0.3.3 go: downloading gopkg.in/cheggaaa/pb.v1 v1.0.25 go: downloading github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 go: downloading github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4 go: downloading github.com/inconshreveable/mousetrap v1.0.0 go: downloading github.com/gorilla/websocket v0.0.0-20170926233335-4201258b820c go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1 go: downloading github.com/bgentry/speakeasy v0.1.0 go: downloading github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903 go: downloading github.com/sirupsen/logrus v1.4.2 go: downloading github.com/konsorten/go-windows-terminal-sequences v1.0.1 go mod vendor 3.62s user 1.30s system 127% cpu 3.854 total ``` 9. Size of the repository - before: 39M, after: 18M Before: ``` % time git clone https://github.com/etcd-io/etcd.git Cloning into 'etcd'... remote: Enumerating objects: 97872, done. remote: Total 97872 (delta 0), reused 0 (delta 0), pack-reused 97872 Receiving objects: 100% (97872/97872), 58.97 MiB | 20.53 MiB/s, done. Resolving deltas: 100% (63091/63091), done. git clone https://github.com/etcd-io/etcd.git 4.66s user 1.02s system 93% cpu 6.068 total % du -h --exclude .git -d 1 944K ./clientv3 108K ./etcdmain 5.4M ./Documentation 384K ./security 384K ./mvcc 28K ./.github 8.0K ./version 144K ./contrib 240K ./proxy 2.5M ./etcdserver 112K ./embed 536K ./integration 332K ./tools 116K ./lease 108K ./logos 896K ./tests 960K ./raft 216K ./client 52K ./scripts 100K ./hack 464K ./etcdctl 3.0M ./pkg 620K ./functional 136K ./wal 152K ./auth 21M ./vendor 39M ``` After: ``` % time git clone https://github.com/ptabor/etcd.git -b 20200908-no-vendor Cloning into 'etcd'... remote: Enumerating objects: 38, done. remote: Counting objects: 100% (38/38), done. remote: Compressing objects: 100% (37/37), done. remote: Total 98489 (delta 10), reused 8 (delta 1), pack-reused 98451 Receiving objects: 100% (98489/98489), 59.23 MiB | 21.26 MiB/s, done. Resolving deltas: 100% (63572/63572), done. git clone https://github.com/ptabor/etcd.git -b 20200908-no-vendor 5.56s user 1.05s system 105% cpu 6.260 total % du -h --exclude .git -d 1 944K ./clientv3 108K ./etcdmain 5.4M ./Documentation 384K ./security 384K ./mvcc 28K ./.github 8.0K ./version 144K ./contrib 240K ./proxy 2.5M ./etcdserver 112K ./embed 536K ./integration 332K ./tools 116K ./lease 108K ./logos 896K ./tests 960K ./raft 216K ./client 56K ./scripts 100K ./hack 464K ./etcdctl 3.0M ./pkg 620K ./functional 136K ./wal 152K ./auth 19M . ```
etcd
Note: The master
branch may be in an unstable or even broken state during development. Please use releases instead of the master
branch in order to get stable binaries.
etcd is a distributed reliable key-value store for the most critical data of a distributed system, with a focus on being:
- Simple: well-defined, user-facing API (gRPC)
- Secure: automatic TLS with optional client cert authentication
- Fast: benchmarked 10,000 writes/sec
- Reliable: properly distributed using Raft
etcd is written in Go and uses the Raft consensus algorithm to manage a highly-available replicated log.
etcd is used in production by many companies, and the development team stands behind it in critical deployment scenarios, where etcd is frequently teamed with applications such as Kubernetes, locksmith, vulcand, Doorman, and many others. Reliability is further ensured by rigorous testing.
See etcdctl for a simple command line client.
Community meetings
etcd contributors and maintainers have monthly (every four weeks) meetings at 11:00 AM (USA Pacific) on Thursday.
An initial agenda will be posted to the shared Google docs a day before each meeting, and everyone is welcome to suggest additional topics or other agendas.
Time:
- Jan 10th, 2019 11:00 AM video
- Feb 7th, 2019 11:00 AM video
- Mar 7th, 2019 11:00 AM video
- Apr 4th, 2019 11:00 AM video
- May 2nd, 2019 11:00 AM video
- May 30th, 2019 11:00 AM video
- Jul 11th, 2019 11:00 AM video
- Jul 25, 2019 11:00 AM video
- Aug 22, 2019 11:00 AM video
- Sep 19, 2019 11:00 AM video
- Nov 14, 2019 11:00 AM
- Dec 12, 2019 11:00 AM
- Jan 09, 2020 11:00 AM
- Feb 06, 2020 11:00 AM
- Mar 05, 2020 11:00 AM
- Apr 02, 2020 11:00 AM
- Apr 30, 2020 11:00 AM
- May 28, 2020 11:00 AM
- Jun 25, 2020 11:00 AM
- Jul 23, 2020 11:00 AM
- Aug 20, 2020 11:00 AM
- Sep 17, 2020 11:00 AM
- Oct 15, 2020 11:00 AM
- Nov 12, 2020 11:00 AM
- Dec 10, 2020 11:00 AM
Join Hangouts Meet: meet.google.com/umg-nrxn-qvs
Join by phone: +1 405-792-0633 PIN: 299 906#
Getting started
Getting etcd
The easiest way to get etcd is to use one of the pre-built release binaries which are available for OSX, Linux, Windows, and Docker on the release page.
For more installation guides, please check out play.etcd.io and operating etcd.
For those wanting to try the very latest version, build the latest version of etcd from the master
branch. This first needs Go installed (version 1.13+ is required). All development occurs on master
, including new features and bug fixes. Bug fixes are first targeted at master
and subsequently ported to release branches, as described in the branch management guide.
Running etcd
First start a single-member cluster of etcd.
If etcd is installed using the pre-built release binaries, run it from the installation location as below:
/tmp/etcd-download-test/etcd
The etcd command can be simply run as such if it is moved to the system path as below:
mv /tmp/etcd-download-test/etcd /usr/local/bin/
etcd
If etcd is built from the master branch, run it as below:
./bin/etcd
This will bring up etcd listening on port 2379 for client communication and on port 2380 for server-to-server communication.
Next, let's set a single key, and then retrieve it:
etcdctl put mykey "this is awesome"
etcdctl get mykey
etcd is now running and serving client requests. For more, please check out:
etcd TCP ports
The official etcd ports are 2379 for client requests, and 2380 for peer communication.
Running a local etcd cluster
First install goreman, which manages Procfile-based applications.
Our Procfile script will set up a local example cluster. Start it with:
goreman start
This will bring up 3 etcd members infra1
, infra2
and infra3
and etcd grpc-proxy
, which runs locally and composes a cluster.
Every cluster member and proxy accepts key value reads and key value writes.
Follow the steps in Procfile.learner to add a learner node to the cluster. Start the learner node with:
goreman -f ./Procfile.learner start
Next steps
Now it's time to dig into the full etcd API and other guides.
- Read the full documentation.
- Explore the full gRPC API.
- Set up a multi-machine cluster.
- Learn the config format, env variables and flags.
- Find language bindings and tools.
- Use TLS to secure an etcd cluster.
- Tune etcd.
Contact
- Mailing list: etcd-dev
- IRC: #etcd on freenode.org
- Planning/Roadmap: milestones, roadmap
- Bugs: issues
Contributing
See CONTRIBUTING for details on submitting patches and the contribution workflow.
Reporting bugs
See reporting bugs for details about reporting any issues.
Reporting a security vulnerability
See security disclosure and release process for details on how to report a security vulnerability and how the etcd team manages it.
Issue and PR management
See issue triage guidelines for details on how issues are managed.
See PR management for guidelines on how pull requests are managed.
etcd Emeritus Maintainers
These emeritus maintainers dedicated a part of their career to etcd and reviewed code, triaged bugs, and pushed the project forward over a substantial period of time. Their contribution is greatly appreciated.
- Fanmin Shi
- Anthony Romano
License
etcd is under the Apache 2.0 license. See the LICENSE file for details.