304e260038
``` for sha in :/^Revert :/BenchmarkRawNode :/^raft:.directly; do git checkout raft-single-voter && git checkout $(git log -n 1 '--pretty=format:%H' $sha) && f=$(git log -1 --pretty=%s | sed -E 's/[^A-Za-z0-9]+/_/g').txt && go test -run - -count 10 -bench BenchmarkRawNode -benchmem -benchtime=100000x . > $f; done; git checkout raft-single-voter ``` The two possible solutions (directly updating progress and calling maybeCommit in `(*raft).advance` vs calling `r.Step`) are identical. In fact, we've gotten a tiny bit better with the `.Step` solution in terms of not calling `firstIndex` as much, in the common case of not being a single voter. ``` $ benchstat raft_directly_update_leader_in_advance.txt Revert_raft_directly_update_leader_in_advance_.txt name old time/op new time/op delta RawNode/single-voter-10 482ns ± 2% 742ns ± 1% +54.02% (p=0.000 n=9+9) RawNode/two-voters-10 1.29µs ± 1% 1.31µs ± 2% +1.70% (p=0.000 n=9+10) name old firstIndex/op new firstIndex/op delta RawNode/single-voter-10 4.00 ± 0% 5.00 ± 0% +25.00% (p=0.000 n=10+10) RawNode/two-voters-10 10.0 ± 0% 9.0 ± 0% -10.00% (p=0.000 n=10+10) name old lastIndex/op new lastIndex/op delta RawNode/single-voter-10 1.00 ± 0% 2.00 ± 0% +100.00% (p=0.000 n=10+10) RawNode/two-voters-10 2.00 ± 0% 2.00 ± 0% ~ (all equal) name old ready/op new ready/op delta RawNode/single-voter-10 1.00 ± 0% 2.00 ± 0% +100.00% (p=0.000 n=10+10) RawNode/two-voters-10 2.00 ± 0% 2.00 ± 0% ~ (all equal) name old term/op new term/op delta RawNode/single-voter-10 0.00 ± 0% 0.00 ± 0% ~ (all equal) RawNode/two-voters-10 1.00 ± 0% 1.00 ± 0% ~ (all equal) name old alloc/op new alloc/op delta RawNode/single-voter-10 372B ± 0% 388B ± 0% +4.30% (p=0.000 n=10+10) RawNode/two-voters-10 964B ± 0% 964B ± 0% ~ (all equal) name old allocs/op new allocs/op delta RawNode/single-voter-10 4.00 ± 0% 5.00 ± 0% +25.00% (p=0.000 n=10+10) RawNode/two-voters-10 7.00 ± 0% 7.00 ± 0% ~ (all equal) ``` We then compare the `.Step` solution against the previous "status quo" that prematurely emitted uncommitted entries for command application below. Importantly, we don't regress in the case of multiple peers. We actually gain slightly in terms of `lastIndex` calls, but run a bit more code; acceptable. In the single-voter case, since we now need two Ready handling cycles per op instead of one, we see additional calls to lastIndex and firstIndex as well as slightly increased allocations. These are expected and trade-offs we're willing to make to avoid correctness problems. Note that the benchmark intentionally forces full processing of each individual entries, so some of the new overhead would likely amortize on a singleton voter seeing high throughput as multiple proposals could share the Ready cycles. ``` $ benchstat raft_add_BenchmarkRawNode.txt Revert_raft_directly_update_leader_in_advance_.txt name old time/op new time/op delta RawNode/single-voter-10 482ns ± 2% 742ns ± 1% +54.02% (p=0.000 n=9+9) RawNode/two-voters-10 1.29µs ± 1% 1.31µs ± 2% +1.70% (p=0.000 n=9+10) name old firstIndex/op new firstIndex/op delta RawNode/single-voter-10 4.00 ± 0% 5.00 ± 0% +25.00% (p=0.000 n=10+10) RawNode/two-voters-10 10.0 ± 0% 9.0 ± 0% -10.00% (p=0.000 n=10+10) name old lastIndex/op new lastIndex/op delta RawNode/single-voter-10 1.00 ± 0% 2.00 ± 0% +100.00% (p=0.000 n=10+10) RawNode/two-voters-10 2.00 ± 0% 2.00 ± 0% ~ (all equal) name old ready/op new ready/op delta RawNode/single-voter-10 1.00 ± 0% 2.00 ± 0% +100.00% (p=0.000 n=10+10) RawNode/two-voters-10 2.00 ± 0% 2.00 ± 0% ~ (all equal) name old term/op new term/op delta RawNode/single-voter-10 0.00 ± 0% 0.00 ± 0% ~ (all equal) RawNode/two-voters-10 1.00 ± 0% 1.00 ± 0% ~ (all equal) name old alloc/op new alloc/op delta RawNode/single-voter-10 372B ± 0% 388B ± 0% +4.30% (p=0.000 n=10+10) RawNode/two-voters-10 964B ± 0% 964B ± 0% ~ (all equal) name old allocs/op new allocs/op delta RawNode/single-voter-10 4.00 ± 0% 5.00 ± 0% +25.00% (p=0.000 n=10+10) RawNode/two-voters-10 7.00 ± 0% 7.00 ± 0% ~ (all equal) ``` `tools/benchmark put`: ``` Summary[main]: | Summary[this PR]: Total: 284.4443 secs. | Total: 288.1100 secs. Slowest: 0.1626 secs. | Slowest: 0.1456 secs. Fastest: 0.0027 secs. | Fastest: 0.0018 secs. Average: 0.0284 secs. | Average: 0.0288 secs. Stddev: 0.0178 secs. | Stddev: 0.0182 secs. Requests/sec: 35.1563 | Requests/sec: 34.7090 [=0.98727681809x main] Response time histogram: | Response time histogram: 0.0027 [1] | | 0.0018 [1] | 0.0187 [137] | | 0.0162 [34] | 0.0347 [7895] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ | 0.0305 [7938] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ 0.0507 [86] | | 0.0449 [103] | 0.0667 [1328] |∎∎∎∎∎∎ | 0.0593 [1056] |∎∎∎∎∎ 0.0827 [480] |∎∎ | 0.0737 [420] |∎∎ 0.0987 [45] | | 0.0881 [370] |∎ 0.1147 [18] | | 0.1025 [48] | 0.1306 [7] | | 0.1168 [19] | 0.1466 [2] | | 0.1312 [6] | 0.1626 [1] | | 0.1456 [5] | Latency distribution: | Latency distribution: 10% in 0.0195 secs. | 10% in 0.0194 secs. 25% in 0.0198 secs. | 25% in 0.0198 secs. 50% in 0.0201 secs. | 50% in 0.0201 secs. 75% in 0.0210 secs. | 75% in 0.0214 secs. 90% in 0.0585 secs. | 90% in 0.0589 secs. 95% in 0.0727 secs. | 95% in 0.0731 secs. 99% in 0.0762 secs. | 99% in 0.0788 secs. 99.9% in 0.1244 secs. | 99.9% in 0.1240 secs. ``` Signed-off-by: Tobias Grieger <tobias.b.grieger@gmail.com>
etcd
Note: The main branch may be in an unstable or even broken state during development. For stable versions, see releases.
etcd is a distributed reliable key-value store for the most critical data of a distributed system, with a focus on being:
- Simple: well-defined, user-facing API (gRPC)
- Secure: automatic TLS with optional client cert authentication
- Fast: benchmarked 10,000 writes/sec
- Reliable: properly distributed using Raft
etcd is written in Go and uses the Raft consensus algorithm to manage a highly-available replicated log.
etcd is used in production by many companies, and the development team stands behind it in critical deployment scenarios, where etcd is frequently teamed with applications such as Kubernetes, locksmith, vulcand, Doorman, and many others. Reliability is further ensured by rigorous testing.
See etcdctl for a simple command line client.
Maintainers
MAINTAINERS strive to shape an inclusive open source project culture where users are heard and contributors feel respected and empowered. MAINTAINERS maintain productive relationships across different companies and disciplines. Read more about MAINTAINERS role and responsibilities.
Getting started
Getting etcd
The easiest way to get etcd is to use one of the pre-built release binaries which are available for OSX, Linux, Windows, and Docker on the release page.
For more installation guides, please check out play.etcd.io and operating etcd.
Running etcd
First start a single-member cluster of etcd.
If etcd is installed using the pre-built release binaries, run it from the installation location as below:
/tmp/etcd-download-test/etcd
The etcd command can be simply run as such if it is moved to the system path as below:
mv /tmp/etcd-download-test/etcd /usr/local/bin/
etcd
This will bring up etcd listening on port 2379 for client communication and on port 2380 for server-to-server communication.
Next, let's set a single key, and then retrieve it:
etcdctl put mykey "this is awesome"
etcdctl get mykey
etcd is now running and serving client requests. For more, please check out:
etcd TCP ports
The official etcd ports are 2379 for client requests, and 2380 for peer communication.
Running a local etcd cluster
First install goreman, which manages Procfile-based applications.
Our Procfile script will set up a local example cluster. Start it with:
goreman start
This will bring up 3 etcd members infra1, infra2 and infra3 and optionally etcd grpc-proxy, which runs locally and composes a cluster.
Every cluster member and proxy accepts key value reads and key value writes.
Follow the steps in Procfile.learner to add a learner node to the cluster. Start the learner node with:
goreman -f ./Procfile.learner start
Install etcd client v3
go get go.etcd.io/etcd/client/v3
Next steps
Now it's time to dig into the full etcd API and other guides.
- Read the full documentation.
- Explore the full gRPC API.
- Set up a multi-machine cluster.
- Learn the config format, env variables and flags.
- Find language bindings and tools.
- Use TLS to secure an etcd cluster.
- Tune etcd.
Contact
- Email: etcd-dev
- Slack: #etcd channel on Kubernetes (get an invite)
- Community meetings
Community meetings
etcd contributors and maintainers have monthly (every four weeks) meetings at 11:00 AM (USA Pacific) on Thursday.
An initial agenda will be posted to the shared Google docs a day before each meeting, and everyone is welcome to suggest additional topics or other agendas.
Meeting recordings are uploaded to official etcd YouTube channel.
Get calendar invitation by joining etcd-dev mailing group.
Join Hangouts Meet: meet.google.com/umg-nrxn-qvs
Join by phone: +1 405-792-0633 PIN: 299 906#
Contributing
See CONTRIBUTING for details on submitting patches and the contribution workflow.
Reporting bugs
See reporting bugs for details about reporting any issues.
Reporting a security vulnerability
See security disclosure and release process for details on how to report a security vulnerability and how the etcd team manages it.
Issue and PR management
See issue triage guidelines for details on how issues are managed.
See PR management for guidelines on how pull requests are managed.
etcd Emeritus Maintainers
These emeritus maintainers dedicated a part of their career to etcd and reviewed code, triaged bugs and pushed the project forward over a substantial period of time. Their contribution is greatly appreciated.
- Fanmin Shi
- Anthony Romano
- Brandon Philips
- Joe Betz
- Gyuho Lee
- Jingyi Hu
- Wenjia Zhang
- Xiang Li
- Ben Darnell
- Tobias Grieger
License
etcd is under the Apache 2.0 license. See the LICENSE file for details.