etcd/CHANGELOG-3.5.md

Previous change logs can be found at CHANGELOG-3.4.

The minimum recommended etcd versions to run in production are 3.2.28+, 3.3.18+, and 3.4.2+.

---

v3.5.0 (2021-06)

See code changes and v3.5 upgrade guide for any breaking changes.

• v3.5.0 (2020 TBD), see code changes.
• v3.5.0-rc.1 (2020 TBD), see code changes.
• v3.5.0-rc.0 (2020 TBD), see code changes.

Again, before running upgrades from any previous release, please make sure to read change logs below and v3.5 upgrade guide.

Breaking Changes

• go.etcd.io/etcd Go packages have moved to go.etcd.io/etcd/{api,pkg,raft,client,etcdctl,server,raft,tests}/v3 to follow the Go modules conventions
• go.etcd.io/clientv3/snapshot SnapshotManager class have moved to go.etcd.io/clientv3/etcdctl. The method snapshot.Save to download a snapshot from the remote server was preserved in 'go.etcd.io/clientv3/snapshot`.
• `go.etcd.io/client' package got migrated to 'go.etcd.io/client/v2'.
• Changed behavior of clienv3 API MemberList.
  • Previously, it is directly served with server's local data, which could be stale.
  • Now, it is served with linearizable guarantee. If the server is disconnected from quorum, MemberList call will fail.
• gRPC gateway only supports /v3 endpoint.
  • Deprecated /v3beta.
  • curl -L http://localhost:2379/v3beta/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}' does work in v3.5. Use curl -L http://localhost:2379/v3/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}' instead.
• etcd --experimental-enable-v2v3 flag has been deprecated. Use etcd --enable-v2v3 instead.
  • Change etcd --experimental-enable-v2v3 flag to etcd --enable-v2v3; v2 storage emulation is now stable.
• etcd --experimental-backend-bbolt-freelist-type flag has been deprecated. Use etcd --backend-bbolt-freelist-type instead. The default type is hashmap and it is stable now.
• etcd --debug flag has been deprecated. Use etcd --log-level=debug instead.
• Remove embed.Config.Debug.
• etcd --log-output flag has been deprecated. Use etcd --log-outputs instead.
• etcd --logger=zap --log-outputs=stderr is now the default.
• etcd --logger=capnslog flag value has been deprecated.
• etcd --logger=zap --log-outputs=default flag value is not supported..
  • Use etcd --logger=zap --log-outputs=stderr.
  • Or, use etcd --logger=zap --log-outputs=systemd/journal to send logs to the local systemd journal.
  • Previously, if etcd parent process ID (PPID) is 1 (e.g. run with systemd), etcd --logger=capnslog --log-outputs=default redirects server logs to local systemd journal. And if write to journald fails, it writes to os.Stderr as a fallback.
  • However, even with PPID 1, it can fail to dial systemd journal (e.g. run embedded etcd with Docker container). Then, every single log write will fail and fall back to os.Stderr, which is inefficient.
  • To avoid this problem, systemd journal logging must be configured manually.
• etcd --log-outputs=stderr is now the default.
• etcd --log-package-levels flag for capnslog has been deprecated. Now, etcd --logger=zap --log-outputs=stderr is the default.
• [CLIENT-URL]/config/local/log endpoint has been deprecated, as is etcd --log-package-levels flag.
  • curl http://127.0.0.1:2379/config/local/log -XPUT -d '{"Level":"DEBUG"}' won't work.
  • Please use etcd --logger=zap --log-outputs=stderr instead.
• Deprecated etcd_debugging_mvcc_db_total_size_in_bytes Prometheus metric. Use etcd_mvcc_db_total_size_in_bytes instead.
• Deprecated etcd_debugging_mvcc_put_total Prometheus metric. Use etcd_mvcc_put_total instead.
• Deprecated etcd_debugging_mvcc_delete_total Prometheus metric. Use etcd_mvcc_delete_total instead.
• Deprecated etcd_debugging_mvcc_txn_total Prometheus metric. Use etcd_mvcc_txn_total instead.
• Deprecated etcd_debugging_mvcc_range_total Prometheus metric. Use etcd_mvcc_range_total instead.
• Master branch /version outputs 3.5.0-pre, instead of 3.4.0+git.
• Changed proxy package function signature to support structured logger.
  • Previously, NewClusterProxy(c *clientv3.Client, advaddr string, prefix string) (pb.ClusterServer, <-chan struct{}), now NewClusterProxy(lg *zap.Logger, c *clientv3.Client, advaddr string, prefix string) (pb.ClusterServer, <-chan struct{}).
  • Previously, Register(c *clientv3.Client, prefix string, addr string, ttl int), now Register(lg *zap.Logger, c *clientv3.Client, prefix string, addr string, ttl int) <-chan struct{}.
  • Previously, NewHandler(t *http.Transport, urlsFunc GetProxyURLs, failureWait time.Duration, refreshInterval time.Duration) http.Handler, now NewHandler(lg *zap.Logger, t *http.Transport, urlsFunc GetProxyURLs, failureWait time.Duration, refreshInterval time.Duration) http.Handler.
• Changed pkg/flags function signature to support structured logger.
  • Previously, SetFlagsFromEnv(prefix string, fs *flag.FlagSet) error, now SetFlagsFromEnv(lg *zap.Logger, prefix string, fs *flag.FlagSet) error.
  • Previously, SetPflagsFromEnv(prefix string, fs *pflag.FlagSet) error, now SetPflagsFromEnv(lg *zap.Logger, prefix string, fs *pflag.FlagSet) error.
• ClientV3 supports grpc resolver API.
  • Endpoints can be managed using endpoints.Manager
  • Previously supported GRPCResolver was decomissioned. Use resolver instead.
• Turned on --pre-vote by default. Should prevent disrupting RAFT leader by an individual member.
• ETCD_CLIENT_DEBUG env: Now supports log levels (debug, info, warn, error, dpanic, panic, fatal). Only when set, overrides application-wide grpc logging settings.
• Embed Etcd.Close() needs to called exactly once and closes Etcd.Err() stream.
• Embed Etcd does not override global/grpc logger be default any longer. If desired, please call embed.Config::SetupGlobalLoggers() explicitly.
• Errors: context cancelled or context deadline exceeded are exposed as codes.Canceled, codes.DeadlineExceeded instead of 'codes.Unknown'.

• Make sure save snapshot downloads checksum for integrity checks.

Security

• Add TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 and TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 to etcd --cipher-suites.
• Changed the format of WAL entries related to auth for not keeping password as a plain text.
• Add third party Security Audit Report.
• A log warning is added when etcd uses any existing directory that has a permission different than 700 on Linux and 777 on Windows.

Metrics, Monitoring

See List of metrics for all metrics per release.

Note that any etcd_debugging_* metrics are experimental and subject to change.

• Deprecated etcd_debugging_mvcc_db_total_size_in_bytes Prometheus metric. Use etcd_mvcc_db_total_size_in_bytes instead.
• Deprecated etcd_debugging_mvcc_put_total Prometheus metric. Use etcd_mvcc_put_total instead.
• Deprecated etcd_debugging_mvcc_delete_total Prometheus metric. Use etcd_mvcc_delete_total instead.
• Deprecated etcd_debugging_mvcc_txn_total Prometheus metric. Use etcd_mvcc_txn_total instead.
• Deprecated etcd_debugging_mvcc_range_total Prometheus metric. Use etcd_mvcc_range_total instead.
• Add etcd_debugging_mvcc_current_revision Prometheus metric.
• Add etcd_debugging_mvcc_compact_revision Prometheus metric.
• Change etcd_cluster_version Prometheus metrics to include only major and minor version.
• Add etcd_debugging_mvcc_total_put_size_in_bytes Prometheus metric.
• Add etcd_server_client_requests_total with "type" and "client_api_version" labels.
• Add etcd_wal_write_bytes_total.
• Add etcd_debugging_auth_revision.
• Add os_fd_used and os_fd_limit to monitor current OS file descriptors.

etcd server

• etcd --enable-v2v3 flag is now stable.
  • etcd --experimental-enable-v2v3 has been deprecated.
  • Added more v2v3 integration tests.
  • etcd --enable-v2=true --enable-v2v3='' by default, to enable v2 API server that is backed by v2 store.
  • etcd --enable-v2=true --enable-v2v3=/aaa to enable v2 API server that is backed by v3 storage.
  • etcd --enable-v2=false --enable-v2v3='' to disable v2 API server.
  • etcd --enable-v2=false --enable-v2v3=/aaa to disable v2 API server. TODO: error?
  • Add TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 and TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 to etcd --cipher-suites.
  • Automatically create parent directory if it does not exist (fix issue#9609).
  • v4.0 will configure etcd --enable-v2=true --enable-v2v3=/aaa to enable v2 API server that is backed by v3 storage.
• [etcd --backend-bbolt-freelist-type] flag is now stable.
  • etcd --experimental-backend-bbolt-freelist-type has been deprecated.
• Support rollback/downgrade.
• Deprecate v2 apply on cluster version. Use v3 request to set cluster version and recover cluster version from v3 backend.
• Fix corruption bug in defrag.
• Fix quorum protection logic when promoting a learner.
• Improve peer corruption checker to work when peer mTLS is enabled.
• Log [CLIENT-PORT]/health check in server side.
• Log successful etcd server-side health check in debug level.
• Improve compaction performance when latest index is greater than 1-million.
• Refactor consistentindex.
• Add log when etcdserver failed to apply command.
• Improve count-only range performance.
• Remove redundant storage restore operation to shorten the startup time.
  • With 40 million key test data,it can shorten the startup time from 5 min to 2.5 min.
• Fix deadlock bug in mvcc.
• Fix inconsistency between WAL and server snapshot.
  • Previously, server restore fails if it had crashed after persisting raft hard state but before saving snapshot.
  • See https://github.com/etcd-io/etcd/issues/10219 for more.
  • Add missing CRC checksum check in WAL validate method otherwise causes panic.
  • See https://github.com/etcd-io/etcd/issues/11918.
• Improve logging around snapshot send and receive.
• Push down RangeOptions.limit argv into index tree to reduce memory overhead.
• Add reason field for /health response.
• Add exclude alarms from health check conditionally
• Add --unsafe-no-fsync flag.
  • Setting the flag disables all uses of fsync, which is unsafe and will cause data loss. This flag makes it possible to run an etcd node for testing and development without placing lots of load on the file system.
• Add etcd --auth-token-ttl flag to customize simpleTokenTTL settings.
• Improve runtime.FDUsage call pattern to reduce objects malloc of Memory Usage and CPU Usage.
• Improve mvcc.watchResponse channel Memory Usage.
• Log expensive request info in UnaryInterceptor.
• Fix invalid Go type in etcdserverpb.
• Improve healthcheck by using v3 range request and its corresponding timeout.
• Add etcd --experimental-watch-progress-notify-interval flag to make watch progress notify interval configurable.
• Fix server panic in slow writes warnings.
  • Fixed via PR#12238.
• Fix server panic when force-new-cluster flag is enabled in a cluster which had learner node.
• Add --self-signed-cert-validity flag to support setting certificate expiration time.
  • Notice, certificates generated by etcd are valid for 1 year by default when specifying the auto-tls or peer-auto-tls option.
• Add --experimental-warning-apply-duration flag which allows apply duration threshold to be configurable.
• Add --experimental-memory-mlock" flag which prevents etcd memory pages to be swapped out.
• Add --socket-reuse-port flag
  • Setting this flag enables SO_REUSEPORT which allows rebind of a port already in use. User should take caution when using this flag to ensure flock is properly enforced.
• Add --socket-reuse-address flag
  • Setting this flag enables SO_REUSEADDR which allows binding to an address in TIME_WAIT state, improving etcd restart time.
• Reduce around 30% memory allocation by logging range response size without marshal.
• ETCD_VERIFY="all" enviroment triggers additional verification of consistency of etcd data-dir files.
• Add etcd --enable-log-rotation boolean flag which enables log rotation if true.
• Add etcd --log-rotation-config-json flag which allows passthrough of JSON config to configure log rotation for a file output target.
• Add experimental distributed tracing boolean flag --experimental-enable-distributed-tracing which enables tracing.
• Add --experimental-distributed-tracing-address string flag which allows configuring the OpenTelemetry collector address.
• Add --experimental-distributed-tracing-service-name string flag which allows changing the default "etcd" service name.
• Add --experimental-distributed-tracing-instance-id string flag which configures an instance ID, which must be unique per etcd instance.

Package runtime

• Optimize runtime.FDUsage by removing unnecessary sorting.

Package embed

• Remove embed.Config.Debug.
  • Use embed.Config.LogLevel instead.
• Add embed.Config.ZapLoggerBuilder to allow creating a custom zap logger.
• Replace global *zap.Logger with etcd server logger object.
• Add embed.Config.EnableLogRotation which enables log rotation if true.
• Add embed.Config.LogRotationConfigJSON to allow passthrough of JSON config to configure log rotation for a file output target.
• Add embed.Config.ExperimentalEnableDistributedTracing which enables experimental distributed tracing if true.
• Add embed.Config.ExperimentalDistributedTracingAddress which allows overriding default collector address.
• Add embed.Config.ExperimentalDistributedTracingServiceName which allows overriding default "etcd" service name.
• Add embed.Config.ExperimentalDistributedTracingServiceInstanceID which allows configuring an instance ID, which must be uniquer per etcd instance.

Package clientv3

• Remove excessive watch cancel logging messages.
  • See kubernetes/kubernetes#93450.
• Add TryLock method to clientv3/concurrency/Mutex. A non-blocking method on Mutex which does not wait to get lock on the Mutex, returns immediately if Mutex is locked by another session.
• Fix client balancer failover against multiple endpoints.
  • Fix "kube-apiserver: failover on multi-member etcd cluster fails certificate check on DNS mismatch".
• Fix IPv6 endpoint parsing in client.
  • Fix "1.16: etcd client does not parse IPv6 addresses correctly when members are joining" (kubernetes#83550).
• Fix errors caused by grpc changing balancer/resolver API. This change is compatible with grpc >= v1.26.0, but is not compatible with < v1.26.0 version.
• Use ServerName as the authority after bumping to grpc v1.26.0. Remove workaround in #11184.
• Fix "hasleader" metadata embedding.
  • Previously, clientv3.WithRequireLeader(ctx) was overwriting existing context keys.
• Fix watch leak caused by lazy cancellation. When clients cancel their watches, a cancel request will now be immediately sent to the server instead of waiting for the next watch event.
• Make sure save snapshot downloads checksum for integrity checks.
• Fix auth token invalid after watch reconnects. Get AuthToken automatically when clientConn is ready.
• Improve clientv3:get AuthToken gracefully without extra connection.
• Changed clientv3 dialing code to use grpc resolver API instead of custom balancer.
  • Endpoints self identify now as etcd-endpoints://{id}/#initially={list of endpoints} e.g. etcd-endpoints://0xc0009d8540/#initially=[localhost:2079]

Package lease

• Fix memory leak in follower nodes.
  • https://github.com/etcd-io/etcd/issues/11495
  • https://github.com/etcd-io/etcd/issues/11730
• Make sure grant/revoke won't be applied repeatedly after restarting etcd.

Package wal

• Add etcd_wal_write_bytes_total.
• Handle out-of-range slice bound in ReadAll and entry limit in decodeRecord.

etcdctl v3

• Fix etcdctl member add command to prevent potential timeout. (PR#11194 and PR#11638)
• Add etcdctl watch --progress-notify flag.
• Add etcdctl auth status command to check if authentication is enabled
• Add etcdctl get --count-only flag for output type fields.
• Add etcdctl member list -w=json --hex flag to print memberListResponse in hex format json.
• Changed etcdctl lock <lockname> exec-command to return exit code of exec-command.

gRPC gateway

• gRPC gateway only supports /v3 endpoint.
  • Deprecated /v3beta.
  • curl -L http://localhost:2379/v3beta/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}' does work in v3.5. Use curl -L http://localhost:2379/v3/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}' instead.

gRPC Proxy

• Fix panic on error for metrics handler.
• Add gRPC keepalive related flags grpc-keepalive-min-time, grpc-keepalive-interval and grpc-keepalive-timeout.
• Fix grpc watch proxy hangs when failed to cancel a watcher .
• Add metrics handler for grpcproxy self.
• Add health handler for grpcproxy self.

Auth

• Fix NoPassword check when adding user through GRPC gateway (issue#11414)
• Fix bug where some auth related messages are logged at wrong level
• Fix a data corruption bug by saving consistent index.
• Improve checkPassword performance.
• Add authRevision field in AuthStatus.

API

• Add /v3/auth/status endpoint to check if authentication is enabled
• Add Linearizable field to etcdserverpb.MemberListRequest.
• Learner support Snapshot RPC.

Package netutil

• Remove netutil.DropPort/RecoverPort/SetLatency/RemoveLatency.
  • These are not used anymore. They were only used for older versions of functional testing.
  • Removed to adhere to best security practices, minimize arbitrary shell invocation.

tools/etcd-dump-metrics

• Implement input validation to prevent arbitrary shell invocation.

Dependency

• Upgrade google.golang.org/grpc from v1.23.0 to v1.26.0.
• Upgrade go.uber.org/zap from v1.14.1 to v1.15.0.

Release

• Add s390x build support (PR#11548 and PR#11358)

Go

• Require Go 1.16+.
• Compile with Go 1.16
• etcd uses go modules (instead of vendor dir) to track dependencies.

Project Governance

• The etcd team has added, a well defined and openly discussed, project governance.

---