ec13797407
The watch count metrics are not robust to duplicate cancellations. These cause the count to be decremented twice, leading eventually to negative counts. We are seeing this in production. The duplicate cancellations themselves are not themselves a big problem (except performance), but they are caused by the new proactive cancellation logic (#11850) which cancels proactively even immediately before initiating a Close, thus nearly guaranteeing a Close-cancel race, as discussed in watchable_store.go. We can avoid this in most cases by not sending a cancellation when we are going to Close.
etcd/clientv3
etcd/clientv3 is the official Go etcd client for v3.
Install
go get go.etcd.io/etcd/clientv3
Get started
Create client using clientv3.New:
cli, err := clientv3.New(clientv3.Config{
Endpoints: []string{"localhost:2379", "localhost:22379", "localhost:32379"},
DialTimeout: 5 * time.Second,
})
if err != nil {
// handle error!
}
defer cli.Close()
etcd v3 uses gRPC for remote procedure calls. And clientv3 uses
grpc-go to connect to etcd. Make sure to close the client after using it.
If the client is not closed, the connection will have leaky goroutines. To specify client request timeout,
pass context.WithTimeout to APIs:
ctx, cancel := context.WithTimeout(context.Background(), timeout)
resp, err := cli.Put(ctx, "sample_key", "sample_value")
cancel()
if err != nil {
// handle error!
}
// use the response
For full compatibility, it is recommended to vendor builds using etcd's vendored packages, using tools like golang/dep, as in vendor directories.
Error Handling
etcd client returns 2 types of errors:
- context error: canceled or deadline exceeded.
- gRPC error: see api/v3rpc/rpctypes.
Here is the example code to handle client errors:
resp, err := cli.Put(ctx, "", "")
if err != nil {
switch err {
case context.Canceled:
log.Fatalf("ctx is canceled by another routine: %v", err)
case context.DeadlineExceeded:
log.Fatalf("ctx is attached with a deadline is exceeded: %v", err)
case rpctypes.ErrEmptyKey:
log.Fatalf("client-side error: %v", err)
default:
log.Fatalf("bad cluster endpoints, which are not etcd servers: %v", err)
}
}
Metrics
The etcd client optionally exposes RPC metrics through go-grpc-prometheus. See the examples.
Namespacing
The namespace package provides clientv3 interface wrappers to transparently isolate client requests to a user-defined prefix.
Request size limit
Client request size limit is configurable via clientv3.Config.MaxCallSendMsgSize and MaxCallRecvMsgSize in bytes. If none given, client request send limit defaults to 2 MiB including gRPC overhead bytes. And receive limit defaults to math.MaxInt32.
Examples
More code examples can be found at GoDoc.