From 6f0a67603a7aba822c8e3af395fba5b33fd39bf9 Mon Sep 17 00:00:00 2001 From: Anthony Romano Date: Thu, 7 Jul 2016 17:07:18 -0700 Subject: [PATCH] Documentation: tuning advice for peer prioritization --- Documentation/tuning.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/Documentation/tuning.md b/Documentation/tuning.md index dbd5c6b3e..f1cdc38cc 100644 --- a/Documentation/tuning.md +++ b/Documentation/tuning.md @@ -71,4 +71,23 @@ $ etcd --snapshot-count=5000 $ ETCD_SNAPSHOT_COUNT=5000 etcd ``` +## Network + +If the etcd leader serves a large number of concurrent client requests, it may delay processing follower peer requests due to network congestion. This manifests as send buffer error messages on the follower nodes: + +``` +dropped MsgProp to 247ae21ff9436b2d since streamMsg's sending buffer is full +dropped MsgAppResp to 247ae21ff9436b2d since streamMsg's sending buffer is full +``` + +These errors may be resolved by prioritizing etcd's peer traffic over its client traffic. On Linux, peer traffic can be prioritized by using the traffic control mechanism: + +``` +tc qdisc add dev eth0 root handle 1: prio bands 3 +tc filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip sport 2380 0xffff flowid 1:1 +tc filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip dport 2380 0xffff flowid 1:1 +tc filter add dev eth0 parent 1: protocol ip prio 2 u32 match ip sport 2739 0xffff flowid 1:1 +tc filter add dev eth0 parent 1: protocol ip prio 2 u32 match ip dport 2739 0xffff flowid 1:1 +``` + [ping]: https://en.wikipedia.org/wiki/Ping_(networking_utility)