Based on Fu Wei's idea discussed in the issue [1], we employ the network traffic blocking on L7, using a forward proxy, without the need to use external tools.
[Background]
A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).
Thus, the current mechanism of only blocking peer traffic via the peer's existing reverse proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all.
[Proposed solution]
We introduce a forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers.
We will remove the current use of the reverse proxy, as the forward proxy holds the information of the destination, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables.
The modified architecture will look something like this:
```
A --- A's forward proxy ----- B
^ newly introduced
```
It's verified that the blocking of traffic is complete, compared to previous solutions attempted in PRs [2][3].
[Implementation]
The main subtasks are
- redesigned as an L7 forward proxy
- Unix socket support is dropped: e2e test supports unix sockets for peer communication, but only several e2e test cases use Unix sockets as majority of e2e test cases use HTTP/HTTPS
- introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP`
- implement L7 forward proxy by drastically reducing the existing proxy server code and design to use blocking traffic
Known limitations are
- Doesn't support unix socket (L7 HTTP transport proxy only supports HTTP/HTTPS/and socks5)
- It's L7 so we need to send a perfectly crafted HTTP request
-Doesn’t support reordering, dropping, etc. packets on-the-fly
[Testing]
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast`
[References]
[1] issue https://github.com/etcd-io/etcd/issues/17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) https://github.com/etcd-io/etcd/pull/17891
[4] https://github.com/etcd-io/etcd/pull/17938#discussion_r1615622709
[5] https://github.com/etcd-io/etcd/pull/17985#discussion_r1598020110
Signed-off-by: Siyuan Zhang <sizhang@google.com>
Co-authored-by: Iván Valdés Castillo <iv@nvald.es>
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
