Based on the ideas discussed in the issues [1] and PRs [2][3][6], it's
been decided to switch from using an L4 reverse proxy to an L7 forward
proxy to properly block peer network traffic, without the need to use
external tools.
The design aims to implement only the minimal required features that
satisfy blocking incoming and outgoing peer traffic. Complicated features
such as packet reordering, packet delivery delay, etc. to a future
container-based solution.
A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).
Thus, the current mechanism of only blocking peer traffic via the peer's
existing reverse proxy is insufficient, since only scenario (a) is
handled, and network traffic in scenario (b) is not blocked at all.
We introduce an L7 forward proxy for each peer, which will be proxying
all the connections initiated from a peer to its peers.
We will remove the current use of the L4 reverse proxy, as the L7
forward proxy holds the information of the destination, we can block all
incoming and outgoing traffic that is initiated from a peer to others,
without having to resort to external tools, such as iptables.
The modified architecture will look something like this:
```
A --- A's forward proxy --- B
^ newly introduced
```
The main subtasks are
- redesigned as an L7 forward proxy
- introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP` to
bypass the limitation of `http.ProxyFromEnvironment`
- implement an L7 forward proxy
Known limitations are
- Doesn't support unix socket, as L7 HTTP transport proxy only supports
HTTP/HTTPS/and socks5 -> Although the e2e test supports unix sockets for
peer communication, only a few of the e2e test cases use unix sockets as
the majority of e2e test cases use HTTP/HTTPS. It's been discussed and
decided that it is ok for now without the unix socket support.
- it's L7 so we need to send a perfectly crafted HTTP request
- doesn’t support reordering, dropping, or manipulating packets on the
fly
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast`
[1] issue https://github.com/etcd-io/etcd/issues/17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) https://github.com/etcd-io/etcd/pull/17891
[4] https://github.com/etcd-io/etcd/pull/17938#discussion_r1615622709
[5] https://github.com/etcd-io/etcd/pull/17985#discussion_r1598020110
[6] https://github.com/etcd-io/etcd/pull/17938
Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.
Signed-off-by: Siyuan Zhang <sizhang@google.com>
Signed-off-by: Iván Valdés Castillo <iv@nvald.es>
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Part of the patches to fix https://github.com/etcd-io/etcd/issues/17737
During the development of https://github.com/etcd-io/etcd/pull/17938,
we agreed that during the transition to L7 forward proxy, unused
features and features targeting L4 reverse proxy will be dropped.
This feature falls under the unused feature.
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Part of the patches to fix https://github.com/etcd-io/etcd/issues/17737
During the development of https://github.com/etcd-io/etcd/pull/17938,
we agreed that during the transition to L7 forward proxy, unused
features and features targeting L4 reverse proxy will be dropped.
This feature falls under the unused feature. Also, the initial
implementation has a bug: if connections are not created continuously,
the latency accept will not work. Consider the following case:
a) set latency accept
b) put latency accept into effect
c) latency accept will start idling the goroutine
d) block-wait at accept() - waiting for new connections
e) new connection comes in - establish it
f) go to c -> as we can see, if the request come every x seconds, where
x is larger than the latency accept time we set, we can see that the
latency accept has no effect.
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Part of the patches to fix https://github.com/etcd-io/etcd/issues/17737
During the development of https://github.com/etcd-io/etcd/pull/17938,
we agreed that during the transition to L7 forward proxy, unused
features and features targeting L4 reverse proxy will be dropped.
This feature falls under the unused feature.
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Part of the patches to fix https://github.com/etcd-io/etcd/issues/17737
During the development of https://github.com/etcd-io/etcd/pull/17938,
we agreed that during the transition to L7 forward proxy, unused
features and features targeting L4 reverse proxy will be dropped.
This feature falls under the unused feature.
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
This job was migrated to the prow infrastructure in May
(https://github.com/kubernetes/test-infra/pull/32672). Deleting the
GitHub workflow to remove duplicate jobs should be fine.
Signed-off-by: Ivan Valdes <ivan@vald.es>
As `tools/mod` also contains the `go.mod` file. We should add it to the
`module_dirs` variable, so that when executing `./scripts/fix.sh`, the
proper checks and fixes can be applied.
To address the issue of broken unit tests and code coverage due to the
directory's lack of Go code, we've introduced a new doc.go file. This
file acts as a placeholder, enabling tools like golangci-lint and
go test to function correctly.
---
Discovered when working on https://github.com/etcd-io/etcd/pull/18575
The directories are checked against the following:
- Command: `find . -type f -name go.mod -exec dirname {} \;`
- Output:
```
./etcdutl
.
./tools/testgrid-analysis
./tools/rw-heatmaps
./tools/mod
./etcdctl
./tests
./server
./api
./client/internal/v2
./client/v3
./client/pkg
./pkg
```
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
