The cluster version will be initialized after the member becomes leader.
The update is handled asynchronously. It couldn't be updated if the member
has been closed and the go-runtime picks the `s.stopping` channel first.
```go
// e2a5df534c/server/etcdserver/server.go (L2170)
func (s *EtcdServer) monitorClusterVersions() {
...
for {
select {
case <-s.firstCommitInTerm.Receive():
case <-time.After(monitorVersionInterval):
case <-s.stopping:
return
}
...
}
}
```
Or after the `s.stopping` has been closed, the [UpdateClusterVersion][1] won't
file GoAttach successfully. For the #15409, we can see the warn log
`server has stopped; skipping GoAttach` from GoAttach:
```plain
https://github.com/etcd-io/etcd/actions/runs/4340931587/jobs/7580103902
logger.go:130: 2023-03-06T07:36:44.253Z WARN default stopping grpc server due to error {"error": "accept tcp 127.0.0.1:2379: use of closed network connection"}
logger.go:130: 2023-03-06T07:36:44.253Z WARN default stopped grpc server due to error {"error": "accept tcp 127.0.0.1:2379: use of closed network connection"}
logger.go:130: 2023-03-06T07:36:44.253Z ERROR default setting up serving from embedded etcd failed. {"error": "accept tcp 127.0.0.1:2379: use of closed network connection"}
logger.go:130: 2023-03-06T07:36:44.253Z ERROR default setting up serving from embedded etcd failed. {"error": "http: Server closed"}
logger.go:130: 2023-03-06T07:36:44.253Z INFO default skipped leadership transfer for single voting member cluster {"local-member-id": "8e9e05c52164694d", "current-leader-member-id": "8e9e05c52164694d"}
logger.go:130: 2023-03-06T07:36:44.253Z WARN default server has stopped; skipping GoAttach
...
```
If the cluster version isn't updated, the minimum storage version will
be v3.5 because the [AuthStatus][2] is introduced in [v3.5][3].
The compare will fail.
To fix this issue, we should wait for cluster version to become ready
after server is ready to serve request.
[1]: <e2a5df534c/server/etcdserver/adapters.go (L45)>
[2]: <071e70cdc4>
[3]: <1b4e54c238>
Signed-off-by: Wei Fu <fuweid89@gmail.com>
This is because etcd v3.5 will panic when it encounters
ClusterVersionSet entry with version >3.5.0. For downgrades to v3.5 to
work we need to make sure this entry is snapshotted.
