streamTypeMsgApp is only used in etcd 2.0. etcd 2.3 should not talk to
etcd 2.0, either send or receive requests. So I deprecate streamTypeMsgApp
and its related stuffs from rafthttp package.
updating term is only used from streamTypeMsgApp, so it is removed too.
Use snapshotSender to send v3 snapshot message. It puts raft snapshot
message and v3 snapshot into request body, then sends it to the target peer.
When it receives http.StatusNoContent, it knows the message has been
received and processed successfully.
As receiver, snapHandler saves v3 snapshot and then processes the raft snapshot
message, then respond with http.StatusNoContent.
rafthttp has different requirements for connections created by the
transport for different usage, and this is hard to achieve when giving
one http.RoundTripper. Pass into pkg the data needed to build transport
now, and let rafthttp build its own transports.
It specifies request timeout error possibly caused by connection lost,
and print out better log for user to understand.
It handles two cases:
1. the leader cannot connect to majority of cluster.
2. the connection between follower and leader is down for a while,
and it losts proposals.
log format:
```
20:04:19 etcd3 | 2015-08-25 20:04:19.368126 E | etcdhttp: etcdserver:
request timed out, possibly due to connection lost
20:04:19 etcd3 | 2015-08-25 20:04:19.368227 E | etcdhttp: etcdserver:
request timed out, possibly due to connection lost
```
The original workflow may fail to cancel if stop() cancels the finished
request just before dial() assigning a new cancel. This commit checks
streamReader status before setting cancel to avoid this problem.
It is tested at travis for 300 times. go 1.5 always works well, while
go 1.4 fails to stop once.
This problem is totally fixed at 1.5.
go1.5 adds a Request.Cancel channel, which allows for "race free"
cancellation
(8b4278ffb7).
Our implementation relies on it to always cancel in-flight request.
The original test code before fb4b0b5cf075212f79fcdfa58533915a29c0912d
doesn't work because reader side may update the
stream, while writer side writes message to the old stream and fails.
This PR removes unnecessary call to set term, and avoids this problem to
happen on term > 1 in the future.
Update term when AddPeer, or the term in peer will not be updated until
the term is changed. This fixes the log flood happended when the v2.1
follower applies the snapshot from v2.0 leader:
```
rafthttp: cannot attach out of data stream server [0 / 17]
```
or
```
rafthttp: server streaming to 6e3bd23ae5f1eae0 at term 0 has been
stopped
```
So it ensures that the message will not be discarded because the receive
side has not been ready, which happens easily in multiple core test.
Use log.fatal instead of log.error. The test exits when there is
something wrong because the error may affect following test cases.
before:
```
2015/06/12 20:06:19 rafthttp: dropped MsgApp from %!s(uint64=2) since
receiving buffer is full
```
after:
```
2015/06/12 13:51:38 rafthttp: dropped MsgProp from 2 since receiving
buffer is full
```
Refactor the fake cancel implementation.
The old one may cancel other in-flight message in random, which leaves
the original target message blocked forever.
ForceGosched() performs bad when GOMAXPROCS>1. When GOMAXPROCS=1, it
could promise that other goroutines run long enough
because it always yield the processor to other goroutines. But it cannot
yield processor to goroutine running on other processors. So when
GOMAXPROCS>1, the yield may finish when goroutine on the other
processor just runs for little time.
Here is a test to confirm the case:
```
package main
import (
"fmt"
"runtime"
"testing"
)
func ForceGosched() {
// possibility enough to sched up to 10 go routines.
for i := 0; i < 10000; i++ {
runtime.Gosched()
}
}
var d int
func loop(c chan struct{}) {
for {
select {
case <-c:
for i := 0; i < 1000; i++ {
fmt.Sprintf("come to time %d", i)
}
d++
}
}
}
func TestLoop(t *testing.T) {
c := make(chan struct{}, 1)
go loop(c)
c <- struct{}{}
ForceGosched()
if d != 1 {
t.Fatal("d is not incremented")
}
}
```
`go test -v -race` runs well, but `GOMAXPROCS=2 go test -v -race` fails.
Change the functionality to waiting for schedule to happen.
