Currently the only way we know that a peer isn't getting a heartbeat is
an edge triggered event from go raft on every missed heartbeat. This
means that we need to do some book keeping in order to do exponential
backoff.
The upside is that instead of screaming thousands of log lines before a
machine hits the default removal of 30 minutes it is only ~100.