Set GOTOOLCHAIN directive in order to successfully run tests
from root. Else, go will try and download a family of releases
(of the form 1.x), which are not published binaries.
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Discovered turning off LazyFS before creating the report might result in
empty server directory. This PR moves cluster shutdown to defer executed
after we generate report and copies the data from lazyfs directory.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
Simplifying bookmarkable to just validate revision order between events
and progress notifies.
Use reliable to validate if events are missing, but still report
broken resumable if first event after revision is missing. It's easier
to have one place that validates event slices.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
Limit can cause multiple request due to pagination.
For reads after a failed write we would like to return to normal write
request as soon as possible.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
The old leader demotes lessor and all the leases' expire time will be
updated. Instead of returning incorrect remaining TTL, we should return
errors to force client retry.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Kubernetes uses WithRequireLeader to modify the context passed
to the Watch() and RequestProgress() calls in order to ensure
that a leader is present in the cluster before serving the request.
This commit mimics that behaviour in the Kubernetes traffic.
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
We can check for the condition that Kubernetes checks for, i.e.
prevKV can be nil iff the event is not a create a event, only if
we know whether compaction has occured or not. If compaction has
occured, prevKV can be nil and that is completely valid.
This commit checks if compaction took place during the test run
by querying the /metrics endpoint. Based on if compaction occured,
we now check the Kubernetes condition in the prevKV robustness test.
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Split off valdiating create events from the prevKV test.
The added test tests the following two:
- A create event should not exist in our past history
- A non-create event should exist in our past history
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Event histories after merging serve as an authorotative list of
events that can be seen as ones persisted by etcd, we don't need
PrevValue as part of it.
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Kubernetes relies on the PrevKV() option in the watches it opens
against etcd. This commit adds a robustness test to validate the
same.
A watch response returned with PrevKV() is valid if:
The value in current event's prevKV matches the previous
event's value of the same key if this is not a create event.
There are cases where there can be a prevKV for a create event
as well, for example if a watch is opened after the key is creatd.
Since we don't simulate for that, we don't check for that.
Further, this adjusts revision numbers such that we can successfully create
a new replay. Needed now since we will have unit tests with
and without PrevKV co-existing and we requite creation of a
new replay everytime we validate PrevKV.
We also regenerate test data with so that prevKV exists in it
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Creation of event history requires each client to return consistent
events. If clients observed inconsistent view of some revision, merging
will fail and return diff between two clients. This however doesn't
provide hint on what kind of issue happend.
This PR helps cases where there is an error with single watch
stream (like event duplication) by running normal watch validation even
without full event history.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
If one of nodes is unhealthy the test would never finish as watchers
would never reach max revision.
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
