15 Commits

Author SHA1 Message Date
Wei Fu
fc1863086c tests/robustness: unlock Delete/LeaseRevoke ops
We should return token to that bucket if `nonUniqueWriteLimiter.Take()`
return true. After unlock Delete/LeaseRevoke ops, the model should be
updated for replay function. There are two updates for `toWatchEvents`.

1. When leaveRevokes op has deleted few keys, we should generate
   `delete-operation` events based on alphabetical order of deleted
   keys.
2. When putWithLease op hits non-exist lease, we should ignore that
   update event.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2024-06-06 20:40:08 +08:00
Marek Siarkowicz
d8bb19327b Prevent picking a failpoint that waiting till snapshot that doesn't support lower snapshot catchup entries but allow reproducing issue #15271
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2024-05-13 12:08:42 +02:00
Marek Siarkowicz
c4e3b61a1c Record operation from failpoint injection
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2024-05-01 19:20:22 +02:00
Marek Siarkowicz
9fcde37447 Persist member data with lazyfs enabled
Discovered turning off LazyFS before creating the report might result in
empty server directory. This PR moves cluster shutdown to defer executed
after we generate report and copies the data from lazyfs directory.

Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2024-04-23 10:19:05 +02:00
Marek Siarkowicz
f285330d46 Don't require minimal for failpoint injection period
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2024-04-20 10:34:51 +02:00
Marek Siarkowicz
569693be8d Utilize WAL to patch operation history
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2024-04-14 12:09:38 +02:00
Marek Siarkowicz
3a23994fbf Make no failpoint error more readable
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2024-04-07 15:13:59 +02:00
Marek Siarkowicz
e2bb8c698f Limit a timeout in testing robustness validation
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2024-04-06 12:28:57 +02:00
Ivan Valdes
0976398964
tests/robustness: address golangci var-naming issues
Signed-off-by: Ivan Valdes <ivan@vald.es>
2024-03-25 16:27:05 -07:00
Marek Siarkowicz
3a351c2fec Revert "tests/robustness: check for compaction before prevKV validation"
This reverts commit 5d7f58d14be1da5135d158dad6fc43391cbf6283.

Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2024-02-21 16:15:40 +01:00
Madhav Jivrajani
5d7f58d14b tests/robustness: check for compaction before prevKV validation
We can check for the condition that Kubernetes checks for, i.e.
prevKV can be nil iff the event is not a create a event, only if
we know whether compaction has occured or not. If compaction has
occured, prevKV can be nil and that is completely valid.

This commit checks if compaction took place during the test run
by querying the /metrics endpoint. Based on if compaction occured,
we now check the Kubernetes condition in the prevKV robustness test.

Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
2024-02-19 17:05:59 +05:30
Marek Siarkowicz
f0d73c9d12 Separate robustness test scenarios and increase number of times we run exploratory tests in nightly
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2024-01-16 17:17:54 +01:00
Marek Siarkowicz
5175652a8e Abort if failpoint injecton failed
If one of nodes is unhealthy the test would never finish as watchers
would never reach max revision.

Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-12-03 17:26:51 +01:00
Marek Siarkowicz
452e820516 Make the main_test the entrypoint and move senario generation to separate file
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-10-16 22:10:41 +02:00
Marek Siarkowicz
d475cf81a0 tests: Rename linearizability tests to robustness
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-02-26 14:36:18 +01:00