62 Commits

Author SHA1 Message Date
Marek Siarkowicz
a36951ad18 tests: Extract report related functions and clean up main file
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-02-15 16:13:43 +01:00
Marek Siarkowicz
116a3150c0
Merge pull request #15282 from serathius/linearizability-report-watch
test: Report watch histories
2023-02-15 16:00:10 +01:00
Marek Siarkowicz
0cd5c9ca37 tests: Refactor reporting results
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-02-15 13:02:42 +01:00
Marek Siarkowicz
d99b1dbdaf tests: Move results reporting to top and add reporting watch histories
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-02-15 13:02:42 +01:00
Marek Siarkowicz
58d74e2b73 test: Refactor TestLinearizability function
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-02-15 13:02:42 +01:00
Marek Siarkowicz
a64263cf49 tests: Refactor persisting single member dir
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-02-15 13:02:42 +01:00
Marek Siarkowicz
d0e5c44f67 tests: Refactor getting test results directory
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-02-15 13:02:42 +01:00
Marek Siarkowicz
2afaddd5b7 tests: Refactor getting longest history
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-02-15 13:02:42 +01:00
Bogdan Kanivets
d2ccb12f36 tests: use monotonic clock for linearizability test history
Problem: time.Now() uses wall clock reading. See https://github.com/golang/go/blob/master/src/time/time.go#L17
"later time-telling operations use the wall clock reading, but later time-measuring operations, specifically comparisons and subtractions, use the monotonic clock reading."
This can cause 'Return' to be before 'Call' and wrong order of operations from different clients.

Solution: use same base time for all clients and only use 'time-measuring' operations to record timestamps for history.
Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
2023-02-15 01:37:21 -08:00
Marek Siarkowicz
04ba936ea6 tests: Implement multi operation Txn
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-02-14 17:57:52 +01:00
Benjamin Wang
b4dfa9d34f test: correct the logic of comparing revision in linearizablity/watch.go
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2023-02-13 19:22:31 +08:00
Marek Siarkowicz
d0575e4f18
Merge pull request #15268 from lavacat/main-before-follower-send-failpoint
tests: trigger raftBeforeFollowerSend failpoint in linearizability test
2023-02-09 09:45:58 +01:00
Bogdan Kanivets
4b91ae2800 tests: trigger raftBeforeFollowerSend failpoint in linearizability test
raftBeforeFollowerSend can only be triggered on Follower and won't work on 1 node cluster.
Had to split RandomFailpoint into RandomOneNodeClusterFailpoint and RandomMultiNodeClusterFailpoint

Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
2023-02-08 23:28:23 -08:00
Marek Siarkowicz
39d98522d6 tests: Enable progress notify in linearizability tests
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-02-09 07:32:54 +08:00
Marek Siarkowicz
189d90ab7b tests: Validate watch responses in linearizability tests
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-02-06 16:58:17 +01:00
Bogdan Kanivets
91b056968d tests linearizability: trigger snapshot related failpoints
Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>
2023-02-05 00:03:39 -08:00
Wei Fu
2183a55b47 tests: forcestop member procs with signal kill
When the linearizability test cases run with three members, it might take
7-8s to stop three members, especially stopping the leader. The leader
will transfer the leadership and it might take more time to stop peer
listener.

In order to reduce the runtime, this commit is using signal kill to force
stop members instead of graceful shutdown.

REF: #15086

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-02-03 23:06:40 +08:00
Wei Fu
5a249fb3e2 tests/linearizability: remove PeerProxy for ClusterSize=1
Skip the peer traffic failpoint.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-02-01 18:04:11 +08:00
Marek Siarkowicz
d814adbf94
Merge pull request #15227 from serathius/linearizability-trigger-fail
Linearizability disable defragment request in traffic to remove flakes
2023-02-01 09:26:08 +01:00
Marek Siarkowicz
90b839dda7 tests: Disable defrag command as it caused problems with defragBeforeCopy failpoint
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-31 12:48:23 +01:00
Marek Siarkowicz
0c004a6ce4 tests: Propagage logger through linearizability tests
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-30 14:30:21 +01:00
Marek Siarkowicz
ef0bdbe489 tests: Implement LargePut requests
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-24 22:29:31 +01:00
Marek Siarkowicz
c50e602458 tests: Add defragment request to linearizability tests
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-24 21:49:57 +01:00
Marek Siarkowicz
cc612821e7 tests: Deduplicate cluster test scenarios
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-24 18:41:42 +01:00
Marek Siarkowicz
45e11687c4 tests: Stop cluster after tests finish
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-24 15:48:01 +01:00
Marek Siarkowicz
844ac9c76d tests: Separate request type from Txn operation type
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-23 22:59:28 +01:00
Marek Siarkowicz
a581062c7a tests: Fix linearizability nightly
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-20 09:13:51 +01:00
Benjamin Wang
2af9d67943
Merge pull request #15133 from serathius/linearizability-history-persist
tests: Persist operation history
2023-01-19 05:08:49 +08:00
Marek Siarkowicz
f629d48600 Close file while persisting operation history
Co-authored-by: Benjamin Wang <wachao@vmware.com>
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-18 09:43:30 +01:00
Marek Siarkowicz
a0d12d316d tests: Add reproduce #13766 scenario
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-17 14:34:19 +01:00
Marek Siarkowicz
f763c53d51 tests: Persist operation history
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-17 11:58:56 +01:00
Marek Siarkowicz
96e2a7fbd6 tests: Move linearizability model and identity to dedicated packages
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-16 19:16:27 +01:00
Marek Siarkowicz
1257d0d919 tests: Fix history patch window
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-13 16:19:20 +01:00
Marek Siarkowicz
7b2dfece70 tests: Use watch events to patch history to speed up linearization
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-12 09:31:23 +01:00
Marek Siarkowicz
3306639b76
Merge pull request #15080 from geetasg/lease_support_refactor
Add support for lease api to linearizability tests
2023-01-11 15:34:36 +01:00
Geeta Gharpure
5b84526e9a Add support for lease api to linearizability tests
Signed-off-by: Geeta Gharpure <geetagh@amazon.com>
2023-01-11 05:19:57 +00:00
Marek Siarkowicz
8a9f848d33 tests: Refactor how linearizability test components are run in parallel
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-10 15:12:12 +01:00
Marek Siarkowicz
6821e226dd tests: Watch events during linearizability test and compare history
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-10 15:11:23 +01:00
Marek Siarkowicz
deb4291485 tests: Trigger only failpoints available in binary
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-09 17:29:38 +01:00
Marek Siarkowicz
acf3782c20
Merge pull request #15048 from serathius/linearizability-network
tests: Implement network delay and blackholing in linearizability tests
2023-01-09 14:51:54 +01:00
Marek Siarkowicz
064fad5ae4 tests: Implement network delay and blackholing in linearizability tests
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2023-01-09 14:18:07 +01:00
Piotr Tabor
9abc895122 Goimports: Apply automated fixing to test files as well.
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-29 13:04:45 +01:00
Marek Siarkowicz
b108fabf86 tests: Reproduce issue 14685
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-12-24 21:26:57 +01:00
Marek Siarkowicz
d8da98c49f tests: Fix return time for failed requests globally
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-12-06 10:11:22 +01:00
Marek Siarkowicz
45fdc2bbac tests: Extract history from client
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-12-06 10:11:22 +01:00
Marek Siarkowicz
68fd863c87 tests: Mark failed requests as timed out
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-12-06 10:11:22 +01:00
Marek Siarkowicz
6a68141db5 tests: Allow dynamic number of clients
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-12-06 10:11:22 +01:00
Marek Siarkowicz
a573d8af69 tests: Use golang count to repeat tests
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-12-03 11:21:07 +01:00
Marek Siarkowicz
7f70b8cf76 tests: Separate retries count from failure trigger count
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-12-03 11:21:07 +01:00
Marek Siarkowicz
df2d075e1a tests: Sleep between failpoint failure
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
2022-12-03 11:21:07 +01:00