291 Commits

Author SHA1 Message Date
Ori Newman
6f53da18b1
Increase stores cache (#1485)
Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-02-04 10:06:02 +02:00
Ori Newman
dbababb978
Limit mempool size to million transactions and remove the least profitable transactions (#1483)
* Limit mempool size to million transactions and remove the least profitable transactions

* Simplify insert

* Fix typo

* Improve findTxIndexInOrderedTransactionsByFeeRate readability
2021-02-03 19:45:39 +02:00
Ori Newman
238950cb98
Add logs (#1484)
* Add logs

* Fix log name
2021-02-03 17:47:59 +02:00
stasatdaglabs
97be133cee
Add logs to help debug long virtual parent selection. (#1470)
Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-02-01 19:15:50 +02:00
Ori Newman
669a9ab4c3
Ban by IP (#1471)
* Ban by IP

* Fix panic

* Fix error format

* Remove failed addresses

Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-02-01 10:51:18 +02:00
Ori Newman
a9a810a2b2
Add block type to MineJSON (#1464) 2021-01-28 13:22:20 +02:00
Michael Sutton
c9b591f2d3
Final reindex algorithm (#1430)
* Mine JSON

* [Reindex tests] add test_params and validate_mining flag to test_consensus

* Rename file and extend tests

* Ignore local test datasets

* Use spaces over tabs

* Reindex algorithm - full algorithm, initial commit, some tests fail

* Reindex algorithm - a few critical fixes

* Reindex algorithm - move reindex struct and all related operations to new file

* Reindex algorithm - added a validateIntervals method and modified tests to use it (instead of exact comparisons)

* Reindex algorithm - modified reindexIntervals to receive the new child as argument and fixed an important related bug

* Reindex attack tests - move logic to helper function and add stretch test

* Reindex algorithm - variable names and some comments

* Reindex algorithm - minor changes

* Reindex algorithm - minor changes 2

* Reindex algorithm - extended stretch test

* Reindex algorithm - small fix to validate function

* Reindex tests - move tests and add DAG files

* go format fixes

* TestParams doc comment

* Reindex tests - exact comparisons are not needed

* Update to version 0.8.6

* Remove TestParams and use AddUTXOInvalidHeader instead

* Use gzipeed test files

* This unintended change somehow slipped in through branch merges

* Rename test

* Move interval increase/decrease methods to reachability interval file

* Addressing a bunch of minor review comments

* Addressed a few more minor review comments

* Make code of offsetSiblingsBefore and offsetSiblingsAfter symmetric

* Optimize reindex logic in cases where reorg occurs + reorg test

* Do not change reindex root too fast (on reorg)

* Some comments

* A few more comments

* Addressing review comments

* Remove TestNoAttackAlternateReorg and assert chain attack

* Minor

Co-authored-by: Elichai Turkel <elichai.turkel@gmail.com>
Co-authored-by: Mike Zak <feanorr@gmail.com>
Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-01-27 17:09:20 +02:00
Elichai Turkel
01aee62cb0
Add log and measure to pruning points (#1457) 2021-01-27 11:40:51 +02:00
Elichai Turkel
0561347ff1
Remove default dns/grpc seeders (#1454) 2021-01-26 17:40:54 +02:00
stasatdaglabs
819ec9f2a7
Pass fromOutpoint into GetPruningPointUTXOs instead of offset, so it could Seek over the cursor (#1447)
* Implement TestGetPruningPointUTXOs.

* Fix a bad error message.

* Fix TestGetPruningPointUTXOs for testnet.

* Make sure all the UTXOs are returned in TestGetPruningPointUTXOs.

* Implement BenchmarkGetPruningPointUTXOs.

* Pass fromOutpoint into GetPruningPointUTXOs instead of offset, so it could Seek over the cursor.

* Fix weird benchmark timer calls.

* Remove unnecessary collection of outpointAndUTXOEntryPairs from BenchmarkGetPruningPointUTXOs.

* Fix a comment.
2021-01-25 13:38:59 +02:00
stasatdaglabs
ca04c049ab
When the pruning point moves, update its UTXO set outside of a database transaction (#1444)
* Remove pruningPointUTXOSetStaging and implement UpdatePruningPointUTXOSet.

* Implement StageStartSavingNewPruningPointUTXOSet, HadStartedSavingNewPruningPointUTXOSet, and FinishSavingNewPruningPointUTXOSet.

* Fix a bad return.

* Implement UpdatePruningPointUTXOSetIfRequired.

* Call UpdatePruningPointUTXOSetIfRequired on consensus creation.

* Rename savingNewPruningPointUTXOSetKey to updatingPruningPointUTXOSet.

* Add a log.

* Add calls to runtime.GC() at its start and end.

* Rename a variable.

* Replace calls to runtime.GC to calls to LogMemoryStats.

* Wrap the contents of LogMemoryStats in a log closure.
2021-01-24 14:48:11 +02:00
stasatdaglabs
756f40c59a
Sync pruning point UTXO sets incrementally instead of all at once (#1431)
* Replaced the content of MsgIBDRootUTXOSetChunk with pairs of outpoint-utxo entry pairs.

* Rename utxoIter to utxoIterator.

* Add a big stinky TODO on an assert.

* Replace pruningStore staging with a UTXO set iterator.

* Reimplement receiveAndInsertIBDRootUTXOSet.

* Extract OutpointAndUTXOEntryPairsToDomainOutpointAndUTXOEntryPairs into domainconverters.go.

* Pass the outpoint and utxy entry pairs to the pruning store.

* Implement InsertCandidatePruningPointUTXOs.

* Implement ClearCandidatePruningPointUTXOs.

* Implement UpdateCandidatePruningPointMultiset.

* Use the candidate pruning point multiset in updatePruningPoint.

* Implement CandidatePruningPointUTXOIterator.

* Use the pruning point utxo set iterator for StageVirtualUTXOSet.

* Defer ClearCandidatePruningPointUTXOs.

* Implement OverwriteVirtualUTXOSet.

* Implement CommitCandidatePruningPointUTXOSet.

* Implement BeginOverwritingVirtualUTXOSet and FinishOverwritingVirtualUTXOSet.

* Implement overwriteVirtualUTXOSetAndCommitPruningPointUTXOSet.

* Rename ClearCandidatePruningPointUTXOs to ClearCandidatePruningPointData.

* Add missing methods to dbManager.

* Implement PruningPointUTXOs.

* Implement RecoverUTXOIfRequired.

* Delete the utxoserialization package.

* Fix compilation errors in TestValidateAndInsertPruningPoint.

* Switch order of operations in the if statements in PruningPointUTXOs so that Next() wouldn't be unnecessarily called.

* Fix missing pruning point utxo set staging and bad slice length.

* Fix no default multiset in InsertCandidatePruningPointUTXOs.

* Make go vet happy.

* Rename candidateXXX to importedXXX.

* Do some more renaming.

* Rename some more.

* Fix bad MsgIBDRootNotFound logic.

* Fix an error message.

* Simplify receiveIBDRootBlock.

* Fix error message in receiveAndInsertIBDRootUTXOSet.

* Do some more renaming.

* Fix merge errors.

* Fix a bug caused by calling iterator.First() unnecessarily.

* Remove databaseContext from stores and don't use a transaction in ClearXXX functions.

* Simplify receiveAndInsertIBDRootUTXOSet.

* Fix offset count in PruningPointUTXOs().

* Fix readOnlyUTXOIteratorWithDiff.First().

* Split handleRequestIBDRootUTXOSetAndBlockFlow into smaller methods.

* Rename IbdRootNotFound to UnexpectedPruningPoint.

* Rename requestIBDRootHash to requestPruningPointHash.

* Rename IBDRootHash to PruningPointHash.

* Rename RequestIBDRootUTXOSetAndBlock to RequestPruningPointUTXOSetAndBlock.

* Rename IBDRootUTXOSetChunk to PruningPointUTXOSetChunk.

* Rename RequestNextIBDRootUTXOSetChunk to RequestNextPruningPointUTXOSetChunk.

* Rename DoneIBDRootUTXOSetChunks to DonePruningPointUTXOSetChunks.

* Rename remaining references to IBD root.

* Fix an error message.

* Add a check for HadStartedImportingPruningPointUTXOSet in commitVirtualUTXODiff.

* Add a check for HadStartedImportingPruningPointUTXOSet in ImportPruningPointUTXOSetIntoVirtualUTXOSet.

* Move FinishImportingPruningPointUTXOSet closer to HadStartedImportingPruningPointUTXOSet.

* Remove reference to pruningStore in utxoSetIterator.

* Pointerify utxoSetIterator receivers.

* Fix bad insert in CommitImportedPruningPointUTXOSet.

* Rename commitImportedPruningPointUTXOSetAll to applyImportedPruningPointUTXOSet.

* Simplify PruningPointUTXOs.

* Add populateTransactionWithUTXOEntriesFromUTXOSet.

* Fix a TODO comment.

* Rename InsertImportedPruningPointUTXOs to AppendImportedPruningPointUTXOs.

* Extract handleRequestPruningPointUTXOSetAndBlockMessage to a separate method.

* Rename stuff in readOnlyUTXOIteratorWithDiff.First().

* Address toAddIterator in readOnlyUTXOIteratorWithDiff.First().

* Call First() before any full iteration on ReadOnlyUTXOSetIterator.

* Call First() before any full iteration on a database Cursor.

* Put StartImportingPruningPointUTXOSet inside the pruning point transaction.

* Make serializeOutpoint and serializeUTXOEntry free functions in pruningStore.

* Fix readOnlyUTXOIteratorWithDiff.First().

* Fix bad validations in importPruningPoint.

* Remove superfluous call to validateBlockTransactionsAgainstPastUTXO.
2021-01-21 17:24:52 +02:00
Elichai Turkel
6a03d31f98
Remove accidental pointer indirection in dbKey (#1441) 2021-01-21 12:14:52 +02:00
Ori Newman
319ab6cfcd
Always request orphan roots, even when you get an inv of a known orphan (#1436)
Co-authored-by: Svarog <feanorr@gmail.com>
2021-01-20 10:58:45 +02:00
Svarog
ad9c213a06
Restructure database to prevent double-slashes in keys, causing bugs in cursors (#1432)
* Add TestValidateAndInsertPruningPointWithSideBlocks

* Optimize infrastracture bucket paths

* Update infrastracture tests

* Refactor the consensus/database layer

* Remove utils/dbkeys

* Use consensus/database in consensus instead of infrastructure

* Fix a bug in dbBucketToDatabaseBucket and MakeBucket combination

Co-authored-by: Elichai Turkel <elichai.turkel@gmail.com>
Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-01-19 14:19:08 +02:00
Svarog
a4adbabf96
TestBuildBlockErrorCases and remove redundant check of coinbase script length (#1427)
* Write structure of TestBlockBuilderErrorCases

* Remove double verification of script length in serializeCoinbasePayload

* Remove redundant code in TestBuildBlockErrorCases

* Rename povTransactionHash -> povBlockHash

* Convert coinbasePayloadScriptPublicKeyMaxLength to uint8

* Re-use consensus in TestBuildBlockErrorCases
2021-01-19 10:37:51 +02:00
Ori Newman
799eb7515c
Test validateAndInsertPruningPoint (#1420)
* Add TestValidateAndInsertPruningPoint

* Check fake UTXO set and validate that the pruning point changed
2021-01-18 18:17:13 +02:00
Ori Newman
d4f3a252ff
Add TestIsFinalizedTransaction (#1422)
Co-authored-by: Svarog <feanorr@gmail.com>
2021-01-17 15:47:49 +02:00
Ori Newman
dd57e6abe6
Fix checkParentHeadersExist and cover pruning_violation_proof_of_work_and_difficulty.go with tests (#1418)
* Fix checkParentHeadersExist and cover pruning_violation_proof_of_work_and_difficulty.go with tests

* Remove unused variable

* Change consensus violation

* Change condition order

* Get rid of irrelevant error codes in extractRejectCode

* Fix wrong test db names

* Fix checkParentHeadersExist
2021-01-17 11:27:04 +02:00
Ori Newman
67be4d82bf
Don't mark bad merkle root as invalid (#1419)
* Don't mark bad merkle root as invalid

* Fix TestBlockStatus

* Move discardAllChanges inside the inner if
2021-01-17 10:40:05 +02:00
Ori Newman
a1381d6768
Add TestCheckParentBlockBodiesExist (#1405)
* Add TestCheckParentBlockBodiesExist

* Use block in pruning point's anticone for the test

* Fix test db name
2021-01-14 13:31:17 +02:00
Ori Newman
10b519a3e2
Add tests to ValidateHeaderInIsolation (#1415)
* Add tests to ValidateHeaderInIsolation

* Fix tests db names

Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-01-14 11:06:08 +02:00
Ori Newman
a35f8269ea
Add checkBlockIsNotPruned (#1413)
* Add checkBlockIsNotPruned

* Fix test name and comment

Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-01-14 10:56:22 +02:00
Svarog
1b97cfb302
Prevent a race condition in findHighestSharedBlockHash where we get headersSelectedTip and then pass it as highHash to GetBlockLocator, without locking consensus (#1410)
* Prevent a race condition in findHighestSharedBlockHash where we get headersSelectedTip and then pass it as highHash to GetBlockLocator, without locking consensus

* Restart findHighestSharedBlockHash if lowHash or highHash are no longer in selectedParentChain

* Test for specifically ErrBlockNotInSelectedParentChain instead of database NotFound error

* Fix TestCreateHeadersSelectedChainBlockLocator

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-01-13 17:55:37 +02:00
Ori Newman
61be80a60c
Add TestCheckMergeSizeLimit (#1408)
Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-01-13 16:19:52 +02:00
Elichai Turkel
68bd8330ac
Log the networks hashrate (#1406)
* Log the hashrate of each block

* Add a test for GetHashrateString

* Move difficulty related functions to its own package

* Convert the validated log in validateAndInsertBlock to a log function

* Add tests for max/min int
2021-01-13 12:51:23 +02:00
Svarog
ce348373c6
Delete existing UTXOSet when commiting VirtualUTXOSet. (#1403) 2021-01-12 16:51:56 +02:00
talelbaz
8ad5725421
Adding a test for the error cases on the function 'checkBlockStatus()' (#1398)
* Adds test for error cases on the function checkBlockStatus.

* Fix review's comments.

* Move test to validateandinsertblock_test.go

Co-authored-by: tal <tal@daglabs.com>
Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-01-12 15:55:02 +02:00
Ori Newman
23a2fbf401
Remove erroneous finality optimization from LowestChainBlockAboveOrEqualToBlueScore (#1402)
* Remove finality erroneous optimization from LowestChainBlockAboveOrEqualToBlueScore

* Add TestLowestChainBlockAboveOrEqualToBlueScore

* Remove unnecessary fields from dagTraversalManager
2021-01-12 15:27:08 +02:00
Elichai Turkel
c1361e5b3e
Change log sizes and add some new features to logger (#1400)
* Increase default log sizes, and increase kaspad log sizes

* Add an option to not print logs to stdout

* Allow logs to be printed in the current working directory

* Add more pruning related logs

* Add comment and increase log rotations to save last 64 logs
2021-01-12 13:26:29 +02:00
Ori Newman
bcf2302460
Add high hash to block locator, and add block locator tests (#1397)
* Include high hash in the block locator

* Add tests for block locator

* Remove redundant function

* Remove redundant assignments
2021-01-12 11:16:25 +02:00
Ori Newman
6101e6bdb6
Fix UTXO serialization, its test, and the static check that missed it (#1396)
* Fix UTXO serialization, its test, and the static check that missed it

* Remove duplicate case

* Use one line for static check

Co-authored-by: Elichai Turkel <elichai.turkel@gmail.com>
2021-01-11 17:45:17 +02:00
stasatdaglabs
d9b97afb92
Don't swallow errors in HandleNewBlockTransactions. (#1390)
Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-01-11 17:16:15 +02:00
Ori Newman
b8ca33d91d
Add selected chain store and optimize block locator with it (#1394)
* Add selected chain store and optimize block locator with it

* Fix build error

* Fix comments

* Fix IsStaged

* Rename CalculateSelectedParentChainChanges to CalculateChainPath and SelectedParentChainChanges->SelectedChainPath

* Use binary.LittleEndian directly to allow compiler optimizations

* Remove boolean from HeadersSelectedChainStore interface

* Prevent endless loop in block locator
2021-01-11 15:51:45 +02:00
Svarog
c7deda41c6
Fix deserialization of script version in UTXOSet deserialization (#1395)
* Initalize protoUTXOSetIterator with index = -1

* Handle error when failed to deserialize Script version

* Add support for (de)serialization of (u)int16

* Log the error when converting it into ErrMalformedUTXO
2021-01-11 15:23:27 +02:00
talelbaz
434cf45112
Adds a new test to validate POW, and Fix Main-net and Test-net genesis block data. (#1389)
* commit for do fetch&merge

* Adds a new test to validate POW, and Fix Main-net and Testnet genesis block data.

* Fix window's test for testnet and change the expected pruning point for mainnet and testnet.

* Delete function "solveBlock" on proof_of_work_test.go and call the function mining.SolvaBlock instead. Also, remove using of random in "solveBlockWithWrongPOW" function.

* Replace 0xFFFFFFFFFFFFFFFF to math.MaxUint64 in "solveBlockWithWrongPOW" function and change the function's comment of "TestPOW"

* Replace 0xFFFFFFFFFFFFFFFF to math.MaxUint64 in "solveBlockWithWrongPOW" function and change the function's comment of "TestPOW"

* Change from <= to < in the for statement in "solveBlockWithWrongPOW" function

* Adds one arg to the function call "NewTestConsensus" (the function sig has changed).

Co-authored-by: tal <tal@daglabs.com>
Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-01-11 13:15:26 +02:00
Elichai Turkel
2cc0bf1639
Optimize block locator using finality store (#1386)
* Make sure block locator doesn't include a hash lower than the lowHash in
the block locator

* Use finalityStore to optimize LowestChainBlockAboveOrEqualToBlueScore
2021-01-10 13:36:02 +02:00
Svarog
49e0a2a2e7
Add basic support for archival node (#1370)
* Add archival cli flag

* If --archival was activated - don't delete anything

* Fix tests

* Still change block status to StatusHeaderOnly even in archival nodes
2021-01-10 10:25:15 +02:00
Ori Newman
4ad89056c2
Implement getMempoolEntries (#1369)
* Implement getMempoolEntries

* Fix comment

* Fix comment
2021-01-06 17:26:21 +02:00
Svarog
9ea4c0fa38
Add sanity check that makes sure that the utxoset that we save fits the pruningPoints's commitment. (#1366)
Co-authored-by: Elichai Turkel <elichai.turkel@gmail.com>
2021-01-06 13:27:02 +02:00
Elichai Turkel
a04a5462ae
Update devnet genesis with older timestamp (#1364)
* Update devnet genesis with older timestamp

* Update BlueWindow test for new genesis
2021-01-06 12:50:27 +02:00
Elichai Turkel
4577023e44
Add script pubkey version to signature hash (#1360)
* Replace 0xffff with math.MaxUint16 on version checks

* Add script version to the signature hash

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-01-06 11:51:12 +02:00
Ori Newman
d8293ef635
Revert "Add recoverability for UTXO index (#1342)" (#1353)
This reverts commit 778375c4
2021-01-06 11:41:52 +02:00
Ori Newman
3ec1cbe236
Add TestBlockStatus (#1361)
Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-01-06 10:37:10 +02:00
Ori Newman
741e0962be
Remove diffs from restoreUTXO logs (#1354)
Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-01-06 10:31:42 +02:00
Elichai Turkel
6279db2bf1
Reverse ghostdag tie break direction (#1359)
* Remove hash reversal in ghostdag

* Update tests after chaning ghostdag hash direction

Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-01-05 19:10:59 +02:00
stasatdaglabs
e24bc527f3
Update the max block size and mass constants to reasonable values (#1344)
Co-authored-by: Elichai Turkel <elichai.turkel@gmail.com>
2021-01-05 18:36:08 +02:00
talelbaz
8a309a7d2a
Upgradability mechanisms script version (#1313)
* ''

* ''

* ''

* Changes genesis block version to 0.

* a

* a

* All tests are done.

* All tests passed for changed block version from int32 to uint16

* Adds validation of rejecting blocks with unknown versions.

* Changes txn version from int32 to uint16.

* .

* Adds comments to exported functions.

* Change functions name from ConvertFromRpcScriptPubKeyToRPCScriptPubKey to ConvertFromAppMsgRPCScriptPubKeyToRPCScriptPubKey and from ConvertFromRPCScriptPubKeyToRpcScriptPubKey to ConvertFromRPCScriptPubKeyToAppMsgRPCScriptPubKey

* change comment to "ScriptPublicKey represents a Kaspad ScriptPublicKey"

* delete part (tx.Version < 0) that cannot be exist on the if statement.

* Revert protobuf version.

* Fix a comment.

* Fix a comment.

* Rename a variable.

* Rename a variable.

* Remove a const.

* Rename a type.

* Rename a field.

* Rename a field.

* Remove commented-out code.

* Remove dangerous nil case in DomainTransactionOutput.Clone().

* Remove a constant.

* Fix a string.

* Fix wrong totalScriptPubKeySize in transactionMassStandalonePart.

* Remove a constant.

* Remove an unused error.

* Fix a serialization error.

* Specify version types to be uint16 explicitly.

* Use constants.ScriptPublicKeyVersion.

* Fix a bad test.

* Remove some whitespace.

* Add a case to utxoEntry.Equal().

* Rename scriptPubKey to scriptPublicKey.

* Remove a TODO.

* Rename constants.

* Rename a variable.

* Add version to parseShortForm.

Co-authored-by: tal <tal@daglabs.com>
Co-authored-by: stasatdaglabs <stas@daglabs.com>
2021-01-05 17:50:09 +02:00
Elichai Turkel
70d515a5a9
PruningManager: Delete tips that are in pruningPoint.Anticone from the tips list (#1351) 2021-01-05 14:14:40 +02:00
Elichai Turkel
72a7ca53e6
Save and expose the database in TestConsensus (#1349) 2021-01-05 14:13:33 +02:00