189 Commits

Author SHA1 Message Date
Svarog
c13a4d90ed
Mempool redesign (#1752)
* Added model and stubs for all main methods

* Add constructors to all main objects

* Implement BlockCandidateTransactions

* implement expireOldTransactions and expireOrphanTransactions

* Rename isHighPriority to neverExpires

* Add stub for checkDoubleSpends

* Revert "Rename isHighPriority to neverExpires"

This reverts commit b2da9a4a00c02fb380d2518cf54fa16257bd8423.

* Imeplement transactionsOrderedByFeeRate

* Orphan maps should be idToOrphan

* Add error.go to mempool

* Invert the condition for banning when mempool rejects a transaction

* Move all model objects to model package

* Implement getParentsInPool

* Implemented mempoolUTXOSet.addTransaction

* Implement removeTransaction, remove sanity checks

* Implemented mempoolUTXOSet.checkDoubleSpends

* Implemented removeOrphan

* Implement removeOrphan

* Implement maybeAddOrphan and AddOrphan

* Implemented processOrphansAfterAcceptedTransaction

* Implement transactionsPool.addTransaction

* Implement RemoveTransaction

* If a transaction was removed from the mempool - update it's redeemers in orphan pool as well

* Use maximumOrphanTransactionCount

* Add allowOrphans to ValidateAndInsertTransaction stub

* Implement validateTransaction functions

* Implement fillInputs

* Implement ValidateAndInsertTransaction

* Implement HandleNewBlockTransactions

* Implement missing mempool interface methods

* Add comments to exported functions

* Call ValidateTransactionInIsolation where needed

* Implement RevalidateHighPriorityTransactions

* Rewire kaspad to use new mempool, and fix compilation errors

* Update rebroadcast logic to use new structure

* Handle non-standard transaction errors properly

* Add mutex to mempool

* bugfix: GetTransaction panics when ok is false

* properly calculate targetBlocksPerSecond in config.go

* Fix various lint errors and tests

* Fix expected text in test for duplicate transactions

* Skip the coinbase transaction in HandleNewBlockTransactions

* Unorphan the correct transactions

* Call ValidateTransactionAndPopulateWithConsensusData on unorphanTransaction

* Re-apply policy_test as check_transactions_standard_test

* removeTransaction: Remove redeemers in orphan pool as well

* Remove redundant check for uint64 < 0

* Export and rename isDust -> IsTransactionOutputDust to allow usage by rothschild

* Add allowOrphan to SubmitTransaction RPC request

* Remove all implementation from mempool.go

* tidy go mod

* Don't pass acceptedOrphans to handleNewBlockTransactions

* Use t.Errorf in stead of t.Fatalf

* Remove minimum relay fee from TestDust, as it's no longer configurable

* Add separate VirtualDAASCore method for faster retrieval where it's repeated multiple times

* Broadcast all transactions that were accepted

* Don't re-use GetVirtualDAAScore in GetVirtualInfo - this causes a deadlock

* Use real transaction count, and not Orphan

* Get mempool config from outside, incorporating values received from cli

* Use MinRelayFee and MaxOrphanTxs from global kaspad config

* Add explanation for the seemingly redundant check for transaction version in checkTransactionStandard

* Update some comment

* Convert creation of acceptedTransactions to a single line

* Move mempoolUTXOSet out of checkDoubleSpends

* Add test for attempt to insert double spend into mempool

* fillInputs: Skip check for coinbase - it's always false in mempool

* Clarify comment about removeRedeemers when removing random orphan

* Don't remove high-priority transactions in limitTransactionCount

* Use mempool.removeTransaction in limitTransactionCount

* Add mutex comment to handleNewBlockTransactions

* Return error from limitTransactionCount

* Pluralize the map types

* mempoolUTXOSet.removeTransaction: Don't restore utxo if it was not created in mempool

* Don't evacuate from orphanPool high-priority transactions

* Disallow double-spends in orphan pool

* Don't use exported (and locking) methods from inside mempool

* Check for double spends in mempool during revalidateTransaction

* Add checkOrphanDuplicate

* Add orphan to acceptedOrphans, not current

* Add TestHighPriorityTransactions

* Fix off-by-one error in limitTransactionCount

* Add TestRevalidateHighPriorityTransactions

* Remove checkDoubleSpends from revalidateTransaction

* Fix TestRevalidateHighPriorityTransactions

* Move check for MaximumOrphanCount to beggining of maybeAddOrphan

* Rename all map type to singulateToSingularMap

* limitOrphanPool only after the orphan was added

* TestDoubleSpendInMempool: use createChildTxWhenParentTxWasAddedByConsensus instead of createTransactionWithUTXOEntry

* Fix some comments

* Have separate min/max transaction versions for mempool

* Add comment on defaultMaximumOrphanTransactionCount to keep it small as long as we have recursion

* Fix comment

* Rename: createChildTxWhenParentTxWasAddedByConsensus -> createChildTxWhereParentTxWasAddedByConsensus

* Handle error from createChildTxWhereParentTxWasAddedByConsensus

* Rename createChildTxWhereParentTxWasAddedByConsensus -> createChildAndParentTxsAndAddParentToConsensus

* Convert all MaximumXXX constants to uint64

* Add comment

* remove mutex comments
2021-06-23 15:49:20 +03:00
Svarog
4f4a8934e7
Add option to specify blockHash in EstimateNetworkHashesPerSecond (#1731)
* Add BlockHash optional parameter to EstimateNetworkBlockHashesPerSecond

* Allow to pass '-' for optional values in kaspactl

* Solve a division-by-zero in estimateNetworkHashesPerSecond

* Add BlockHash to toAppMessage/fromAppMessage functions

* Rename: topHash -> StartHash

* Return proper error message if provided startHash doesn't deserialize into a hash
2021-05-27 14:59:29 +03:00
talelbaz
d7f2cf81c0
Change merge set order to topological order (#1654)
* Change mergeSet to be ordered topologically.

* Add special condition for genesis.

* Add check that the coinbase is validated.

* Change names of variables(old: chainHash, blueHash).

* Fix the DAG diagram in the comment above the function.

* Fix variables names.

Co-authored-by: tal <tal@daglabs.com>
Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-05-19 14:40:55 +03:00
Ori Newman
268906a7ce Add VirtualDaaScore to GetBlockDagInfo (#1719)
Co-authored-by: Svarog <feanorr@gmail.com>

(cherry picked from commit 36c56f73bf7b41528c81013d5d0d3bc7d7f6a206)
2021-05-18 17:07:07 +03:00
Ori Newman
9df80957b1 Fix getBlock and getBlocks RPC commands to return blocks and transactions properly (#1716)
* Fix getBlock RPC command to return transactions

* Fix getBlocks RPC command to return transactions and blocks

* Add GetBlockEvenIfHeaderOnly and use it for getBlock and getBlocks

* Implement GetBlockEvenIfHeaderOnly for fakeRelayInvsContext

* Use less nested code

(cherry picked from commit 50fd86e2878a6dcf1399610d8edc146466a5b492)
2021-05-13 16:00:42 +03:00
stasatdaglabs
1f56a68a28
Add an RPC command: EstimateNetworkHashesPerSecond (#1686)
* Implement EstimateNetworkHashesPerSecond.

* Fix failing tests.

* Add request/response messages to the .proto files.

* Add the EstimateNetworkHashesPerSecond RPC command.

* Add the EstimateNetworkHashesPerSecond RPC client function.

* Add the EstimateNetworkHashesPerSecond RPC command to kaspactl.

* Disallow windowSize lesser than 2.

* Fix wrong scale (milliseconds instead of seconds).

* Handle windowHashes being 0.
2021-04-22 15:18:21 +03:00
Svarog
dfd8b3423d
Implement new mechanism for updating UTXO Diffs (#1671)
* Use selectedParent instead of selectedTip for non-selectedTip blocks in restoreSingleBlockStatus

* Cache the selectedParent for re-use in a resolveSingleBlockStatus chain

* Implement and use reverseUTXOSet

* Reverse blocks in correct order

* Support resolveBlockStatus without separate stagingAreas for usage of testConsensus

* Handle the case where the tip of the resolved block is not the next selectedTip

* Unify isResolveTip

* Some minor fixes and cleanup

* Add full finality window re-org test to stability-slow

* rename: useSeparateStagingAreasPerBlock -> useSeparateStagingAreaPerBlock

* Better logs in resolveSingleBlockStatus

* A few retouches to reverseUTXODiffs

* TEMPORARY COMMIT: EXTRAT ALL DIFFFROMS TO SEPARATE METHODS

* TEMPORARY COMMIT: REMOVE DIFFICULTY CHECKS IN DEVNET

* Don't pre-allocate in utxo-algebra, since the numbers are not known ahead-of-time

* Add some logs to reverseUTXODiffs

* Revert "TEMPORARY COMMIT: REMOVE DIFFICULTY CHECKS IN DEVNET"

This reverts commit c0af9dc6ade78a914c970e11bc63c34605565f57.

* Revert "TEMPORARY COMMIT: EXTRAT ALL DIFFFROMS TO SEPARATE METHODS"

This reverts commit 4fcca1b48c3a1183598833a355b9bfaf169edba1.

* Remove redundant paranthesis

* Revise some logs messages

* Rename:oneBlockBeforeCurrentUTXOSet -> lastResolvedBlockUTXOSet

* Don't break if the block was resolved as invalid

* rename unverifiedBlocks to recentlyVerifiedBlcks in reverseUTXODiffs

* Add errors.New to the panic, for a stack trace

* Reverse the UTXODiffs after the main block has been commited

* Use the correct value for previousUTXODiff

* Add test for ReverseUTXODiff

* Fix some names and comments

* Update TestReverseUTXODiffs to use consensus.Config

* Fix comments mentioning 'oneBlockBeforeTip'
2021-04-20 10:26:55 +03:00
Svarog
28bfc0fb9c
Move pow package from model to utils (#1681) 2021-04-19 15:35:36 +03:00
Elichai Turkel
3c3ad1425d
Make moving the pruning point faster (#1660)
* Add oldPruningPoint to pruningStore

* Make the pruning store work with utxo diff and return an iterator over pruning point utxoset

* Redesign pruning point utxo storage by creating a diff and modifying the old pruning utxo set

* Fix review comments

* Rename updatePruningPointUTXOSet
2021-04-11 11:17:13 +03:00
Svarog
347dd8fc4b
Support resolveBlockStatus without separate stagingAreas for usage of testConsensus (#1666) 2021-04-08 11:26:17 +03:00
stasatdaglabs
c5b0394bbc
In RPC, use RPCTransactions and RPCBlocks instead of TransactionMessages and BlockMessages (#1609)
* Replace BlockMessage with RpcBlock in rpc.proto.

* Convert everything in kaspad to use RPCBlocks and fix tests.

* Fix compilation errors in stability tests and the miner.

* Update TransactionVerboseData in rpc.proto.

* Update TransactionVerboseData in the rest of kaspad.

* Make golint happy.

* Include RpcTransactionVerboseData in RpcTransaction instead of the other way around.

* Regenerate rpc.pb.go after merge.

* Update appmessage types.

* Update appmessage request and response types.

* Reimplement conversion functions between appmessage.RPCTransaction and protowire.RpcTransaction.

* Extract RpcBlockHeader toAppMessage/fromAppMessage out of RpcBlock.

* Fix compilation errors in getBlock, getBlocks, and submitBlock.

* Fix compilation errors in getMempoolEntry.

* Fix compilation errors in notifyBlockAdded.

* Update verbosedata.go.

* Fix compilation errors in getBlock and getBlocks.

* Fix compilation errors in getBlocks tests.

* Fix conversions between getBlocks message types.

* Fix integration tests.

* Fix a comment.

* Add selectedParent to the verbose block response.
2021-03-30 17:43:02 +03:00
Svarog
f1451406f7
Add support for multiple staging areas (#1633)
* Add StagingArea struct

* Implemented staging areas in blockStore

* Move blockStagingShard to separate folder

* Apply staging shard to acceptanceDataStore

* Update blockHeaderStore with StagingArea

* Add StagingArea to BlockRelationStore

* Add StagingArea to blockStatusStore

* Add StagingArea to consensusStateStore

* Add StagingArea to daaBlocksStore

* Add StagingArea to finalityStore

* Add StagingArea to ghostdagDataStore

* Add StagingArea to headersSelectedChainStore and headersSelectedTipStore

* Add StagingArea to multisetStore

* Add StagingArea to pruningStore

* Add StagingArea to reachabilityDataStore

* Add StagingArea to utxoDiffStore

* Fix forgotten compilation error

* Update reachability manager and some more things with StagingArea

* Add StagingArea to dagTopologyManager, and some more

* Add StagingArea to GHOSTDAGManager, and some more

* Add StagingArea to difficultyManager, and some more

* Add StagingArea to dagTraversalManager, and some more

* Add StagingArea to headerTipsManager, and some more

* Add StagingArea to constnsusStateManager, pastMedianTimeManager

* Add StagingArea to transactionValidator

* Add StagingArea to finalityManager

* Add StagingArea to mergeDepthManager

* Add StagingArea to pruningManager

* Add StagingArea to rest of ValidateAndInsertBlock

* Add StagingArea to blockValidator

* Add StagingArea to coinbaseManager

* Add StagingArea to syncManager

* Add StagingArea to blockBuilder

* Update consensus with StagingArea

* Add StagingArea to ghostdag2

* Fix remaining compilation errors

* Update names of stagingShards

* Fix forgotten stagingArea passing

* Mark stagingShard.isCommited = true once commited

* Move isStaged to stagingShard, so that it's available without going through store

* Make blockHeaderStore count be avilable from stagingShard

* Fix remaining forgotten stagingArea passing

* commitAllChanges should call dbTx.Commit in the end

* Fix all tests tests in blockValidator

* Fix all tests in consensusStateManager and some more

* Fix all tests in pruningManager

* Add many missing stagingAreas in tests

* Fix many tests

* Fix most of all other tests

* Fix ghostdag_test.go

* Add comment to StagingArea

* Make list of StagingShards an array

* Add comment to StagingShardID

* Make sure all staging shards are pointer-receiver

* Undo bucket rename in block_store

* Typo: isCommited -> isCommitted

* Add comment explaining why stagingArea.shards is an array
2021-03-29 10:34:11 +03:00
Svarog
cebcab7f5c
Implement BIP-143-like sighash (#1598)
* Move CalculateSignatureHash to consensushashing

* Added CalcSignatureHash_BIP143 with all parameters except the re-used hashes

* Add handling of outputHash

* Add sequencesHash to the mix

* Add previousOutputsHash to the mix

* Replace legacy CalculateSigHash with new one, and re-wire to all non-test code

* Add missing types to WriteElement

* Fix tests in txscript

* Fix tests in rest of code

* Add missing comments

* Add SubnetworkID and Gas to sigHash

* Add TestCalculateSignatureHash

* Invert condition in SigHashSingle getOutputsHash

* Explicitly define that payloadHash for native transactions is 0

* added benchmark to CalculateSignatureHash

* Reformat call for signAndCheck

* Change SigHashes to be true bit-fields

* Add check for transaction version

* Write length of byte array in WriteElement

* hashOutpoint should get outpoint, not txIn

* Use inputIndex instead of i to determine SigHashType

* Use correct transaction version + fix some typos

* Fix hashes in test

* Reformat an overly-long line

* Replace checkHashTypeEncoding with caalls to hashType.IsStandardSigHashType

* Convert hashType to uint8

* Add comment
2021-03-17 15:17:38 +02:00
Ori Newman
b84080f3d9
Fix getBlocks to not add the anticone when some blocks were filtered by GetHashesBetween (#1611)
* Fix getBlocks to not add the anticone when some blocks were filtered by GetHashesBetween

* Fix TestSyncManager_GetHashesBetween
2021-03-16 14:43:02 +02:00
Ori Newman
d9449a32b8
Use DAA score where needed (#1602)
* Replace blue score with DAA score in UTXO entries

* Use DAA score for coinbase maturity

* Use DAA score for sequence lock

* Fix calcBlockSubsidy to use DAA score

* Don't pay to blocks that are not included in the DAA added blocks, and bestow red blocks reward to the merging block

* Fix TestGetPruningPointUTXOs

* Fix TestTransactionAcceptance

* Fix TestChainedTransactions

* Fix TestVirtualDiff

* Fix TestBlockWindow

* Fix TestPruning

* Use NewFromSlice instead of manually creating the hash set

* Add assert

* Add comment

* Remove redundant call to UpdateDAADataAndReturnDifficultyBits

* Add RequiredDifficulty, rename UpdateDAADataAndReturnDifficultyBits to StageDAADataAndReturnRequiredDifficulty and add comments

* Make buildUTXOInvalidHeader get bits as an argument

* Fix comments
2021-03-15 13:48:40 +02:00
Ori Newman
ec446ac511
Adding DAA score (#1596)
* Save DAA score and DAA added blocks for each block

* Add test

* Add pruning support

* Replace 8 with uint64Length

* Separate DAABlocksStore cache size to DAA score and daaAddedBlocks
2021-03-14 09:44:44 +02:00
Ori Newman
53781eed4d
Remove payload hash (#1583)
* Remove payload hash

* Fix tests
2021-03-08 15:15:03 +02:00
Elichai Turkel
7909480757
Merge big subdags in pick virtual parents (#1574)
* Refactor mergeSetIncrease to return the current BFS block to allow easier merging

* Remove unneeded Heap/HashSet usages

* Add new IsAnyAncestorOf to DagTopolyManager

* Check if the new candidate is in the future of any existing candidate

* Add comments and fix off-by-one in the mergeSetIncrease queue

* Fixed DAGToplogy test mock

* Fix review comments
2021-03-03 16:17:16 +02:00
Elichai Turkel
6a12428504
Close iterators (#1542)
* Add Close() function to all the iterators

* Add defer iterator.Close() whenever we open an iterator

* Add isClosed to all iterators and panic/return error if used after closing

Co-authored-by: Svarog <feanorr@gmail.com>
2021-03-01 11:15:59 +02:00
Svarog
63b1d2a05a
Add childrenHashes to GetBlock/s RPC commands (#1560)
* Add childrenHashes to GetBlock/s RPC commands

* Fix missed error + implement GetBlockChildren in fakeRelayInvsContext
2021-02-28 18:28:29 +02:00
Elichai Turkel
24a12cf2a1
Fix subnetworks FromString and disable reversal (#1557)
* Remove DomainSubnetworkID reversal

* Fix DomaiNSubnetworkID FromString implementation

* Change RPC conversation logic to use Stringer/FromString

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-02-25 16:35:31 +02:00
Svarog
fb6c9c8f21
Remove virtual diff parents (#1550)
* resolveSingleBlockStatus: If the block being resolved is not going to be the next selectedTip - set it's diffParent to be the old selectedTip

* resolveSingleBlockStatus: If the block being resolved is going to be the next selectedTip - set it as old selectedTip's diffChild

* Remove any mentions of virtualDiffParents

* If block is genesis - don't do all the mumbo-jumbo with oldSelectedTip

* Check an unchecked error

* Write a better log message
2021-02-23 17:19:35 +02:00
Ori Newman
2adb4f5d0f
Fix UTXO index (#1548)
* Add VirtualUTXODiff and VirtualParents to block insertion result

* Add GetVirtualUTXOs

* Add OnPruningPointUTXOSetOverrideHandler

* Add recovery to UTXO index

* Add UTXO set override notification

* Fix compilation error

* Fix iterators in UTXO index and fix TestUTXOIndex

* Change Dialing to DEBUG

* Change LogBlock location

* Rename StopNotify to StopNotifying

* Add sanity check

* Add comment

* Remove receiver from serialization functions

Co-authored-by: Elichai Turkel <elichai.turkel@gmail.com>
2021-02-23 16:51:51 +02:00
Svarog
bee0893660
Renamed BlueBlockWindow to just BlockWindow (#1547)
* Renamed BlueBlockWindow to just BlockWindow

* Update comment
2021-02-22 11:33:39 +02:00
Svarog
995e526dae
Make antiPastHashesBetween return blocks sorted in ghostdag-order (#1536)
* Make antiPastHashesBetween return blocks sorted in ghostdag-order

* Return sortedMergeSet instead of blueMergeSet

* Invert the order of parameters of IsAncestorOf

* Add RenderDAGToDot to TestConsensus

* Add HighHash explicitly, unless lowHash == highHash

* Use Equal instead of == when comparing hashes

* Fixed TestSyncManager_GetHashesBetween

* Fix tests

* findHighHashAccordingToMaxBlueScoreDifference: don't start looking if the whole thing fits

* Handle a missed error

* Remove redundant call to RenderToDot

* Fix bug in findHighHashAccordingToMaxBlueScoreDifference
2021-02-17 18:22:08 +02:00
Svarog
171deded4e
Implement GetBlocks RPC command (#1514)
* Remove BlockHexes from GetBlocks request and response

* Add GetBlocks RPC

* Include the selectedTip's anticone in GetBlocks

* Add Anticone to fakeRelayInvsContext

* Include verbose data only if it was requested + Add comments to HandleGetBlocks

* Allow antiPastHashesBetween to receive unrelated low and high hashes

* Convert to/from protowire GetBlocksResponse with no verbose data correctly

* Removed NextLowHash

* Update GetBlocks in rpc_client

* Validate in consensus.Anticone that blockHash exists

Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-02-10 18:27:04 +02:00
Svarog
1222a555f2
Prune blocks below pruning point when moving pruning point during IBD (#1513)
* Split deletePastBlocks into sub-routines

* Remove SelectedParentIterator, and refactor SelectedChildIterator to support First and Error

* Implement PruneAllBlocks

* Prune all blocks in the store

* Prune only blocks that are below the pruning point

* Defer call onEnd of LogAndMeasureExecutionTime

* Handle a forgotten error

* Minor style fixes
2021-02-10 16:39:36 +02:00
Ori Newman
dbababb978
Limit mempool size to million transactions and remove the least profitable transactions (#1483)
* Limit mempool size to million transactions and remove the least profitable transactions

* Simplify insert

* Fix typo

* Improve findTxIndexInOrderedTransactionsByFeeRate readability
2021-02-03 19:45:39 +02:00
Ori Newman
a9a810a2b2
Add block type to MineJSON (#1464) 2021-01-28 13:22:20 +02:00
Michael Sutton
c9b591f2d3
Final reindex algorithm (#1430)
* Mine JSON

* [Reindex tests] add test_params and validate_mining flag to test_consensus

* Rename file and extend tests

* Ignore local test datasets

* Use spaces over tabs

* Reindex algorithm - full algorithm, initial commit, some tests fail

* Reindex algorithm - a few critical fixes

* Reindex algorithm - move reindex struct and all related operations to new file

* Reindex algorithm - added a validateIntervals method and modified tests to use it (instead of exact comparisons)

* Reindex algorithm - modified reindexIntervals to receive the new child as argument and fixed an important related bug

* Reindex attack tests - move logic to helper function and add stretch test

* Reindex algorithm - variable names and some comments

* Reindex algorithm - minor changes

* Reindex algorithm - minor changes 2

* Reindex algorithm - extended stretch test

* Reindex algorithm - small fix to validate function

* Reindex tests - move tests and add DAG files

* go format fixes

* TestParams doc comment

* Reindex tests - exact comparisons are not needed

* Update to version 0.8.6

* Remove TestParams and use AddUTXOInvalidHeader instead

* Use gzipeed test files

* This unintended change somehow slipped in through branch merges

* Rename test

* Move interval increase/decrease methods to reachability interval file

* Addressing a bunch of minor review comments

* Addressed a few more minor review comments

* Make code of offsetSiblingsBefore and offsetSiblingsAfter symmetric

* Optimize reindex logic in cases where reorg occurs + reorg test

* Do not change reindex root too fast (on reorg)

* Some comments

* A few more comments

* Addressing review comments

* Remove TestNoAttackAlternateReorg and assert chain attack

* Minor

Co-authored-by: Elichai Turkel <elichai.turkel@gmail.com>
Co-authored-by: Mike Zak <feanorr@gmail.com>
Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-01-27 17:09:20 +02:00
stasatdaglabs
819ec9f2a7
Pass fromOutpoint into GetPruningPointUTXOs instead of offset, so it could Seek over the cursor (#1447)
* Implement TestGetPruningPointUTXOs.

* Fix a bad error message.

* Fix TestGetPruningPointUTXOs for testnet.

* Make sure all the UTXOs are returned in TestGetPruningPointUTXOs.

* Implement BenchmarkGetPruningPointUTXOs.

* Pass fromOutpoint into GetPruningPointUTXOs instead of offset, so it could Seek over the cursor.

* Fix weird benchmark timer calls.

* Remove unnecessary collection of outpointAndUTXOEntryPairs from BenchmarkGetPruningPointUTXOs.

* Fix a comment.
2021-01-25 13:38:59 +02:00
stasatdaglabs
ca04c049ab
When the pruning point moves, update its UTXO set outside of a database transaction (#1444)
* Remove pruningPointUTXOSetStaging and implement UpdatePruningPointUTXOSet.

* Implement StageStartSavingNewPruningPointUTXOSet, HadStartedSavingNewPruningPointUTXOSet, and FinishSavingNewPruningPointUTXOSet.

* Fix a bad return.

* Implement UpdatePruningPointUTXOSetIfRequired.

* Call UpdatePruningPointUTXOSetIfRequired on consensus creation.

* Rename savingNewPruningPointUTXOSetKey to updatingPruningPointUTXOSet.

* Add a log.

* Add calls to runtime.GC() at its start and end.

* Rename a variable.

* Replace calls to runtime.GC to calls to LogMemoryStats.

* Wrap the contents of LogMemoryStats in a log closure.
2021-01-24 14:48:11 +02:00
stasatdaglabs
756f40c59a
Sync pruning point UTXO sets incrementally instead of all at once (#1431)
* Replaced the content of MsgIBDRootUTXOSetChunk with pairs of outpoint-utxo entry pairs.

* Rename utxoIter to utxoIterator.

* Add a big stinky TODO on an assert.

* Replace pruningStore staging with a UTXO set iterator.

* Reimplement receiveAndInsertIBDRootUTXOSet.

* Extract OutpointAndUTXOEntryPairsToDomainOutpointAndUTXOEntryPairs into domainconverters.go.

* Pass the outpoint and utxy entry pairs to the pruning store.

* Implement InsertCandidatePruningPointUTXOs.

* Implement ClearCandidatePruningPointUTXOs.

* Implement UpdateCandidatePruningPointMultiset.

* Use the candidate pruning point multiset in updatePruningPoint.

* Implement CandidatePruningPointUTXOIterator.

* Use the pruning point utxo set iterator for StageVirtualUTXOSet.

* Defer ClearCandidatePruningPointUTXOs.

* Implement OverwriteVirtualUTXOSet.

* Implement CommitCandidatePruningPointUTXOSet.

* Implement BeginOverwritingVirtualUTXOSet and FinishOverwritingVirtualUTXOSet.

* Implement overwriteVirtualUTXOSetAndCommitPruningPointUTXOSet.

* Rename ClearCandidatePruningPointUTXOs to ClearCandidatePruningPointData.

* Add missing methods to dbManager.

* Implement PruningPointUTXOs.

* Implement RecoverUTXOIfRequired.

* Delete the utxoserialization package.

* Fix compilation errors in TestValidateAndInsertPruningPoint.

* Switch order of operations in the if statements in PruningPointUTXOs so that Next() wouldn't be unnecessarily called.

* Fix missing pruning point utxo set staging and bad slice length.

* Fix no default multiset in InsertCandidatePruningPointUTXOs.

* Make go vet happy.

* Rename candidateXXX to importedXXX.

* Do some more renaming.

* Rename some more.

* Fix bad MsgIBDRootNotFound logic.

* Fix an error message.

* Simplify receiveIBDRootBlock.

* Fix error message in receiveAndInsertIBDRootUTXOSet.

* Do some more renaming.

* Fix merge errors.

* Fix a bug caused by calling iterator.First() unnecessarily.

* Remove databaseContext from stores and don't use a transaction in ClearXXX functions.

* Simplify receiveAndInsertIBDRootUTXOSet.

* Fix offset count in PruningPointUTXOs().

* Fix readOnlyUTXOIteratorWithDiff.First().

* Split handleRequestIBDRootUTXOSetAndBlockFlow into smaller methods.

* Rename IbdRootNotFound to UnexpectedPruningPoint.

* Rename requestIBDRootHash to requestPruningPointHash.

* Rename IBDRootHash to PruningPointHash.

* Rename RequestIBDRootUTXOSetAndBlock to RequestPruningPointUTXOSetAndBlock.

* Rename IBDRootUTXOSetChunk to PruningPointUTXOSetChunk.

* Rename RequestNextIBDRootUTXOSetChunk to RequestNextPruningPointUTXOSetChunk.

* Rename DoneIBDRootUTXOSetChunks to DonePruningPointUTXOSetChunks.

* Rename remaining references to IBD root.

* Fix an error message.

* Add a check for HadStartedImportingPruningPointUTXOSet in commitVirtualUTXODiff.

* Add a check for HadStartedImportingPruningPointUTXOSet in ImportPruningPointUTXOSetIntoVirtualUTXOSet.

* Move FinishImportingPruningPointUTXOSet closer to HadStartedImportingPruningPointUTXOSet.

* Remove reference to pruningStore in utxoSetIterator.

* Pointerify utxoSetIterator receivers.

* Fix bad insert in CommitImportedPruningPointUTXOSet.

* Rename commitImportedPruningPointUTXOSetAll to applyImportedPruningPointUTXOSet.

* Simplify PruningPointUTXOs.

* Add populateTransactionWithUTXOEntriesFromUTXOSet.

* Fix a TODO comment.

* Rename InsertImportedPruningPointUTXOs to AppendImportedPruningPointUTXOs.

* Extract handleRequestPruningPointUTXOSetAndBlockMessage to a separate method.

* Rename stuff in readOnlyUTXOIteratorWithDiff.First().

* Address toAddIterator in readOnlyUTXOIteratorWithDiff.First().

* Call First() before any full iteration on ReadOnlyUTXOSetIterator.

* Call First() before any full iteration on a database Cursor.

* Put StartImportingPruningPointUTXOSet inside the pruning point transaction.

* Make serializeOutpoint and serializeUTXOEntry free functions in pruningStore.

* Fix readOnlyUTXOIteratorWithDiff.First().

* Fix bad validations in importPruningPoint.

* Remove superfluous call to validateBlockTransactionsAgainstPastUTXO.
2021-01-21 17:24:52 +02:00
Svarog
a4adbabf96
TestBuildBlockErrorCases and remove redundant check of coinbase script length (#1427)
* Write structure of TestBlockBuilderErrorCases

* Remove double verification of script length in serializeCoinbasePayload

* Remove redundant code in TestBuildBlockErrorCases

* Rename povTransactionHash -> povBlockHash

* Convert coinbasePayloadScriptPublicKeyMaxLength to uint8

* Re-use consensus in TestBuildBlockErrorCases
2021-01-19 10:37:51 +02:00
Ori Newman
a1381d6768
Add TestCheckParentBlockBodiesExist (#1405)
* Add TestCheckParentBlockBodiesExist

* Use block in pruning point's anticone for the test

* Fix test db name
2021-01-14 13:31:17 +02:00
Svarog
1b97cfb302
Prevent a race condition in findHighestSharedBlockHash where we get headersSelectedTip and then pass it as highHash to GetBlockLocator, without locking consensus (#1410)
* Prevent a race condition in findHighestSharedBlockHash where we get headersSelectedTip and then pass it as highHash to GetBlockLocator, without locking consensus

* Restart findHighestSharedBlockHash if lowHash or highHash are no longer in selectedParentChain

* Test for specifically ErrBlockNotInSelectedParentChain instead of database NotFound error

* Fix TestCreateHeadersSelectedChainBlockLocator

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-01-13 17:55:37 +02:00
Elichai Turkel
68bd8330ac
Log the networks hashrate (#1406)
* Log the hashrate of each block

* Add a test for GetHashrateString

* Move difficulty related functions to its own package

* Convert the validated log in validateAndInsertBlock to a log function

* Add tests for max/min int
2021-01-13 12:51:23 +02:00
Svarog
ce348373c6
Delete existing UTXOSet when commiting VirtualUTXOSet. (#1403) 2021-01-12 16:51:56 +02:00
Ori Newman
bcf2302460
Add high hash to block locator, and add block locator tests (#1397)
* Include high hash in the block locator

* Add tests for block locator

* Remove redundant function

* Remove redundant assignments
2021-01-12 11:16:25 +02:00
Ori Newman
b8ca33d91d
Add selected chain store and optimize block locator with it (#1394)
* Add selected chain store and optimize block locator with it

* Fix build error

* Fix comments

* Fix IsStaged

* Rename CalculateSelectedParentChainChanges to CalculateChainPath and SelectedParentChainChanges->SelectedChainPath

* Use binary.LittleEndian directly to allow compiler optimizations

* Remove boolean from HeadersSelectedChainStore interface

* Prevent endless loop in block locator
2021-01-11 15:51:45 +02:00
Ori Newman
d8293ef635
Revert "Add recoverability for UTXO index (#1342)" (#1353)
This reverts commit 778375c4
2021-01-06 11:41:52 +02:00
Ori Newman
3ec1cbe236
Add TestBlockStatus (#1361)
Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-01-06 10:37:10 +02:00
Elichai Turkel
6279db2bf1
Reverse ghostdag tie break direction (#1359)
* Remove hash reversal in ghostdag

* Update tests after chaning ghostdag hash direction

Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-01-05 19:10:59 +02:00
talelbaz
8a309a7d2a
Upgradability mechanisms script version (#1313)
* ''

* ''

* ''

* Changes genesis block version to 0.

* a

* a

* All tests are done.

* All tests passed for changed block version from int32 to uint16

* Adds validation of rejecting blocks with unknown versions.

* Changes txn version from int32 to uint16.

* .

* Adds comments to exported functions.

* Change functions name from ConvertFromRpcScriptPubKeyToRPCScriptPubKey to ConvertFromAppMsgRPCScriptPubKeyToRPCScriptPubKey and from ConvertFromRPCScriptPubKeyToRpcScriptPubKey to ConvertFromRPCScriptPubKeyToAppMsgRPCScriptPubKey

* change comment to "ScriptPublicKey represents a Kaspad ScriptPublicKey"

* delete part (tx.Version < 0) that cannot be exist on the if statement.

* Revert protobuf version.

* Fix a comment.

* Fix a comment.

* Rename a variable.

* Rename a variable.

* Remove a const.

* Rename a type.

* Rename a field.

* Rename a field.

* Remove commented-out code.

* Remove dangerous nil case in DomainTransactionOutput.Clone().

* Remove a constant.

* Fix a string.

* Fix wrong totalScriptPubKeySize in transactionMassStandalonePart.

* Remove a constant.

* Remove an unused error.

* Fix a serialization error.

* Specify version types to be uint16 explicitly.

* Use constants.ScriptPublicKeyVersion.

* Fix a bad test.

* Remove some whitespace.

* Add a case to utxoEntry.Equal().

* Rename scriptPubKey to scriptPublicKey.

* Remove a TODO.

* Rename constants.

* Rename a variable.

* Add version to parseShortForm.

Co-authored-by: tal <tal@daglabs.com>
Co-authored-by: stasatdaglabs <stas@daglabs.com>
2021-01-05 17:50:09 +02:00
Elichai Turkel
72a7ca53e6
Save and expose the database in TestConsensus (#1349) 2021-01-05 14:13:33 +02:00
Elichai Turkel
e509cb1597
Smal performance improvements to BlueWindow (#1343)
* Convert BlockGHOSTDAGData from an interface to a public struct with getters

* Move hashes.Less to externalapi so it can access the hashes directly without copying

* Reduce calls to ghostdagstore.Get in blueWindow

* Simplify the logic in RequiredDifficulty and reuse big.Int

* Remove bigintpool as its no longer used

* Use ChooseSelectedParent in RequiredDifficulty instead of looping over the parents

* Remove comment
2021-01-05 12:13:02 +02:00
Ori Newman
778375c4af
Add recoverability for UTXO index (#1342)
* Add recoverability for UTXO index

* Add comment

* Rename UTXOOutpointPair->OutpointUTXOPair

* Get rid of the db transaction on resetStore and collect all keys before deleting

* Use VirtualSelectedParent instead of selected tip

* Fix error
2021-01-04 14:15:51 +02:00
stasatdaglabs
b473a09111
Fix a crash in the UTXO index (#1332)
* Add a field to TransactionAcceptanceData: TransactionInputUTXOEntries.

* Fix failing tests.

* Add transactionInputUtxoEntries to the database.

* Populate transactionInputUTXOEntries in applyMergeSetBlocks and use them in the UTXO index.

* Remove UTXOEntry.Clone().

* Add an additional equality test.
2020-12-31 14:50:14 +02:00
stasatdaglabs
7dd0188838
Move the heavy lifting in BlockLocator from the syncer to the syncee (#1324)
* Add a new message: BlockLocatorHighestHash.

* Add a new message: IBDBlockLocator.

* Implement HandleIBDBlockLocator.

* Reimplement findHighestSharedBlockHash.

* Make HandleIBDBlockLocator only return hashes that are in the selected parent chain of the target hash.

* Increase the cache sizes of blockRelationStore, reachabilityDataStore, and ghostdagDataStore.

* Fix wrong initial highHash in findHighestSharedBlockHash.

* Make go vet happy.

* Protect against receiving wrong messages when expecting MsgIBDBlockLocatorHighestHash.
2020-12-30 15:44:14 +02:00
Svarog
9b12b9c58a
Make ReachabilityData a read-only interface with a writable variant, to prevent cloning (#1316)
* Rename reachabilityManager.data to dataForInsertion, and use it only during insertions

* Make reachabilityData an interface

* Fix db serialization of reachability data

* Fix reachabilityDataStore

* Fix all tests

* Cleanup debugging code

* Fix insertToFutureCoveringSet

* Add comments

* Rename to ReachabilityData and MutableReachabilityData
2020-12-30 09:48:38 +02:00