44 Commits

Author SHA1 Message Date
Elichai Turkel
dbf18d8052
Hard fork - new genesis with the utxo set of the last block (#1856)
* UTXO dump of block 0fca37ca667c2d550a6c4416dad9717e50927128c424fa4edbebc436ab13aeef

* Activate HF immediately and change reward to 1000

* Change protocol version and datadir location

* Delete comments

* Fix zero hash to muhash zero hash in genesis utxo dump check

* Don't omit genesis as direct parent

* Fix tests

* Change subsidy to 500

* Dont assume genesis multiset is empty

* Fix BlockReward test

* Fix TestValidateAndInsertImportedPruningPoint test

* Fix pruning point genesis utxo set

* Fix tests related to mainnet utxo set

* Dont change the difficulty before you have a full window

* Fix TestBlockWindow tests

* Remove global utxo set variable, and persist mainnetnet utxo deserialization between runs

* Fix last tests

* Make peer banning opt-in

* small fix for a test

* Fix go lint

* Fix Ori's review comments

* Change DAA score of genesis to checkpoint DAA score and fix all tests

* Fix the BlockLevel bits counting

* Fix some tests and make them run a little faster

* Change datadir name back to kaspa-mainnet and change db path from /data to /datadir

* Last changes for the release and change the version to 0.11.5

Co-authored-by: Ori Newman <orinewman1@gmail.com>
Co-authored-by: Ori Newman <>
Co-authored-by: msutton <mikisiton2@gmail.com>
2021-11-25 20:18:43 +02:00
Ori Newman
d9bc94a2a8
Replace header finality point with pruning point and enforce finality rules on IBD with headers proof (#1823)
* Replace header finality point with pruning point

* Fix TestTransactionAcceptance

* Fix pruning candidate

* Store all past pruning points

* Pass pruning points on IBD

* Add blue score to block header

* Simplify ArePruningPointsInValidChain

* Fix static check errors

* Fix genesis

* Renames and text fixing

* Use ExpectedHeaderPruningPoint in block builder

* Fix TestCheckPruningPointViolation
2021-08-31 08:01:48 +03:00
Ori Newman
d207888b67
Implement pruned headers node (#1787)
* Pruning headers p2p basic structure

* Remove headers-first

* Fix consensus tests except TestValidateAndInsertPruningPointWithSideBlocks and TestValidateAndInsertImportedPruningPoint

* Add virtual genesis

* Implement PruningPointAndItsAnticoneWithMetaData

* Start fixing TestValidateAndInsertImportedPruningPoint

* Fix TestValidateAndInsertImportedPruningPoint

* Fix BlockWindow

* Update p2p and gRPC

* Fix all tests except TestHandleRelayInvs

* Delete TestHandleRelayInvs parts that cover the old IBD flow

* Fix lint errors

* Add p2p_request_ibd_blocks.go

* Clean code

* Make MsgBlockWithMetaData implement its own representation

* Remove redundant check if highest share block is below the pruning point

* Fix TestCheckLockTimeVerifyConditionedByAbsoluteTimeWithWrongLockTime

* Fix comments, errors ane names

* Fix window size to the real value

* Check reindex root after each block at TestUpdateReindexRoot

* Remove irrelevant check

* Renames and comments

* Remove redundant argument from sendGetBlockLocator

* Don't delete staging on non-recoverable errors

* Renames and comments

* Remove redundant code

* Commit changes inside ResolveVirtual

* Add comment to IsRecoverableError

* Remove blocksWithMetaDataGHOSTDAGDataStore

* Increase windows pagefile

* Move DeleteStagingConsensus outside of defer

* Get rid of mustAccepted in receiveBlockWithMetaData

* Ban on invalid pruning point

* Rename interface_datastructures_daawindowstore.go to interface_datastructures_blocks_with_meta_data_daa_window_store.go

* * Change GetVirtualSelectedParentChainFromBlockResponseMessage and VirtualSelectedParentChainChangedNotificationMessage to show only added block hashes
*  Remove ResolveVirtual
* Use externalapi.ConsensusWrapper inside MiningManager
* Fix pruningmanager.blockwithmetadata

* Set pruning point selected child when importing the pruning point UTXO set

* Change virtual genesis hash

* replace the selected parent with virtual genesis on removePrunedBlocksFromGHOSTDAGData

* Get rid of low hash in block locators

* Remove +1 from everywhere we use difficultyAdjustmentWindowSize and increase the default value by one

* Add comments about consensus wrapper

* Don't use separate staging area when resolving resolveBlockStatus

* Fix netsync stability test

* Fix checkResolveVirtual

* Rename ConsensusWrapper->ConsensusReference

* Get rid of blockHeapNode

* Add comment to defaultDifficultyAdjustmentWindowSize

* Add SelectedChild to DAGTraversalManager

* Remove redundant copy

* Rename blockWindowHeap->calculateBlockWindowHeap

* Move isVirtualGenesisOnlyParent to utils

* Change BlockWithMetaData->BlockWithTrustedData

* Get rid of maxReasonLength

* Split IBD to 100 blocks each time

* Fix a bug in calculateBlockWindowHeap

* Switch to trusted data when encountering virtual genesis in blockWithTrustedData

* Move ConsensusReference to domain

* Update ConsensusReference comment

* Add comment

* Rename shouldNotAddGenesis->skipAddingGenesis
2021-07-26 12:24:07 +03:00
talelbaz
aba44e7bfb
Disable relative time lock by time (#1800)
* ignore type flag

* Ignore type flag of relative time lock - interpret as DAA score

* Split verifyLockTime to functions with and without threshold.relative lockTimes dont need threshold check

* Change function name and order of the functions calls

Co-authored-by: tal <tal@daglabs.com>
2021-07-18 17:52:16 +03:00
Svarog
369a3bac09
Limit block mass instead of merge set limit + Introduce SigOpCount to TransactionInput (#1790)
* Update constants

* Add to transaction SigOpCount

* Update mass calculation, and move it from InContext to InIsolation

* Update block validation accordingly

* Add SigOpCount validation during TransactionInContext

* Remove checking of mass vs maxMassAcceptedByBlock from consensusStateManager

* Update mining manager with latest changes

* Add SigOpCount to MsgTx.Copy()

* Fix initTestTransactionAcceptanceDataForClone

* Fix all tests in transaction_equal_clone_test.go

* Fix TestBlockMass

* Fix tests in transactionvalidator package

* Add SigOpCount to sighash

* Fix TestPruningDepth

* Fix problems in libkaspawalelt

* Fix integration tests

* Fix CalculateSignatureHash tests

* Remove remaining places talking about block size

* Add sanity check to checkBlockMass to make sure all transactions have their mass filled

* always add own sigOpCount to sigHash

* Update protowire/rpc.md

* Start working on removing any remaining reference to block/tx size

* Update rpc transaction verbose data to include mass rather then size

* Convert verboseData and block size check to mass

* Remove remaining usages of tx size in mempool

* Move transactionEstimatedSerializedSize to transactionvalidator

* Add PopulateMass to fakeRelayInvsContext

* Move PopulateMass to beggining of ValidateAndInsertTransaction + fix in it

* Assign mass a new number for backward-compatibility
2021-07-14 14:21:57 +03:00
talelbaz
d7f2cf81c0
Change merge set order to topological order (#1654)
* Change mergeSet to be ordered topologically.

* Add special condition for genesis.

* Add check that the coinbase is validated.

* Change names of variables(old: chainHash, blueHash).

* Fix the DAG diagram in the comment above the function.

* Fix variables names.

Co-authored-by: tal <tal@daglabs.com>
Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-05-19 14:40:55 +03:00
Svarog
dfd8b3423d
Implement new mechanism for updating UTXO Diffs (#1671)
* Use selectedParent instead of selectedTip for non-selectedTip blocks in restoreSingleBlockStatus

* Cache the selectedParent for re-use in a resolveSingleBlockStatus chain

* Implement and use reverseUTXOSet

* Reverse blocks in correct order

* Support resolveBlockStatus without separate stagingAreas for usage of testConsensus

* Handle the case where the tip of the resolved block is not the next selectedTip

* Unify isResolveTip

* Some minor fixes and cleanup

* Add full finality window re-org test to stability-slow

* rename: useSeparateStagingAreasPerBlock -> useSeparateStagingAreaPerBlock

* Better logs in resolveSingleBlockStatus

* A few retouches to reverseUTXODiffs

* TEMPORARY COMMIT: EXTRAT ALL DIFFFROMS TO SEPARATE METHODS

* TEMPORARY COMMIT: REMOVE DIFFICULTY CHECKS IN DEVNET

* Don't pre-allocate in utxo-algebra, since the numbers are not known ahead-of-time

* Add some logs to reverseUTXODiffs

* Revert "TEMPORARY COMMIT: REMOVE DIFFICULTY CHECKS IN DEVNET"

This reverts commit c0af9dc6ade78a914c970e11bc63c34605565f57.

* Revert "TEMPORARY COMMIT: EXTRAT ALL DIFFFROMS TO SEPARATE METHODS"

This reverts commit 4fcca1b48c3a1183598833a355b9bfaf169edba1.

* Remove redundant paranthesis

* Revise some logs messages

* Rename:oneBlockBeforeCurrentUTXOSet -> lastResolvedBlockUTXOSet

* Don't break if the block was resolved as invalid

* rename unverifiedBlocks to recentlyVerifiedBlcks in reverseUTXODiffs

* Add errors.New to the panic, for a stack trace

* Reverse the UTXODiffs after the main block has been commited

* Use the correct value for previousUTXODiff

* Add test for ReverseUTXODiff

* Fix some names and comments

* Update TestReverseUTXODiffs to use consensus.Config

* Fix comments mentioning 'oneBlockBeforeTip'
2021-04-20 10:26:55 +03:00
Svarog
347dd8fc4b
Support resolveBlockStatus without separate stagingAreas for usage of testConsensus (#1666) 2021-04-08 11:26:17 +03:00
Svarog
f1451406f7
Add support for multiple staging areas (#1633)
* Add StagingArea struct

* Implemented staging areas in blockStore

* Move blockStagingShard to separate folder

* Apply staging shard to acceptanceDataStore

* Update blockHeaderStore with StagingArea

* Add StagingArea to BlockRelationStore

* Add StagingArea to blockStatusStore

* Add StagingArea to consensusStateStore

* Add StagingArea to daaBlocksStore

* Add StagingArea to finalityStore

* Add StagingArea to ghostdagDataStore

* Add StagingArea to headersSelectedChainStore and headersSelectedTipStore

* Add StagingArea to multisetStore

* Add StagingArea to pruningStore

* Add StagingArea to reachabilityDataStore

* Add StagingArea to utxoDiffStore

* Fix forgotten compilation error

* Update reachability manager and some more things with StagingArea

* Add StagingArea to dagTopologyManager, and some more

* Add StagingArea to GHOSTDAGManager, and some more

* Add StagingArea to difficultyManager, and some more

* Add StagingArea to dagTraversalManager, and some more

* Add StagingArea to headerTipsManager, and some more

* Add StagingArea to constnsusStateManager, pastMedianTimeManager

* Add StagingArea to transactionValidator

* Add StagingArea to finalityManager

* Add StagingArea to mergeDepthManager

* Add StagingArea to pruningManager

* Add StagingArea to rest of ValidateAndInsertBlock

* Add StagingArea to blockValidator

* Add StagingArea to coinbaseManager

* Add StagingArea to syncManager

* Add StagingArea to blockBuilder

* Update consensus with StagingArea

* Add StagingArea to ghostdag2

* Fix remaining compilation errors

* Update names of stagingShards

* Fix forgotten stagingArea passing

* Mark stagingShard.isCommited = true once commited

* Move isStaged to stagingShard, so that it's available without going through store

* Make blockHeaderStore count be avilable from stagingShard

* Fix remaining forgotten stagingArea passing

* commitAllChanges should call dbTx.Commit in the end

* Fix all tests tests in blockValidator

* Fix all tests in consensusStateManager and some more

* Fix all tests in pruningManager

* Add many missing stagingAreas in tests

* Fix many tests

* Fix most of all other tests

* Fix ghostdag_test.go

* Add comment to StagingArea

* Make list of StagingShards an array

* Add comment to StagingShardID

* Make sure all staging shards are pointer-receiver

* Undo bucket rename in block_store

* Typo: isCommited -> isCommitted

* Add comment explaining why stagingArea.shards is an array
2021-03-29 10:34:11 +03:00
Ori Newman
d9449a32b8
Use DAA score where needed (#1602)
* Replace blue score with DAA score in UTXO entries

* Use DAA score for coinbase maturity

* Use DAA score for sequence lock

* Fix calcBlockSubsidy to use DAA score

* Don't pay to blocks that are not included in the DAA added blocks, and bestow red blocks reward to the merging block

* Fix TestGetPruningPointUTXOs

* Fix TestTransactionAcceptance

* Fix TestChainedTransactions

* Fix TestVirtualDiff

* Fix TestBlockWindow

* Fix TestPruning

* Use NewFromSlice instead of manually creating the hash set

* Add assert

* Add comment

* Remove redundant call to UpdateDAADataAndReturnDifficultyBits

* Add RequiredDifficulty, rename UpdateDAADataAndReturnDifficultyBits to StageDAADataAndReturnRequiredDifficulty and add comments

* Make buildUTXOInvalidHeader get bits as an argument

* Fix comments
2021-03-15 13:48:40 +02:00
Ori Newman
2adb4f5d0f
Fix UTXO index (#1548)
* Add VirtualUTXODiff and VirtualParents to block insertion result

* Add GetVirtualUTXOs

* Add OnPruningPointUTXOSetOverrideHandler

* Add recovery to UTXO index

* Add UTXO set override notification

* Fix compilation error

* Fix iterators in UTXO index and fix TestUTXOIndex

* Change Dialing to DEBUG

* Change LogBlock location

* Rename StopNotify to StopNotifying

* Add sanity check

* Add comment

* Remove receiver from serialization functions

Co-authored-by: Elichai Turkel <elichai.turkel@gmail.com>
2021-02-23 16:51:51 +02:00
Elichai Turkel
2d61a67592
Change some logs (#1511) 2021-02-09 14:00:02 +02:00
Ori Newman
238950cb98
Add logs (#1484)
* Add logs

* Fix log name
2021-02-03 17:47:59 +02:00
stasatdaglabs
97be133cee
Add logs to help debug long virtual parent selection. (#1470)
Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-02-01 19:15:50 +02:00
Ori Newman
741e0962be
Remove diffs from restoreUTXO logs (#1354)
Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-01-06 10:31:42 +02:00
Elichai Turkel
e509cb1597
Smal performance improvements to BlueWindow (#1343)
* Convert BlockGHOSTDAGData from an interface to a public struct with getters

* Move hashes.Less to externalapi so it can access the hashes directly without copying

* Reduce calls to ghostdagstore.Get in blueWindow

* Simplify the logic in RequiredDifficulty and reuse big.Int

* Remove bigintpool as its no longer used

* Use ChooseSelectedParent in RequiredDifficulty instead of looping over the parents

* Remove comment
2021-01-05 12:13:02 +02:00
Svarog
789a7379bd
Require only inputs not be prefilled (#1345)
* bug invalidateAndInsertPruningPoint: if ValidateAndInsertBlock returned a non-RuleError error - the error was ignored

* Convert checkNoPrefilledFields into checkNoPrefilledInputs

* Add log line

* clone pruning point when passing to validateBlockTransactionsAgainstPastUTXO
2021-01-04 15:55:08 +02:00
stasatdaglabs
b473a09111
Fix a crash in the UTXO index (#1332)
* Add a field to TransactionAcceptanceData: TransactionInputUTXOEntries.

* Fix failing tests.

* Add transactionInputUtxoEntries to the database.

* Populate transactionInputUTXOEntries in applyMergeSetBlocks and use them in the UTXO index.

* Remove UTXOEntry.Clone().

* Add an additional equality test.
2020-12-31 14:50:14 +02:00
Svarog
0f93189c16
Don't print the whole UTXODiff to log, it might be quite huge (#1318) 2020-12-29 17:16:38 +02:00
Ori Newman
d91afbfe3b
Change most Tracef to Debugf (#1302)
* Change most Tracef to Debugf

* Remove diff from log
2020-12-29 10:32:28 +02:00
Ori Newman
8aecf961bc
Red inclusion (#1275)
* Accept red blocks transactions

* Add comments to TestTransactionAcceptance

* Fix tests

* Remove fetchUTXOSetIfMissing

* Remove redundant dependency

* Fix comments
2020-12-24 18:12:46 +02:00
Ori Newman
cb9d7e313d
Implement Clone and Equal for all model types (#1155)
* [NOD-1575] Implement Clone and Equal for all model types

* [NOD-1575] Add assertion for transaction ID equality

* [NOD-1575] Use DomainTransaction.Equal to compare to expected coinbase transaction

* [NOD-1575] Add TestDomainBlockHeader_Clone

* [NOD-1575] Don't clone nil values

* [NOD-1575] Add type assertions

* [NOD-1575] Don't clone nil values

* [NOD-1575] Add missing Equals

* [NOD-1575] Add length checks

* [NOD-1575] Update comment

* [NOD-1575] Check length for TransactionAcceptanceData

* [NOD-1575] Explicitly clone nils where needed

* [NOD-1575] Clone tx id

* [NOD-1575] Flip condition

* Nod 1576 make coverage tests for equal clone inside model externalapi (#1177)

* [NOD-1576] Make coverage tests for equal and clone inside model and externalapi

* Some formatting and naming fixes

* Made transactionToCompare type exported

* Added some tests and made some changes to the tests code

* No changes made

* Some formatting and naming changes made

* Made better test coverage for externalapi clone and equal functions

* Changed expected result for two cases

* Added equal and clone functions tests for ghostdag and utxodiff

* Added tests

* [NOD-1576] Implement reachabilitydata equal/clone unit tests

* [NOD-1576]  Full coverage of reachabilitydata equal/clone unit tests

* Made changes and handling panic to transaction_equal_clone_test.go and formating of utxodiff_equal_clone_test.go

* Added recoverForEqual2 for handling panic to transaction_equal_clone_test.go

* [NOD-1576]  Full coverage of transaction equal unit test

* [NOD-1576] Add expects panic

* [NOD-1576] Allow composites in go vet

* [NOD-1576] Code review fixes (#1223)

* [NOD-1576] Code review fixes

* [NOD-1576] Code review fixes part 2

* [NOD-1576] Fix wrong name

Co-authored-by: karim1king <karimkaspersky@yahoo.com>
Co-authored-by: Ori Newman <orinewman1@gmail.com>
Co-authored-by: Karim <karim1king@users.noreply.github.com>

* Fix merge errors

* Use Equal where possible

* Use Equal where possible

* Use Equal where possible

Co-authored-by: andrey-hash <74914043+andrey-hash@users.noreply.github.com>
Co-authored-by: karim1king <karimkaspersky@yahoo.com>
Co-authored-by: Karim <karim1king@users.noreply.github.com>
2020-12-22 17:38:54 +02:00
stasatdaglabs
21a459c0f4
Implement virtual selected parent chain RPC methods (#1249)
* [NOD-1579] Rename AcceptedTxIDs to AcceptedTransactionIDs.

* [NOD-1579] Add InsertBlockResult to ValidateAndInsertBlock results.

* [NOD-1593] Rename InsertBlockResult to BlockInsertionResult.

* [NOD-1593] Add SelectedParentChainChanges to AddBlockToVirtual's result.

* [NOD-1593] Implement findSelectedParentChainChanges.

* [NOD-1593] Implement TestFindSelectedParentChainChanges.

* [NOD-1593] Fix a string.

* [NOD-1593] Finish implementing TestFindSelectedParentChainChanges.

* [NOD-1593] Fix merge errors.

* [NOD-1597] Begin implementing UTXOIndex.

* [NOD-1597] Connect UTXOIndex to RPC.

* [NOD-1597] Connect Consensus to UTXOIndex.

* [NOD-1597] Add AcceptanceData to BlockInfo.

* [NOD-1597] Implement UTXOIndex.Update().

* [NOD-1597] Implement add(), remove(), and discard() in utxoIndexStore.

* [NOD-1597] Add error cases to add() and remove().

* [NOD-1597] Add special cases to add() and remove().

* [NOD-1597] Implement commit.

* [NOD-1597] Add a mutex around UTXOIndex.Update().

* [NOD-1597] Return changes to the UTXO from Update().

* [NOD-1597] Add NotifyUTXOsChangedRequestMessage and related structs.

* [NOD-1597] Implement HandleNotifyUTXOsChanged.

* [NOD-1597] Begin implementing TestUTXOIndex.

* [NOD-1597] Implement RegisterForUTXOsChangedNotifications.

* [NOD-1597] Fix bad transaction.ID usage.

* [NOD-1597] Implement convertUTXOChangesToUTXOsChangedNotification.

* [NOD-1597] Make UTXOsChangedNotificationMessage.Removed UTXOsByAddressesEntry instead of just RPCOutpoint so that the client can discern which address was the UTXO removed for.

* [NOD-1597] Collect outpoints in TestUTXOIndex.

* [NOD-1597] Rename RPC stuff.

* [NOD-1597] Add messages for GetUTXOsByAddresses.

* [NOD-1597] Implement HandleGetUTXOsByAddresses.

* [NOD-1597] Implement GetUTXOsByAddresses.

* [NOD-1597] Implement UTXOs().

* [NOD-1597] Implement getUTXOOutpointEntryPairs().

* [NOD-1597] Expand TestUTXOIndex.

* [NOD-1597] Convert SubmitTransaction to use RPCTransaction instead of MsgTx.

* [NOD-1597] Finish implementing TestUTXOIndex.

* [NOD-1597] Add messages for GetVirtualSelectedParentBlueScore.

* [NOD-1597] Implement HandleGetVirtualSelectedParentBlueScore and GetVirtualSelectedParentBlueScore.

* [NOD-1597] Implement TestVirtualSelectedParentBlueScore.

* [NOD-1597] Implement NotifyVirtualSelectedParentBlueScoreChanged.

* [NOD-1597] Expand TestVirtualSelectedParentBlueScore.

* [NOD-1597] Implement notifyVirtualSelectedParentBlueScoreChanged.

* [NOD-1597] Make go lint happy.

* [NOD-1593] Fix merge errors.

* [NOD-1593] Rename findSelectedParentChainChanges to calculateSelectedParentChainChanges.

* [NOD-1593] Expand TestCalculateSelectedParentChainChanges.

* [NOD-1597] Add logs to utxoindex.go.

* [NOD-1597] Add logs to utxoindex/store.go.

* [NOD-1597] Add logs to RPCManager.NotifyXXX functions.

* Implement notifySelectedParentChainChanged.

* Implement TestSelectedParentChain.

* Rename NotifyChainChanged to NotifyVirtualSelectedParentChainChanged.

* Rename GetChainFromBlock to GetVirtualSelectedParentChainFromBlock.

* Remove AcceptanceIndex from the config.

* Implement HandleGetVirtualSelectedParentChainFromBlock.

* Expand TestVirtualSelectedParentChain.

* Fix merge errors.

* Add a comment.

* Move a comment.
2020-12-21 14:43:32 +02:00
stasatdaglabs
053bb351b5
[NOD-1597] Implement a UTXO index (#1221)
* [NOD-1579] Rename AcceptedTxIDs to AcceptedTransactionIDs.

* [NOD-1579] Add InsertBlockResult to ValidateAndInsertBlock results.

* [NOD-1593] Rename InsertBlockResult to BlockInsertionResult.

* [NOD-1593] Add SelectedParentChainChanges to AddBlockToVirtual's result.

* [NOD-1593] Implement findSelectedParentChainChanges.

* [NOD-1593] Implement TestFindSelectedParentChainChanges.

* [NOD-1593] Fix a string.

* [NOD-1593] Finish implementing TestFindSelectedParentChainChanges.

* [NOD-1593] Fix merge errors.

* [NOD-1597] Begin implementing UTXOIndex.

* [NOD-1597] Connect UTXOIndex to RPC.

* [NOD-1597] Connect Consensus to UTXOIndex.

* [NOD-1597] Add AcceptanceData to BlockInfo.

* [NOD-1597] Implement UTXOIndex.Update().

* [NOD-1597] Implement add(), remove(), and discard() in utxoIndexStore.

* [NOD-1597] Add error cases to add() and remove().

* [NOD-1597] Add special cases to add() and remove().

* [NOD-1597] Implement commit.

* [NOD-1597] Add a mutex around UTXOIndex.Update().

* [NOD-1597] Return changes to the UTXO from Update().

* [NOD-1597] Add NotifyUTXOsChangedRequestMessage and related structs.

* [NOD-1597] Implement HandleNotifyUTXOsChanged.

* [NOD-1597] Begin implementing TestUTXOIndex.

* [NOD-1597] Implement RegisterForUTXOsChangedNotifications.

* [NOD-1597] Fix bad transaction.ID usage.

* [NOD-1597] Implement convertUTXOChangesToUTXOsChangedNotification.

* [NOD-1597] Make UTXOsChangedNotificationMessage.Removed UTXOsByAddressesEntry instead of just RPCOutpoint so that the client can discern which address was the UTXO removed for.

* [NOD-1597] Collect outpoints in TestUTXOIndex.

* [NOD-1597] Rename RPC stuff.

* [NOD-1597] Add messages for GetUTXOsByAddresses.

* [NOD-1597] Implement HandleGetUTXOsByAddresses.

* [NOD-1597] Implement GetUTXOsByAddresses.

* [NOD-1597] Implement UTXOs().

* [NOD-1597] Implement getUTXOOutpointEntryPairs().

* [NOD-1597] Expand TestUTXOIndex.

* [NOD-1597] Convert SubmitTransaction to use RPCTransaction instead of MsgTx.

* [NOD-1597] Finish implementing TestUTXOIndex.

* [NOD-1597] Add messages for GetVirtualSelectedParentBlueScore.

* [NOD-1597] Implement HandleGetVirtualSelectedParentBlueScore and GetVirtualSelectedParentBlueScore.

* [NOD-1597] Implement TestVirtualSelectedParentBlueScore.

* [NOD-1597] Implement NotifyVirtualSelectedParentBlueScoreChanged.

* [NOD-1597] Expand TestVirtualSelectedParentBlueScore.

* [NOD-1597] Implement notifyVirtualSelectedParentBlueScoreChanged.

* [NOD-1597] Make go lint happy.

* [NOD-1593] Fix merge errors.

* [NOD-1593] Rename findSelectedParentChainChanges to calculateSelectedParentChainChanges.

* [NOD-1593] Expand TestCalculateSelectedParentChainChanges.

* [NOD-1597] Add logs to utxoindex.go.

* [NOD-1597] Add logs to utxoindex/store.go.

* [NOD-1597] Add logs to RPCManager.NotifyXXX functions.

* [NOD-1597] Ignore transactions that aren't accepted.

* [NOD-1597] Use GetBlockAcceptanceData instead of GetBlockInfo.

* [NOD-1597] Convert scriptPublicKey to string directly, instead of using hex.

* [NOD-1597] Add a comment.

* [NOD-1597] Guard against calling utxoindex methods when utxoindex is turned off.

* [NOD-1597] Add lock to UTXOs.

* [NOD-1597] Guard against calls to getUTXOOutpointEntryPairs when staging isn't empty.
2020-12-20 17:24:56 +02:00
Ori Newman
bd5f4e8c6a Pruning fixes (#1243)
* Pruning related fixes

* Rename setBlockStatus->setBlockStatusAfterBlockValidation

* Rename StatusValid->StatusUTXOValid

* Add comment

* Fix typo

* Rename hasValidatedOnlyHeader->hasValidatedHeader

* Rename checkBlockBodiesExist->checkParentBlockBodiesExist

* Add comments and logs

* Adding logs

* Add logs and assert

* Add comment

* Fix typo

* Fix log
2020-12-20 09:38:34 +02:00
Svarog
33eaf9edac
[NOD-1548] Re-add test difficulty + Make GHOSTDAGData immutable + don't clone in store (#1178)
* [NOD-1548] Readd TestDifficulty

* [NOD-1548] Make GHOSTDAGData immutable + don't clone in store

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2020-12-06 12:35:14 +02:00
Ori Newman
32a04d1811
Allow to configure consensus (closes #1067)
* Allow to configure consensus with a JSON file

* Define everywhere maxBlockParents as KType

* Move consensus default to consensus_defaults.go
2020-12-03 18:30:01 +02:00
Svarog
a585f32763
[NOD-1551] Make UTXODiff immutable + skip cloning it in datastore (#1167)
* [NOD-1551] Make UTXO-Diff implemented fully in utils/utxo

* [NOD-1551] Fixes everywhere except database

* [NOD-1551] Fix database

* [NOD-1551] Add comments

* [NOD-1551] Partial commit

* [NOD-1551] Comlete making UTXOEntry immutable + don't clone it in UTXOCollectionClone

* [NOD-1551] Rename ToUnmutable -> ToImmutable

* [NOD-1551] Track immutable references generated from mutable UTXODiff, and invalidate them if the mutable one changed

* [NOD-1551] Clone scriptPubKey in NewUTXOEntry

* [NOD-1551] Remove redundant code

* [NOD-1551] Remove redundant call for .CloneMutable and then .ToImmutable

* [NOD-1551] Make utxoEntry pointert-receiver + clone ScriptPubKey in getter
2020-12-03 13:24:24 +02:00
Mike Zak
9866abb75a [NOD-1583] Split consensusserialization to consensushashing and serialization 2020-12-02 13:18:50 +02:00
Svarog
546ea83123
[NOD-1570] Fix the way UTXO iterators work (#1153)
* [NOD-1570] Implement utxo.IteratorWithDiff

* [NOD-1570] Utilize utxo.ITeratorWithDiff in RestorePastUTXOSetIterator and VirtualUTXOSetIterator

* [NOD-1570] Fix comment
2020-11-25 18:28:42 +02:00
Ori Newman
45d9b63572
[NOD-1567] Add clone methods to data stores types (#1149)
* [NOD-1567] Add clone methods to data stores types

* [NOD-1567] Fix comments

* [NOD-1567] Fix test
2020-11-24 17:56:18 +02:00
Mike Zak
5211727206 [NOD-1557] Cover the consensusStateManager package in trace logs (#1135)
* [NOD-1557] Add trace logs in add_block_to_virtual.go.

* [NOD-1557] Add trace logs in resolve_block_status.go.

* [NOD-1557] Add trace logs in calculate_past_utxo.go.

* [NOD-1557] Add trace logs in finality.go.

* [NOD-1557] Add trace logs in multisets.go.

* [NOD-1557] Fix compilation errors.

* [NOD-1557] Add trace logs to verify_and_build_utxo.go.

* [NOD-1557] Add trace logs to update_virtual.go.

* [NOD-1557] Add trace logs to set_pruning_utxo_set.go.

* [NOD-1557] Add trace logs to populate_tx_with_utxo_entries.go.

* [NOD-1557] Add trace logs to pick_virtual_parents.go.

* [NOD-1557] Make go vet happy.

* [NOD-1557] Clarify that some logic in AddBlockToVirtual is there for the sake of logging alone.

* [NOD-1557] Call blockStatusStore directly in AddBlockToVirtual when refetching the block status.
2020-11-23 13:08:10 +02:00
Ori Newman
60c24d8dea Fix TestBlueBlockWindow (#1098)
* Fix TestBlueBlockWindow

* Add comments
2020-11-17 16:00:16 +02:00
Mike Zak
d8f72e2b27 [NOD-1532] Update VirtualUTXODiffParents diffs even if list didn't change 2020-11-17 16:00:16 +02:00
Mike Zak
83a88d9989 [NOD-1532] newUTXOSetIterator should start with -1 index 2020-11-17 16:00:16 +02:00
Ori Newman
56679818be [NOD-1535] Add non coinbase transactions to diff (#1084) 2020-11-17 16:00:16 +02:00
Mike Zak
34be898491 [NOD-1532] utxoSetIterator should be a pointer receiver 2020-11-17 16:00:16 +02:00
Svarog
8ccf381fc7 [NOD-1532] csm unit tests (#1059)
* Revert "[NOD-1500] Delete integration tests"

This reverts commit fcb57a206690a884fa6afb69d5d493282954a8bf.

* [NOD-1518] hashserialization -> consenusserialization

* [NOD-1518] Fix add genesis to virtual

* [NOD-1518] Fix a bug in SerializeCoinbasePayload.

* [NOD-1518] Fix a loop error and make pastMedianTime behave correctly everywhere on genesis.

* [NOD-1518] Fix another bug and an infinite loop.

* [NOD-1518] Fix uninitialized slice.

* [NOD-1518] Fix bad should-commit checks and another infinite loop.

* [NOD-1518] Fix nil serialization.

* [NOD-1518] Rename blockHash to currentBlockHash.

* [NOD-1518] Move the check whether stagedVirtualUTXOSet != nil to the top of commitVirtualUTXODiff.

* [NOD-1518] Simplify utxoDiffStore.Commit.

* [NOD-1518] Unextract resolveBlockStatusAndCheckFinality.

* [NOD-1518] Move no-transactions logic into CalculateIDMerkleRoot.

* [NOD-1518] Remove redundant is-staged check.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Don't write anything if utxoDiffChild is nil.

* [NOD-1518] Stage virtualAcceptanceData and virtualMultiset.

* [NOD-1518] Fix bugs in getBlockTemplate and submitBlock.

* [NOD-1518] Fix bad validation order in validateHeaderInContext.

* [NOD-1518] Fix bug in Next().

* [NOD-1518] Fix nil dereference of subnetworks in AddressCache.

* [NOD-1518] Fix multisetStore.Get returning a pointer to a multiset that is changed in place.

* [NOD-1518] Break on genesis in countSubtrees.

* [NOD-1518] Fix createBlockLocator.

* [NOD-1518] Fix MsgTxToDomainTransaction.

* [NOD-1518] Set MaxTxVersion to 1.

* [NOD-1518] Fix missing error handling, bug in MsgTxToDomainTransaction, and bad subnetwork equality check.

* [NOD-1518] Fix bug in hasUTXOByOutpointFromStagedVirtualUTXODiff.

* [NOD-1518] Remove irrelevant comments.

* [NOD-1518] Generate transactions with sufficient fee in tx_relay_test.

* [NOD-1518] Fix broken RPC handlers.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Fix bad exists check in restorePastUTXO and missing genesis check in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Add a comment.

* [NOD-1518] Use a regular mutex instead of a read-write mutex in consensus to avoid dealing with sneaky not-actually-read functions.

* [NOD-1518] Fix a deadlock in GetVirtualSelectedParent.

* [NOD-1518] Fix missing handler registration for CmdHeader.

* [NOD-1518] Fix processHeader calling OnNewBlock and LogBlock. Also fix conversion errors in IBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad Command() in MsgIBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad SyncStateMissingUTXOSet logic in resolveSyncState.

* [NOD-1518] Rename mode to syncState.

* [NOD-1518] Fix headers-only blocks coming in after the consensus thinks it's synced.

* [NOD-1518] Fix selectedChildIterator.Next not ignoring virtual, infinite loop in HashSet.Length().

* [NOD-1518] Fix not-properly wrapped IBD blocks.

* [NOD-1532] Add TestMultiset

* [NOD-1518] Fix bad conversion in RequestIBDBlocks.

* [NOD-1518] Fix bad string for CmdRequestHeaders.

* [NOD-1518] Fix bad string for CmdDoneHeaders.

* [NOD-1518] Fix bad Command() for MsgIBDRootNotFound.

* [NOD-1532] Add TestPastUTXOMultiset

* [NOD-1518] Fix bad areHeaderTipsSyncedMaxTimeDifference value.

* [NOD-1532] Added TestDoubleSpends

* [NOD-1518] Add missing string for CmdRequestIBDBlocks.

* [NOD-1518] Fix bad check for SyncStateMissingBlockBodies.

* [NOD-1518] Fix bad timeout durations in tests.

* [NOD-1518] Fix IBD blocks not calling OnNewBlock.

* [NOD-1518] Change when IBD finishes.

* [NOD-1518] Properly clone utxoDiffChild.

* [NOD-1532] Update hashes of blocks

* [NOD-1532] Fix genesis blocks and a few more bugs

* [NOD-1532] Bugfix: incorrect key passed to dbTx.Put

* [NOD-1532] Make sure there's no nil payloads

* [NOD-1532] Fix AddBlockToVirtual

* [NOD-1532] Update tips and virtualDiffParents properly

* [NOD-1532] Allow nil payload

* [NOD-1532] Check for actual error and not just some RuleError

* [NOD-1532] Get rid of SimpleCoinbaseData and make OpTrueScript P2SH

* [NOD-1532] If coinbaseData is nil - fill in with generic coinbaseData

Co-authored-by: Ori Newman <orinewman1@gmail.com>
Co-authored-by: stasatdaglabs <stas@daglabs.com>
2020-11-17 16:00:16 +02:00
stasatdaglabs
eef5e3768c
[NOD-1518] Fix genesis block insertion and integration tests (#1013)
* Revert "[NOD-1500] Delete integration tests"

This reverts commit fcb57a206690a884fa6afb69d5d493282954a8bf.

* [NOD-1518] hashserialization -> consenusserialization

* [NOD-1518] Fix add genesis to virtual

* [NOD-1518] Fix a bug in SerializeCoinbasePayload.

* [NOD-1518] Fix a loop error and make pastMedianTime behave correctly everywhere on genesis.

* [NOD-1518] Fix another bug and an infinite loop.

* [NOD-1518] Fix uninitialized slice.

* [NOD-1518] Fix bad should-commit checks and another infinite loop.

* [NOD-1518] Fix nil serialization.

* [NOD-1518] Rename blockHash to currentBlockHash.

* [NOD-1518] Move the check whether stagedVirtualUTXOSet != nil to the top of commitVirtualUTXODiff.

* [NOD-1518] Simplify utxoDiffStore.Commit.

* [NOD-1518] Unextract resolveBlockStatusAndCheckFinality.

* [NOD-1518] Move no-transactions logic into CalculateIDMerkleRoot.

* [NOD-1518] Remove redundant is-staged check.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Don't write anything if utxoDiffChild is nil.

* [NOD-1518] Stage virtualAcceptanceData and virtualMultiset.

* [NOD-1518] Fix bugs in getBlockTemplate and submitBlock.

* [NOD-1518] Fix bad validation order in validateHeaderInContext.

* [NOD-1518] Fix bug in Next().

* [NOD-1518] Fix nil dereference of subnetworks in AddressCache.

* [NOD-1518] Fix multisetStore.Get returning a pointer to a multiset that is changed in place.

* [NOD-1518] Break on genesis in countSubtrees.

* [NOD-1518] Fix createBlockLocator.

* [NOD-1518] Fix MsgTxToDomainTransaction.

* [NOD-1518] Set MaxTxVersion to 1.

* [NOD-1518] Fix missing error handling, bug in MsgTxToDomainTransaction, and bad subnetwork equality check.

* [NOD-1518] Fix bug in hasUTXOByOutpointFromStagedVirtualUTXODiff.

* [NOD-1518] Remove irrelevant comments.

* [NOD-1518] Generate transactions with sufficient fee in tx_relay_test.

* [NOD-1518] Fix broken RPC handlers.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Fix bad exists check in restorePastUTXO and missing genesis check in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Add a comment.

* [NOD-1518] Use a regular mutex instead of a read-write mutex in consensus to avoid dealing with sneaky not-actually-read functions.

* [NOD-1518] Fix a deadlock in GetVirtualSelectedParent.

* [NOD-1518] Fix missing handler registration for CmdHeader.

* [NOD-1518] Fix processHeader calling OnNewBlock and LogBlock. Also fix conversion errors in IBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad Command() in MsgIBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad SyncStateMissingUTXOSet logic in resolveSyncState.

* [NOD-1518] Rename mode to syncState.

* [NOD-1518] Fix headers-only blocks coming in after the consensus thinks it's synced.

* [NOD-1518] Fix selectedChildIterator.Next not ignoring virtual, infinite loop in HashSet.Length().

* [NOD-1518] Fix not-properly wrapped IBD blocks.

* [NOD-1518] Fix bad conversion in RequestIBDBlocks.

* [NOD-1518] Fix bad string for CmdRequestHeaders.

* [NOD-1518] Fix bad string for CmdDoneHeaders.

* [NOD-1518] Fix bad Command() for MsgIBDRootNotFound.

* [NOD-1518] Fix bad areHeaderTipsSyncedMaxTimeDifference value.

* [NOD-1518] Add missing string for CmdRequestIBDBlocks.

* [NOD-1518] Fix bad check for SyncStateMissingBlockBodies.

* [NOD-1518] Fix bad timeout durations in tests.

* [NOD-1518] Fix IBD blocks not calling OnNewBlock.

* [NOD-1518] Change when IBD finishes.

* [NOD-1518] Properly clone utxoDiffChild.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Move call to LogBlock to into OnNewBlock.

* [NOD-1518] Return "not implemented" in unimplemented RPC handlers.

* [NOD-1518] Extract cloning of hashes to a method over DomainHash.

* [NOD-1518] Use isHeaderOnlyBlock.

* [NOD-1518] Use constants.TransactionVersion.

* [NOD-1518] Break immediately if we reached the virtual in SelectedChildIterator.

* [NOD-1518] Don't stage nil utxoDiffChild.

* [NOD-1518] Properly check the genesis hash in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Explain why we break on current == nil in countSubtrees.

* [NOD-1518] Add a comment explaining why we check against StatusValid in resolveSyncState.

Co-authored-by: Mike Zak <feanorr@gmail.com>
Co-authored-by: Ori Newman <orinewman1@gmail.com>
2020-11-12 15:19:39 +02:00
Ori Newman
23c1ea6c31
[NOD-1525] Implement headers first ibd (#1017)
* [NOD-1525] Implement headers first IBD

* [NOD-1525] Fix proto translators

* [NOD-1525] Register missing flows

* [NOD-1525] Rename SyncStateNormal->SyncStateRelay, simplifiy IBD peer selection code and get rid of panic in FinishIBD

* [NOD-1525] Remove redundant methods from interface
2020-11-10 16:14:51 +02:00
Svarog
6db337c8c5
[NOD-1519] Add TestAPI for BuildBlockWithParents (#1011)
* [NOD-1519] Separate BlockBuilder and BlockProcessor

* [NOD-1519] Wire blockBuilder properly + implement buildBlockWithParents

* [NOD-1519] Added testapi package, TestConsensus interface, and TestConsensus factory

* [NOD-1519] Add comments

* [NOD-1519] Separate TestBlockBuilder out of BlockBuilder

* [NOD-1519] TestBlockBuilder should also implement BlockBuilder

* [NOD-1519] Add NewTestConsensus to factory interface
2020-11-09 12:02:42 +02:00
Ori Newman
5566aaf95a
[NOD-1512] Implement utxo deserialization (#1003)
* [NOD-1512] Implement UTXO set deserialization

* [NOD-1512] Remove redundant file

* [NOD-1512] Don't use big endian for serialization

* [NOD-1512] Use Read/Write element

* [NOD-1512] Unexport ReadElement

* [NOD-1512] Fix StageVirtualUTXOSet

* [NOD-1512] Get rid of dagParams in consensusStateManager

* [NOD-1512] Get rid of dagParams in consensusStateManager
2020-11-05 10:59:49 +02:00
Ori Newman
c7f2de73df
[NOD-1502] Implement RestorePastUTXOSetIterator (#993)
* [NOD-1502] Implement RestorePastUTXOSetIterator

* [NOD-1502] Rename newUtxoSetIterator->newUTXOSetIterator
2020-11-02 18:36:55 +02:00
Svarog
c5707f64dc
[NOD-1420] Implement consensusStateManager (#985)
* [NOD-1420] Start working on ConsensusStateManager. Might be redundant due to recent changes

* [NOD-1420] Convert model to externalapi in utxo_algerbra helpers

* [NOD-1420] Add UTXO-diff algebra

* [NOD-1420] Prepare skeleton of calculateAcceptanceDataAndMultiset

* [NOD-1420] Added skeleton for AddBlockToVirtual

* [NOD-1420] Implement PopulateTransactionWithUTXOEntries

* [NOD-1420] Implement restorePastUTXO

* [NOD-1420] Implement finality check

* [NOD-1420] Move handling of tips to consensusStateManager

* [NOD-1420] Implement calculateAcceptanceDataAndMultiset

* [NOD-1420] Start implementing resolveBlockStatus

* [NOD-1420] Implement resolveBlockStatus

* [NOD-1420] Update related fields in end of resolveSingleBlockStatus

* [NOD-1420] Start working on selectVirtualParents

* [NOD-1420] Implemented BlockHeap

* [NOD-1420] Implement selectVirtualParents

* [NOD-1420] Implement updateVirtual

* [NOD-1420] Added comments where they were missing

* [NOD-1420] Place all consensusStateManager functions in correct files

* [NOD-1420] Return the missing outpoints from populateTransactionWithUTXOEntriesFromVirtualOrDiff

* [NOD-1420] Outpoint.ID -> TransactionID

* [NOD-1420] Fix Stringer tests

* [NOD-1420] Copy hash.FromString into utils

* [NOD-1420] SetParents should return an error

* [NOD-1420] Remove all reachabilityManager references from consensusStateManager

* [NOD-1420] Remove VirtualData. Get the info from the stores where needed

* [NOD-1420] Invert parameters to IsAncestorOf

* [NOD-1420] Use model.AcceptanceData

* [NOD-1420] Don't return accumulatedMassBefore in error cases

* [NOD-1420] Don't expect store functions to return nil when the requested data was found - instead add HasXXX functions

* [NOD-1420] addTransactionToMultiset sets isCoinbase properly

* [NOD-1420] expected hash string length is externalapi.DomainHashSize * 2

* [NOD-1420] Rename reachabilityTree -> reachabilityManager + updateReindexRoot if isNextVirtualSelectedParent

* [NOD-1420] ValidateCoinbaseTransaction in csm.verifyAndBuildUTXO

* [NOD-1420] Re-write HAsUTXODiffChild

* [NOD-1420] delete past_utxo.go.bak

* [NOD-1420] Implement validateCoinbaseTransaction in CSM

* [NOD-1420] Imlemented missing functionality in ValidateTransactionAndPopulateWithConsensusData

* [NOD-1420] Moved merge depth logic to MergeDepthManager

* [NOD-1420] Add logs
2020-11-02 16:18:53 +02:00