73 Commits

Author SHA1 Message Date
Ori Newman
6f53da18b1
Increase stores cache (#1485)
Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-02-04 10:06:02 +02:00
stasatdaglabs
ca04c049ab
When the pruning point moves, update its UTXO set outside of a database transaction (#1444)
* Remove pruningPointUTXOSetStaging and implement UpdatePruningPointUTXOSet.

* Implement StageStartSavingNewPruningPointUTXOSet, HadStartedSavingNewPruningPointUTXOSet, and FinishSavingNewPruningPointUTXOSet.

* Fix a bad return.

* Implement UpdatePruningPointUTXOSetIfRequired.

* Call UpdatePruningPointUTXOSetIfRequired on consensus creation.

* Rename savingNewPruningPointUTXOSetKey to updatingPruningPointUTXOSet.

* Add a log.

* Add calls to runtime.GC() at its start and end.

* Rename a variable.

* Replace calls to runtime.GC to calls to LogMemoryStats.

* Wrap the contents of LogMemoryStats in a log closure.
2021-01-24 14:48:11 +02:00
stasatdaglabs
756f40c59a
Sync pruning point UTXO sets incrementally instead of all at once (#1431)
* Replaced the content of MsgIBDRootUTXOSetChunk with pairs of outpoint-utxo entry pairs.

* Rename utxoIter to utxoIterator.

* Add a big stinky TODO on an assert.

* Replace pruningStore staging with a UTXO set iterator.

* Reimplement receiveAndInsertIBDRootUTXOSet.

* Extract OutpointAndUTXOEntryPairsToDomainOutpointAndUTXOEntryPairs into domainconverters.go.

* Pass the outpoint and utxy entry pairs to the pruning store.

* Implement InsertCandidatePruningPointUTXOs.

* Implement ClearCandidatePruningPointUTXOs.

* Implement UpdateCandidatePruningPointMultiset.

* Use the candidate pruning point multiset in updatePruningPoint.

* Implement CandidatePruningPointUTXOIterator.

* Use the pruning point utxo set iterator for StageVirtualUTXOSet.

* Defer ClearCandidatePruningPointUTXOs.

* Implement OverwriteVirtualUTXOSet.

* Implement CommitCandidatePruningPointUTXOSet.

* Implement BeginOverwritingVirtualUTXOSet and FinishOverwritingVirtualUTXOSet.

* Implement overwriteVirtualUTXOSetAndCommitPruningPointUTXOSet.

* Rename ClearCandidatePruningPointUTXOs to ClearCandidatePruningPointData.

* Add missing methods to dbManager.

* Implement PruningPointUTXOs.

* Implement RecoverUTXOIfRequired.

* Delete the utxoserialization package.

* Fix compilation errors in TestValidateAndInsertPruningPoint.

* Switch order of operations in the if statements in PruningPointUTXOs so that Next() wouldn't be unnecessarily called.

* Fix missing pruning point utxo set staging and bad slice length.

* Fix no default multiset in InsertCandidatePruningPointUTXOs.

* Make go vet happy.

* Rename candidateXXX to importedXXX.

* Do some more renaming.

* Rename some more.

* Fix bad MsgIBDRootNotFound logic.

* Fix an error message.

* Simplify receiveIBDRootBlock.

* Fix error message in receiveAndInsertIBDRootUTXOSet.

* Do some more renaming.

* Fix merge errors.

* Fix a bug caused by calling iterator.First() unnecessarily.

* Remove databaseContext from stores and don't use a transaction in ClearXXX functions.

* Simplify receiveAndInsertIBDRootUTXOSet.

* Fix offset count in PruningPointUTXOs().

* Fix readOnlyUTXOIteratorWithDiff.First().

* Split handleRequestIBDRootUTXOSetAndBlockFlow into smaller methods.

* Rename IbdRootNotFound to UnexpectedPruningPoint.

* Rename requestIBDRootHash to requestPruningPointHash.

* Rename IBDRootHash to PruningPointHash.

* Rename RequestIBDRootUTXOSetAndBlock to RequestPruningPointUTXOSetAndBlock.

* Rename IBDRootUTXOSetChunk to PruningPointUTXOSetChunk.

* Rename RequestNextIBDRootUTXOSetChunk to RequestNextPruningPointUTXOSetChunk.

* Rename DoneIBDRootUTXOSetChunks to DonePruningPointUTXOSetChunks.

* Rename remaining references to IBD root.

* Fix an error message.

* Add a check for HadStartedImportingPruningPointUTXOSet in commitVirtualUTXODiff.

* Add a check for HadStartedImportingPruningPointUTXOSet in ImportPruningPointUTXOSetIntoVirtualUTXOSet.

* Move FinishImportingPruningPointUTXOSet closer to HadStartedImportingPruningPointUTXOSet.

* Remove reference to pruningStore in utxoSetIterator.

* Pointerify utxoSetIterator receivers.

* Fix bad insert in CommitImportedPruningPointUTXOSet.

* Rename commitImportedPruningPointUTXOSetAll to applyImportedPruningPointUTXOSet.

* Simplify PruningPointUTXOs.

* Add populateTransactionWithUTXOEntriesFromUTXOSet.

* Fix a TODO comment.

* Rename InsertImportedPruningPointUTXOs to AppendImportedPruningPointUTXOs.

* Extract handleRequestPruningPointUTXOSetAndBlockMessage to a separate method.

* Rename stuff in readOnlyUTXOIteratorWithDiff.First().

* Address toAddIterator in readOnlyUTXOIteratorWithDiff.First().

* Call First() before any full iteration on ReadOnlyUTXOSetIterator.

* Call First() before any full iteration on a database Cursor.

* Put StartImportingPruningPointUTXOSet inside the pruning point transaction.

* Make serializeOutpoint and serializeUTXOEntry free functions in pruningStore.

* Fix readOnlyUTXOIteratorWithDiff.First().

* Fix bad validations in importPruningPoint.

* Remove superfluous call to validateBlockTransactionsAgainstPastUTXO.
2021-01-21 17:24:52 +02:00
Ori Newman
a35f8269ea
Add checkBlockIsNotPruned (#1413)
* Add checkBlockIsNotPruned

* Fix test name and comment

Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-01-14 10:56:22 +02:00
Elichai Turkel
68bd8330ac
Log the networks hashrate (#1406)
* Log the hashrate of each block

* Add a test for GetHashrateString

* Move difficulty related functions to its own package

* Convert the validated log in validateAndInsertBlock to a log function

* Add tests for max/min int
2021-01-13 12:51:23 +02:00
Ori Newman
23a2fbf401
Remove erroneous finality optimization from LowestChainBlockAboveOrEqualToBlueScore (#1402)
* Remove finality erroneous optimization from LowestChainBlockAboveOrEqualToBlueScore

* Add TestLowestChainBlockAboveOrEqualToBlueScore

* Remove unnecessary fields from dagTraversalManager
2021-01-12 15:27:08 +02:00
Ori Newman
b8ca33d91d
Add selected chain store and optimize block locator with it (#1394)
* Add selected chain store and optimize block locator with it

* Fix build error

* Fix comments

* Fix IsStaged

* Rename CalculateSelectedParentChainChanges to CalculateChainPath and SelectedParentChainChanges->SelectedChainPath

* Use binary.LittleEndian directly to allow compiler optimizations

* Remove boolean from HeadersSelectedChainStore interface

* Prevent endless loop in block locator
2021-01-11 15:51:45 +02:00
Elichai Turkel
2cc0bf1639
Optimize block locator using finality store (#1386)
* Make sure block locator doesn't include a hash lower than the lowHash in
the block locator

* Use finalityStore to optimize LowestChainBlockAboveOrEqualToBlueScore
2021-01-10 13:36:02 +02:00
Svarog
49e0a2a2e7
Add basic support for archival node (#1370)
* Add archival cli flag

* If --archival was activated - don't delete anything

* Fix tests

* Still change block status to StatusHeaderOnly even in archival nodes
2021-01-10 10:25:15 +02:00
Svarog
9ea4c0fa38
Add sanity check that makes sure that the utxoset that we save fits the pruningPoints's commitment. (#1366)
Co-authored-by: Elichai Turkel <elichai.turkel@gmail.com>
2021-01-06 13:27:02 +02:00
Elichai Turkel
72a7ca53e6
Save and expose the database in TestConsensus (#1349) 2021-01-05 14:13:33 +02:00
Ori Newman
23304a4977
Add UTXO set cache (#1333)
* Add UTXO set cache

* Add clear method to cache
2020-12-31 16:34:46 +02:00
stasatdaglabs
7dd0188838
Move the heavy lifting in BlockLocator from the syncer to the syncee (#1324)
* Add a new message: BlockLocatorHighestHash.

* Add a new message: IBDBlockLocator.

* Implement HandleIBDBlockLocator.

* Reimplement findHighestSharedBlockHash.

* Make HandleIBDBlockLocator only return hashes that are in the selected parent chain of the target hash.

* Increase the cache sizes of blockRelationStore, reachabilityDataStore, and ghostdagDataStore.

* Fix wrong initial highHash in findHighestSharedBlockHash.

* Make go vet happy.

* Protect against receiving wrong messages when expecting MsgIBDBlockLocatorHighestHash.
2020-12-30 15:44:14 +02:00
Ori Newman
830ddf4735
Fill testConsensus's dagParams (#1283) 2020-12-24 17:47:03 +02:00
Svarog
717914319a
Increase size of reachability and block-relations cache (#1272)
* Increase size of reachability cache

* Increase cache size for BlockRelationStore
2020-12-24 10:27:05 +02:00
Ori Newman
729e3db145
Pruning calculation changes (#1250)
* 1) Calculate pruning point incrementally
2) Add IsValidPruningPoint to pruning manager and consensus
3) Use reachability children for selected child iterator

* Add request IBD root hash flow

* Fix UpdatePruningPointByVirtual and IsValidPruningPoint

* Regenerate messages.pb.go

* Make the pruning point the earliest chain block with finality interval higher than the previous pruning point

* Fix merge errors
2020-12-23 11:37:39 +02:00
stasatdaglabs
f90d7d796a
[NOD-1593] Return SelectedParentChainChanged from ValidateAndInsertBlock (#1202)
* [NOD-1579] Rename AcceptedTxIDs to AcceptedTransactionIDs.

* [NOD-1579] Add InsertBlockResult to ValidateAndInsertBlock results.

* [NOD-1593] Rename InsertBlockResult to BlockInsertionResult.

* [NOD-1593] Add SelectedParentChainChanges to AddBlockToVirtual's result.

* [NOD-1593] Implement findSelectedParentChainChanges.

* [NOD-1593] Implement TestFindSelectedParentChainChanges.

* [NOD-1593] Fix a string.

* [NOD-1593] Finish implementing TestFindSelectedParentChainChanges.

* [NOD-1593] Fix merge errors.

* [NOD-1593] Fix merge errors.

* [NOD-1593] Rename findSelectedParentChainChanges to calculateSelectedParentChainChanges.

* [NOD-1593] Expand TestCalculateSelectedParentChainChanges.
2020-12-15 11:37:52 +02:00
Svarog
77adb6c99f
Make consensus.databaseContext a DBManager and allow keeping data dir in TestConsensus
* Make consensus.databaseContext a DBManager

* Allow keeping data dir in TestConsensus
2020-12-15 10:11:14 +02:00
Ori Newman
48e1a2c396
New headers first flow (#1211)
* Get rid of insertMode

* Rename AddBlockToVirtual->AddBlock

* When F is not in the future of P, enforce finality with P and not with F.

* Don't allow blocks with invalid parents or with missing block body

* Check finality violation before checking block status

* Implement CalculateIndependentPruningPoint

* Move checkBlockStatus to validateBlock

* Add ValidateBlock to block processor interface

* Adjust SetPruningPoint to the new IBD flow

* Add pruning store to CSM's constructor

* Flip wrong condition on AddHeaderTip

* Fix func (hts *headerSelectedTipStore) Has

* Fix block stage order

* Call to ValidateBodyInContext from validatePostProofOfWork

* Enable overrideDAGParams

* Update log

* Rename SetPruningPoint to ValidateAndInsertPruningPoint and move most of its logic inside block processor

* Rename hasValidatedHeader->hasValidatedOnlyHeader

* Fix typo

* Name return values for fetchMissingUTXOSet

* Add comment

* Return ErrMissingParents when block body is missing

* Add logs and comments

* Fix merge error

* Fix pruning point calculation to be by virtual selected parent

* Replace CalculateIndependentPruningPoint to CalculatePruningPointByHeaderSelectedTip

* Fix isAwaitingUTXOSet to check pruning point by headers

* Change isAwaitingUTXOSet indication

* Remove IsBlockInHeaderPruningPointFuture from BlockInfo

* Fix LowestChainBlockAboveOrEqualToBlueScore

* Add validateNewPruningPointTransactions

* Add validateNewPruningAgainstPastUTXO

* Rename set_pruning_utxo_set.go to update_pruning_utxo_set.go

* Check missing block body hashes by missing block instead of status

* Validate pruning point against past UTXO with the pruning point as block hash

* Remove virtualHeaderHash

* Fix comment

* Fix imports
2020-12-14 17:53:08 +02:00
Svarog
b7ca3f4461
[NOD-1590] Implement optimized finalityPoint calculation mechanism (#1190)
* [NOD-1590] Moved all finality logic to FinalityManager

* [NOD-1590] Add finality store

* [NOD-1590] Implement optimized finalityPoint calculation mechanism

* [NOD-1590] Add comments

* [NOD-1590] Add finalityStore to consensus object, and TestConsensus

* [NOD-1590] Added logs to finalityPoint calculation
2020-12-08 10:26:39 +02:00
talelbaz
78550d3639
Adds "checkDelayedBlock" - checks if the block timeStamp is in the future. (#1183)
* [#1056] Adds "checkDelayedBlock" - check if the block timeStamp is in future.
Adds 2 new fields to blockValidator struct.
Adds new Rule Error "ErrDelayedBlock" .

* [#1056] Replace "ErrDelayedBlock" to "ErrBlockIsTooMuchInTheFuture".

* [#1056] Replace "checkDelayedBlock" to "checkBlockTimeStampInIsolation".

* [#1056] Cosmetics changes: timeStamp -> timestamp .

* Merge remote-tracking branch 'origin/v0.8.2-dev' into droppedDelayedBlock

# Conflicts:
#	domain/consensus/factory.go
#	domain/consensus/processes/blockvalidator/block_header_in_isolation.go
#	domain/consensus/processes/blockvalidator/blockvalidator.go

Co-authored-by: tal <tal@daglabs.com>
2020-12-06 18:41:07 +02:00
stasatdaglabs
7f899b0d09
[NOD-1579] Improve the IBD mechanism (#1174)
* [NOD-1579] Remove selected tip hash messages.

* [NOD-1579] Start moving IBD stuff into blockrelay.

* [NOD-1579] Rename relaytransactions to transactionrelay.

* [NOD-1579] Move IBD files into blockrelay.

* [NOD-1579] Remove flow stuff from ibd.go.

* [NOD-1579] Bring back IsInIBD().

* [NOD-1579] Simplify block relay flow.

* [NOD-1579] Check orphan pool for missing parents to avoid unnecessary processing.

* [NOD-1579] Implement processOrphan.

* [NOD-1579] Implement addToOrphanSetAndRequestMissingParents.

* [NOD-1579] Fix TestIBD.

* [NOD-1579] Implement isBlockInOrphanResolutionRange.

* [NOD-1579] Implement limited block locators.

* [NOD-1579] Add some comments.

* [NOD-1579] Specifically check for StatusHeaderOnly in blockrelay.

* [NOD-1579] Simplify runIBDIfNotRunning.

* [NOD-1579] Don't run IBD if it is already running.

* [NOD-1579] Fix a comment.

* [NOD-1579] Rename mode to syncInfo.

* [NOD-1579] Simplify validateAndInsertBlock.

* [NOD-1579] Fix bad SyncStateSynced condition.

* [NOD-1579] Implement validateAgainstSyncStateAndResolveInsertMode.

* [NOD-1579] Use insertModeHeader.

* [NOD-1579] Add logs to TrySetIBDRunning and UnsetIBDRunning.

* [NOD-1579] Implement and use dequeueIncomingMessageAndSkipInvs.

* [NOD-1579] Fix a log.

* [NOD-1579] Fix a bug in createBlockLocator.

* [NOD-1579] Rename a variable.

* [NOD-1579] Fix a slew of bugs in missingBlockBodyHashes and selectedChildIterator.

* [NOD-1579] Fix bad chunk size in syncMissingBlockBodies.

* [NOD-1579] Remove maxOrphanBlueScoreDiff.

* [NOD-1579] Fix merge errors.

* [NOD-1579] Remove a debug log.

* [NOD-1579] Add logs.

* [NOD-1579] Make various go quality tools happy.

* [NOD-1579] Fix a typo in a variable name.

* [NOD-1579] Fix full blocks over header-only blocks not failing the missing-parents validation.

* [NOD-1579] Add an error log about a condition that should never happen.

* [NOD-1579] Check all antiPast hashes instead of just the lowHash's anticone to filter for header-only blocks.

* [NOD-1579] Remove the nil stuff from GetBlockLocator.

* [NOD-1579] Remove superfluous condition in handleRelayInvsFlow.start().

* [NOD-1579] Return a boolean from requestBlock instead of comparing to nil.

* [NOD-1579] Fix a bad log.Debugf.

* [NOD-1579] Remove a redundant check.

* [NOD-1579] Change an info log to a warning log.

* [NOD-1579] Move OnNewBlock out of relayBlock.

* [NOD-1579] Remove redundant exists check from runIBDIfNotRunning.

* [NOD-1579] Fix bad call to OnNewBlock.

* [NOD-1579] Remove an impossible check.

* [NOD-1579] Added a log.

* [NOD-1579] Rename insertModeBlockWithoutUpdatingVirtual to insertModeBlockBody.

* [NOD-1579] Add a check for duplicate headers.

* [NOD-1579] Added a comment.

* [NOD-1579] Tighten a stop condition.

* [NOD-1579] Simplify a log.

* [NOD-1579] Clarify a log.

* [NOD-1579] Move a log.
2020-12-06 16:23:56 +02:00
Svarog
4886425caf
[NOD-1589] Re-enable DisableDifficultyAdjustment (#1182)
* [NOD-1589] Re-enable DisableDifficultyAdjustment

* [NOD-1589] Remove simnet from TestDifficulty

* [NOD-1589] Update comment
2020-12-06 16:02:48 +02:00
Elichai Turkel
c3902ed7a8
Replace blue score with blue work in ghostdag (#1172)
* Replace blueScore with blueWork in ghostDAG SelectedParent selection

* Add blueWork to protopuf ghostdag data

* Auto generate protobuf go code

* Serialize/Deserialize blueWork when converting to protobuf

* pass block header store to ghostdagmanager

* Convert tal's ghostdag2 implementation to blueWork

* Change finality test to check the blueWork instead of blueScore

* Update ghostdag_test to pass blockHeaderStore to ghostdag, and test all networks genesis headers

* Add sanity blueWork check to ghostdag_test
2020-12-06 14:45:21 +02:00
Svarog
33eaf9edac
[NOD-1548] Re-add test difficulty + Make GHOSTDAGData immutable + don't clone in store (#1178)
* [NOD-1548] Readd TestDifficulty

* [NOD-1548] Make GHOSTDAGData immutable + don't clone in store

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2020-12-06 12:35:14 +02:00
Ori Newman
32a04d1811
Allow to configure consensus (closes #1067)
* Allow to configure consensus with a JSON file

* Define everywhere maxBlockParents as KType

* Move consensus default to consensus_defaults.go
2020-12-03 18:30:01 +02:00
talelbaz
f407c44a8d
[Issue - #1126] - Checking pruning point violation - pruning point in the past. (#1160)
* [NOD-1126]
1. Change function name in BlockValidator interface from: "ValidateProofOfWorkAndDifficulty" to "ValidatePruningPointViolationAndProofOfWorkAndDifficulty".
2. Add to the blockValidator struct the pruningManager (also added to the function "New" Respectively).
3. Added new function "checkPruningPointViolation" of blockValidator type.
4. Add new internal check - "checkPruningPointViolation", on the function "ValidateProofOfWorkAndDifficulty".(The third check).
5. Add new error rule - "ErrPruningPointViolation".

* [Issue-1126]
1. Remove the function "PruningPoint" from PruningManager interface.
2. Changes in blockValidator struct - remove pruningManager, and adding pruningStore.
3. Reads for "pruningPoint" function from pruningStore instead of pruningManager (because of note 1 above) in the functions: * "checkPruningPointViolation" of type blockValidator.
             * "FindNextPruningPoint" of type pruningManager.

* [Issue-1126]
1. Add missing error handling.

* [Issue-1126] Changes in function "checkPruningPointViolation": If header = genesis, stop checking and return nil.

* [Issue-1126] In function "checkPruningPointViolation" - change from a for loop to the "IsAncestorOfAny" function.

* [#1126] "FindNextPruningPoint" - save the pruning point in case the point is the genesis and change code internal order.

* [#1126] "FindNextPruningPoint" - cosmetics change.

* [#1126] "FindNextPruningPoint" - remove "return nil" when there is no pruning point on the if expression.

Co-authored-by: tal <tal@daglabs.com>
2020-11-30 09:57:15 +02:00
Svarog
048caebda3
[NOD-1551] Add SigCache to TransactionValidator + Option to manipulate it in TestConsensus (#1159)
* [NOD-1551] Add SigCache

* [NOD-1551] Add option to edit SigCache in TestConsensus

* [NOD-1551] Fix comments and make SetSigCache pointer-receiver
2020-11-29 10:18:00 +02:00
Ori Newman
0e91b44fc6
[NOD-1577] Change cache size to 200 (#1156) 2020-11-26 17:11:49 +02:00
Svarog
f7fa823f17
[NOD-1551] Requirements for performance tests (#1154)
* [NOD-1551] Add NewTestConsensusWithDataDir to factory

* [NOD-1551] Cache transaction ID

* [NOD-1551] Should return err if err != nil

* [NOD-1551] BuildBlockWithParents returns the blocks pastUTXOData

* [NOD-1551] Set BlockCoinbaseMaturity to 0 in TestDoubleSpends

* [NOD-1551] Fix comments

* --amend

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2020-11-26 12:12:01 +02:00
stasatdaglabs
0fa13357c3
[NOD-1566] Add caching to all stores (#1152)
* [NOD-1566] Add a dependency to golang-lru.

* [NOD-1566] Add caching to blockstore.go.

* [NOD-1566] Add LRUCache to all store objects and initialize them.

* [NOD-1566] Add caching to acceptanceDataStore.

* [NOD-1566] Add caching to blockHeaderStore.

* [NOD-1566] Implement a simpler LRU cache.

* [NOD-1566] Use the simpler cache implementation everywhere.

* [NOD-1566] Remove dependency in golang-lru.

* [NOD-1566] Fix object reuse issues in store Get functions.

* [NOD-1566] Add caching to blockRelationStore.

* [NOD-1566] Add caching to blockStatusStore.

* [NOD-1566] Add caching to ghostdagDataStore.

* [NOD-1566] Add caching to multisetStore.

* [NOD-1566] Add caching to reachabilityDataStore.

* [NOD-1566] Add caching to utxoDiffStore.

* [NOD-1566] Add caching to reachabilityReindexRoot.

* [NOD-1566] Add caching to pruningStore.

* [NOD-1566] Add caching to headerTipsStore.

* [NOD-1566] Add caching to consensusStateStore.

* [NOD-1566] Add comments explaining why we don't discard staging at the normal location in consensusStateStore.

* [NOD-1566] Make go vet happy.

* [NOD-1566] Fix merge errors.

* [NOD-1566] Add a missing break statement.

* [NOD-1566] Run go mod tidy.

* [NOD-1566] Remove serializedUTXOSetCache.
2020-11-25 13:41:13 +02:00
stasatdaglabs
bb244706ea
[NOD-1543] Optimize the performance of Count() in BlockHeaderStore and BlockStore (#1109)
* [NOD-1543] Optimize Count() in BlockHeaderStore.

* [NOD-1543] Optimize Count() in BlockStore.

* [NOD-1543] Fix commitCount.

* [NOD-1543] Explicitly initialize count to 0.
2020-11-18 16:35:32 +02:00
Ori Newman
8500acd86b
[NOD-1548] Remove PoW check from tests (#1105)
* [NOD-1548] Add TestDifficulty and remove PoW check from tests

* [NOD-1548] Add TestSkipProofOfWork

* [NOD-1548] Remove TestDifficulty
2020-11-18 10:27:29 +02:00
stasatdaglabs
48d8137604 [NOD-1538] Implement GetBlockCount. 2020-11-17 16:00:16 +02:00
Mike Zak
a1fa17d872 [NOD-1532] Add DiscardAllStores to TestConsensus 2020-11-17 16:00:16 +02:00
Mike Zak
aeded07815 [NOD-1532] Make BuildBlockWithParents resolve the status of the new block's selectedParent 2020-11-17 16:00:16 +02:00
Ori Newman
fc5e39f6cc [NOD-1535] fix reachability test (#1061)
* Revert "[NOD-1500] Delete integration tests"

This reverts commit fcb57a206690a884fa6afb69d5d493282954a8bf.

* [NOD-1518] hashserialization -> consenusserialization

* [NOD-1518] Fix add genesis to virtual

* [NOD-1518] Fix a bug in SerializeCoinbasePayload.

* [NOD-1518] Fix a loop error and make pastMedianTime behave correctly everywhere on genesis.

* [NOD-1518] Fix another bug and an infinite loop.

* [NOD-1518] Fix uninitialized slice.

* [NOD-1518] Fix bad should-commit checks and another infinite loop.

* [NOD-1518] Fix nil serialization.

* [NOD-1518] Rename blockHash to currentBlockHash.

* [NOD-1518] Move the check whether stagedVirtualUTXOSet != nil to the top of commitVirtualUTXODiff.

* [NOD-1518] Simplify utxoDiffStore.Commit.

* [NOD-1518] Unextract resolveBlockStatusAndCheckFinality.

* [NOD-1518] Move no-transactions logic into CalculateIDMerkleRoot.

* [NOD-1518] Remove redundant is-staged check.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Don't write anything if utxoDiffChild is nil.

* [NOD-1518] Stage virtualAcceptanceData and virtualMultiset.

* [NOD-1518] Fix bugs in getBlockTemplate and submitBlock.

* [NOD-1518] Fix bad validation order in validateHeaderInContext.

* [NOD-1518] Fix bug in Next().

* [NOD-1518] Fix nil dereference of subnetworks in AddressCache.

* [NOD-1518] Fix multisetStore.Get returning a pointer to a multiset that is changed in place.

* [NOD-1518] Break on genesis in countSubtrees.

* [NOD-1518] Fix createBlockLocator.

* [NOD-1518] Fix MsgTxToDomainTransaction.

* [NOD-1518] Set MaxTxVersion to 1.

* [NOD-1518] Fix missing error handling, bug in MsgTxToDomainTransaction, and bad subnetwork equality check.

* [NOD-1518] Fix bug in hasUTXOByOutpointFromStagedVirtualUTXODiff.

* [NOD-1518] Remove irrelevant comments.

* [NOD-1518] Generate transactions with sufficient fee in tx_relay_test.

* [NOD-1518] Fix broken RPC handlers.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Fix bad exists check in restorePastUTXO and missing genesis check in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Add a comment.

* [NOD-1518] Use a regular mutex instead of a read-write mutex in consensus to avoid dealing with sneaky not-actually-read functions.

* [NOD-1518] Fix a deadlock in GetVirtualSelectedParent.

* [NOD-1518] Fix missing handler registration for CmdHeader.

* [NOD-1518] Fix processHeader calling OnNewBlock and LogBlock. Also fix conversion errors in IBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad Command() in MsgIBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad SyncStateMissingUTXOSet logic in resolveSyncState.

* [NOD-1518] Rename mode to syncState.

* [NOD-1518] Fix headers-only blocks coming in after the consensus thinks it's synced.

* [NOD-1518] Fix selectedChildIterator.Next not ignoring virtual, infinite loop in HashSet.Length().

* [NOD-1518] Fix not-properly wrapped IBD blocks.

* [NOD-1518] Fix bad conversion in RequestIBDBlocks.

* [NOD-1518] Fix bad string for CmdRequestHeaders.

* [NOD-1518] Fix bad string for CmdDoneHeaders.

* [NOD-1518] Fix bad Command() for MsgIBDRootNotFound.

* [NOD-1518] Fix bad areHeaderTipsSyncedMaxTimeDifference value.

* [NOD-1518] Add missing string for CmdRequestIBDBlocks.

* [NOD-1518] Fix bad check for SyncStateMissingBlockBodies.

* [NOD-1518] Fix bad timeout durations in tests.

* [NOD-1518] Fix IBD blocks not calling OnNewBlock.

* [NOD-1518] Change when IBD finishes.

* [NOD-1518] Properly clone utxoDiffChild.

* [NOD-1535] Fix reachability tests

* [NOD-1518] Fix merge errors.

* [NOD-1518] Move call to LogBlock to into OnNewBlock.

* [NOD-1518] Return "not implemented" in unimplemented RPC handlers.

* [NOD-1518] Extract cloning of hashes to a method over DomainHash.

* [NOD-1518] Use isHeaderOnlyBlock.

* [NOD-1518] Use constants.TransactionVersion.

* [NOD-1518] Break immediately if we reached the virtual in SelectedChildIterator.

* [NOD-1518] Don't stage nil utxoDiffChild.

* [NOD-1518] Properly check the genesis hash in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Explain why we break on current == nil in countSubtrees.

* [NOD-1518] Add a comment explaining why we check against StatusValid in resolveSyncState.

* [NOD-1535] Add external reachability tests

* [NOD-1535] Fix reachability tests and fix related bugs

* [NOD-1535] Add setters fox reindex slack and window

* [NOD-1535] Remove redundant line

* [NOD-1535] Add comment

* [NOD-1535] Fix comments

* [NOD-1535] Rename DBReader->DatabaseContext

* [NOD-1535] Check that reindex root is changed

* [NOD-1535] Fix calculateNewTips

Co-authored-by: Mike Zak <feanorr@gmail.com>
Co-authored-by: stasatdaglabs <stas@daglabs.com>
2020-11-17 16:00:16 +02:00
Svarog
8ccf381fc7 [NOD-1532] csm unit tests (#1059)
* Revert "[NOD-1500] Delete integration tests"

This reverts commit fcb57a206690a884fa6afb69d5d493282954a8bf.

* [NOD-1518] hashserialization -> consenusserialization

* [NOD-1518] Fix add genesis to virtual

* [NOD-1518] Fix a bug in SerializeCoinbasePayload.

* [NOD-1518] Fix a loop error and make pastMedianTime behave correctly everywhere on genesis.

* [NOD-1518] Fix another bug and an infinite loop.

* [NOD-1518] Fix uninitialized slice.

* [NOD-1518] Fix bad should-commit checks and another infinite loop.

* [NOD-1518] Fix nil serialization.

* [NOD-1518] Rename blockHash to currentBlockHash.

* [NOD-1518] Move the check whether stagedVirtualUTXOSet != nil to the top of commitVirtualUTXODiff.

* [NOD-1518] Simplify utxoDiffStore.Commit.

* [NOD-1518] Unextract resolveBlockStatusAndCheckFinality.

* [NOD-1518] Move no-transactions logic into CalculateIDMerkleRoot.

* [NOD-1518] Remove redundant is-staged check.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Don't write anything if utxoDiffChild is nil.

* [NOD-1518] Stage virtualAcceptanceData and virtualMultiset.

* [NOD-1518] Fix bugs in getBlockTemplate and submitBlock.

* [NOD-1518] Fix bad validation order in validateHeaderInContext.

* [NOD-1518] Fix bug in Next().

* [NOD-1518] Fix nil dereference of subnetworks in AddressCache.

* [NOD-1518] Fix multisetStore.Get returning a pointer to a multiset that is changed in place.

* [NOD-1518] Break on genesis in countSubtrees.

* [NOD-1518] Fix createBlockLocator.

* [NOD-1518] Fix MsgTxToDomainTransaction.

* [NOD-1518] Set MaxTxVersion to 1.

* [NOD-1518] Fix missing error handling, bug in MsgTxToDomainTransaction, and bad subnetwork equality check.

* [NOD-1518] Fix bug in hasUTXOByOutpointFromStagedVirtualUTXODiff.

* [NOD-1518] Remove irrelevant comments.

* [NOD-1518] Generate transactions with sufficient fee in tx_relay_test.

* [NOD-1518] Fix broken RPC handlers.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Fix bad exists check in restorePastUTXO and missing genesis check in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Add a comment.

* [NOD-1518] Use a regular mutex instead of a read-write mutex in consensus to avoid dealing with sneaky not-actually-read functions.

* [NOD-1518] Fix a deadlock in GetVirtualSelectedParent.

* [NOD-1518] Fix missing handler registration for CmdHeader.

* [NOD-1518] Fix processHeader calling OnNewBlock and LogBlock. Also fix conversion errors in IBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad Command() in MsgIBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad SyncStateMissingUTXOSet logic in resolveSyncState.

* [NOD-1518] Rename mode to syncState.

* [NOD-1518] Fix headers-only blocks coming in after the consensus thinks it's synced.

* [NOD-1518] Fix selectedChildIterator.Next not ignoring virtual, infinite loop in HashSet.Length().

* [NOD-1518] Fix not-properly wrapped IBD blocks.

* [NOD-1532] Add TestMultiset

* [NOD-1518] Fix bad conversion in RequestIBDBlocks.

* [NOD-1518] Fix bad string for CmdRequestHeaders.

* [NOD-1518] Fix bad string for CmdDoneHeaders.

* [NOD-1518] Fix bad Command() for MsgIBDRootNotFound.

* [NOD-1532] Add TestPastUTXOMultiset

* [NOD-1518] Fix bad areHeaderTipsSyncedMaxTimeDifference value.

* [NOD-1532] Added TestDoubleSpends

* [NOD-1518] Add missing string for CmdRequestIBDBlocks.

* [NOD-1518] Fix bad check for SyncStateMissingBlockBodies.

* [NOD-1518] Fix bad timeout durations in tests.

* [NOD-1518] Fix IBD blocks not calling OnNewBlock.

* [NOD-1518] Change when IBD finishes.

* [NOD-1518] Properly clone utxoDiffChild.

* [NOD-1532] Update hashes of blocks

* [NOD-1532] Fix genesis blocks and a few more bugs

* [NOD-1532] Bugfix: incorrect key passed to dbTx.Put

* [NOD-1532] Make sure there's no nil payloads

* [NOD-1532] Fix AddBlockToVirtual

* [NOD-1532] Update tips and virtualDiffParents properly

* [NOD-1532] Allow nil payload

* [NOD-1532] Check for actual error and not just some RuleError

* [NOD-1532] Get rid of SimpleCoinbaseData and make OpTrueScript P2SH

* [NOD-1532] If coinbaseData is nil - fill in with generic coinbaseData

Co-authored-by: Ori Newman <orinewman1@gmail.com>
Co-authored-by: stasatdaglabs <stas@daglabs.com>
2020-11-17 16:00:16 +02:00
stasatdaglabs
eef5e3768c
[NOD-1518] Fix genesis block insertion and integration tests (#1013)
* Revert "[NOD-1500] Delete integration tests"

This reverts commit fcb57a206690a884fa6afb69d5d493282954a8bf.

* [NOD-1518] hashserialization -> consenusserialization

* [NOD-1518] Fix add genesis to virtual

* [NOD-1518] Fix a bug in SerializeCoinbasePayload.

* [NOD-1518] Fix a loop error and make pastMedianTime behave correctly everywhere on genesis.

* [NOD-1518] Fix another bug and an infinite loop.

* [NOD-1518] Fix uninitialized slice.

* [NOD-1518] Fix bad should-commit checks and another infinite loop.

* [NOD-1518] Fix nil serialization.

* [NOD-1518] Rename blockHash to currentBlockHash.

* [NOD-1518] Move the check whether stagedVirtualUTXOSet != nil to the top of commitVirtualUTXODiff.

* [NOD-1518] Simplify utxoDiffStore.Commit.

* [NOD-1518] Unextract resolveBlockStatusAndCheckFinality.

* [NOD-1518] Move no-transactions logic into CalculateIDMerkleRoot.

* [NOD-1518] Remove redundant is-staged check.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Don't write anything if utxoDiffChild is nil.

* [NOD-1518] Stage virtualAcceptanceData and virtualMultiset.

* [NOD-1518] Fix bugs in getBlockTemplate and submitBlock.

* [NOD-1518] Fix bad validation order in validateHeaderInContext.

* [NOD-1518] Fix bug in Next().

* [NOD-1518] Fix nil dereference of subnetworks in AddressCache.

* [NOD-1518] Fix multisetStore.Get returning a pointer to a multiset that is changed in place.

* [NOD-1518] Break on genesis in countSubtrees.

* [NOD-1518] Fix createBlockLocator.

* [NOD-1518] Fix MsgTxToDomainTransaction.

* [NOD-1518] Set MaxTxVersion to 1.

* [NOD-1518] Fix missing error handling, bug in MsgTxToDomainTransaction, and bad subnetwork equality check.

* [NOD-1518] Fix bug in hasUTXOByOutpointFromStagedVirtualUTXODiff.

* [NOD-1518] Remove irrelevant comments.

* [NOD-1518] Generate transactions with sufficient fee in tx_relay_test.

* [NOD-1518] Fix broken RPC handlers.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Fix bad exists check in restorePastUTXO and missing genesis check in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Add a comment.

* [NOD-1518] Use a regular mutex instead of a read-write mutex in consensus to avoid dealing with sneaky not-actually-read functions.

* [NOD-1518] Fix a deadlock in GetVirtualSelectedParent.

* [NOD-1518] Fix missing handler registration for CmdHeader.

* [NOD-1518] Fix processHeader calling OnNewBlock and LogBlock. Also fix conversion errors in IBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad Command() in MsgIBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad SyncStateMissingUTXOSet logic in resolveSyncState.

* [NOD-1518] Rename mode to syncState.

* [NOD-1518] Fix headers-only blocks coming in after the consensus thinks it's synced.

* [NOD-1518] Fix selectedChildIterator.Next not ignoring virtual, infinite loop in HashSet.Length().

* [NOD-1518] Fix not-properly wrapped IBD blocks.

* [NOD-1518] Fix bad conversion in RequestIBDBlocks.

* [NOD-1518] Fix bad string for CmdRequestHeaders.

* [NOD-1518] Fix bad string for CmdDoneHeaders.

* [NOD-1518] Fix bad Command() for MsgIBDRootNotFound.

* [NOD-1518] Fix bad areHeaderTipsSyncedMaxTimeDifference value.

* [NOD-1518] Add missing string for CmdRequestIBDBlocks.

* [NOD-1518] Fix bad check for SyncStateMissingBlockBodies.

* [NOD-1518] Fix bad timeout durations in tests.

* [NOD-1518] Fix IBD blocks not calling OnNewBlock.

* [NOD-1518] Change when IBD finishes.

* [NOD-1518] Properly clone utxoDiffChild.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Move call to LogBlock to into OnNewBlock.

* [NOD-1518] Return "not implemented" in unimplemented RPC handlers.

* [NOD-1518] Extract cloning of hashes to a method over DomainHash.

* [NOD-1518] Use isHeaderOnlyBlock.

* [NOD-1518] Use constants.TransactionVersion.

* [NOD-1518] Break immediately if we reached the virtual in SelectedChildIterator.

* [NOD-1518] Don't stage nil utxoDiffChild.

* [NOD-1518] Properly check the genesis hash in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Explain why we break on current == nil in countSubtrees.

* [NOD-1518] Add a comment explaining why we check against StatusValid in resolveSyncState.

Co-authored-by: Mike Zak <feanorr@gmail.com>
Co-authored-by: Ori Newman <orinewman1@gmail.com>
2020-11-12 15:19:39 +02:00
Svarog
135ffbd4f2
[NOD-1529] Add getters + AddBlock to TestConsensus (#1025)
* [NOD-1529] Add all stores and processes to consensus, and add access to TestConsensus

* [NOD-1529] Move the getters of TestConsensus to separate file

* [NOD-1529] Add AddBlock to TestConsensus

* [NOD-1529] Update NewTestConsensus to be more all-encompassing

* [NOD-1529] Remove test directory in teardown

* [NOD-1529] Add ForAllNets function

* [NOD-1529] Add comment
2020-11-11 12:31:13 +02:00
Ori Newman
32da4440ba
[NOD-1495] Disallow non native transactions (#988)
* [NOD-1495] Disallow non native transactions

* [NOD-1495] Use deserializeUTXOSetBytes

* [NOD-1495] Delete checkNoNonNativeTransactions

* [NOD-1495] Invert condition in checkTransactionPayload

Co-authored-by: Mike Zak <feanorr@gmail.com>
2020-11-09 17:15:16 +02:00
Svarog
e7a61c7edf
[NOD-1524] Add lock to consensus (#1014) 2020-11-09 16:21:09 +02:00
Svarog
6db337c8c5
[NOD-1519] Add TestAPI for BuildBlockWithParents (#1011)
* [NOD-1519] Separate BlockBuilder and BlockProcessor

* [NOD-1519] Wire blockBuilder properly + implement buildBlockWithParents

* [NOD-1519] Added testapi package, TestConsensus interface, and TestConsensus factory

* [NOD-1519] Add comments

* [NOD-1519] Separate TestBlockBuilder out of BlockBuilder

* [NOD-1519] TestBlockBuilder should also implement BlockBuilder

* [NOD-1519] Add NewTestConsensus to factory interface
2020-11-09 12:02:42 +02:00
Ori Newman
9a344152aa
[NOD-1517] Properly initialize consensus with Genesis block (#1009)
* [NOD-1517] Properly initialize consensus with Genesis block

* [NOD-1517] Remove redundant AddHeaderTip

* [NOD-1517] Don't return nil from dbHash<->DomainHash converters

* [NOD-1517] Use pointer receivers

* [NOD-1517] Use domain block in dagParams

* [NOD-1517] Remove boolean from SelectedTip

* [NOD-1517] Rename hasHeader to isHeadersOnlyBlock

* [NOD-1517] Add comment

* [NOD-1517] Change genesis version

* [NOD-1517] Rename TestNewFactory->TestNewConsensus
2020-11-08 15:17:20 +02:00
stasatdaglabs
1c710daf98
[NOD-1501] Finish implementing GetSyncInfo (#1002)
* [NOD-1501] Rename IsBlockInHeaderPruningPointFutureAndVirtualPast to IsBlockInHeaderPruningPointFuture.

* [NOD-1501] Create syncinfo.go.

* [NOD-1501] Implement resolveSyncState.

* [NOD-1501] Fix ChooseSelectedParent.

* [NOD-1501] Fix merge errors.

* [NOD-1501] Finish implementing getSyncState.

* [NOD-1501] Fix bad equality check.

* [NOD-1501] Fix merge errors.

* [NOD-1501] Pass targetTimePerBlock as int64 milliseconds.
2020-11-05 11:50:59 +02:00
Ori Newman
5566aaf95a
[NOD-1512] Implement utxo deserialization (#1003)
* [NOD-1512] Implement UTXO set deserialization

* [NOD-1512] Remove redundant file

* [NOD-1512] Don't use big endian for serialization

* [NOD-1512] Use Read/Write element

* [NOD-1512] Unexport ReadElement

* [NOD-1512] Fix StageVirtualUTXOSet

* [NOD-1512] Get rid of dagParams in consensusStateManager

* [NOD-1512] Get rid of dagParams in consensusStateManager
2020-11-05 10:59:49 +02:00
Ori Newman
52c73d3a08
[NOD-1511] Implement missingBlockBodyHashes (#1000)
* [NOD-1511] Implement missingBlockBodyHashes

* [NOD-1511] Rename selectedparentiterator.go to blockiterator.go

* [NOD-1511] Fix condition

* [NOD-1511] Simplify missingBlocks logic
2020-11-05 10:11:42 +02:00
Ori Newman
baf8d25656
[NOD-1510] Update reachability reindex root based on header tips selected tip (#1001) 2020-11-04 16:55:59 +02:00
Elichai Turkel
ce95c6dc9d
[NOD-1464] difficulty refactoring (#986)
* Refactor the Difficulty adjastment to the new design

* Add the necessary things to the factory for the DAA constructor

* Add missing dagParams to difficultymanager constructor

* Use DAGTraversal for blueBlockWindow, and don't store PowMax compactBits
2020-11-04 11:35:29 +02:00
Elichai Turkel
6a46cb2be6
[NOD-1503] Pruning Manager (#994)
* Update pruningmanager interface

* Add a ProtoUTXOSet to hashserialization

* Update miningmanager with all the necessary stores and managers

* Implement mining manager

* Prune P.AC not in V.Past

* PruningManager fix all review comments
2020-11-04 10:29:45 +02:00