16 Commits

Author SHA1 Message Date
Ori Newman
d9bc94a2a8
Replace header finality point with pruning point and enforce finality rules on IBD with headers proof (#1823)
* Replace header finality point with pruning point

* Fix TestTransactionAcceptance

* Fix pruning candidate

* Store all past pruning points

* Pass pruning points on IBD

* Add blue score to block header

* Simplify ArePruningPointsInValidChain

* Fix static check errors

* Fix genesis

* Renames and text fixing

* Use ExpectedHeaderPruningPoint in block builder

* Fix TestCheckPruningPointViolation
2021-08-31 08:01:48 +03:00
Elichai Turkel
3c3ad1425d
Make moving the pruning point faster (#1660)
* Add oldPruningPoint to pruningStore

* Make the pruning store work with utxo diff and return an iterator over pruning point utxoset

* Redesign pruning point utxo storage by creating a diff and modifying the old pruning utxo set

* Fix review comments

* Rename updatePruningPointUTXOSet
2021-04-11 11:17:13 +03:00
Svarog
f1451406f7
Add support for multiple staging areas (#1633)
* Add StagingArea struct

* Implemented staging areas in blockStore

* Move blockStagingShard to separate folder

* Apply staging shard to acceptanceDataStore

* Update blockHeaderStore with StagingArea

* Add StagingArea to BlockRelationStore

* Add StagingArea to blockStatusStore

* Add StagingArea to consensusStateStore

* Add StagingArea to daaBlocksStore

* Add StagingArea to finalityStore

* Add StagingArea to ghostdagDataStore

* Add StagingArea to headersSelectedChainStore and headersSelectedTipStore

* Add StagingArea to multisetStore

* Add StagingArea to pruningStore

* Add StagingArea to reachabilityDataStore

* Add StagingArea to utxoDiffStore

* Fix forgotten compilation error

* Update reachability manager and some more things with StagingArea

* Add StagingArea to dagTopologyManager, and some more

* Add StagingArea to GHOSTDAGManager, and some more

* Add StagingArea to difficultyManager, and some more

* Add StagingArea to dagTraversalManager, and some more

* Add StagingArea to headerTipsManager, and some more

* Add StagingArea to constnsusStateManager, pastMedianTimeManager

* Add StagingArea to transactionValidator

* Add StagingArea to finalityManager

* Add StagingArea to mergeDepthManager

* Add StagingArea to pruningManager

* Add StagingArea to rest of ValidateAndInsertBlock

* Add StagingArea to blockValidator

* Add StagingArea to coinbaseManager

* Add StagingArea to syncManager

* Add StagingArea to blockBuilder

* Update consensus with StagingArea

* Add StagingArea to ghostdag2

* Fix remaining compilation errors

* Update names of stagingShards

* Fix forgotten stagingArea passing

* Mark stagingShard.isCommited = true once commited

* Move isStaged to stagingShard, so that it's available without going through store

* Make blockHeaderStore count be avilable from stagingShard

* Fix remaining forgotten stagingArea passing

* commitAllChanges should call dbTx.Commit in the end

* Fix all tests tests in blockValidator

* Fix all tests in consensusStateManager and some more

* Fix all tests in pruningManager

* Add many missing stagingAreas in tests

* Fix many tests

* Fix most of all other tests

* Fix ghostdag_test.go

* Add comment to StagingArea

* Make list of StagingShards an array

* Add comment to StagingShardID

* Make sure all staging shards are pointer-receiver

* Undo bucket rename in block_store

* Typo: isCommited -> isCommitted

* Add comment explaining why stagingArea.shards is an array
2021-03-29 10:34:11 +03:00
Ori Newman
2adb4f5d0f
Fix UTXO index (#1548)
* Add VirtualUTXODiff and VirtualParents to block insertion result

* Add GetVirtualUTXOs

* Add OnPruningPointUTXOSetOverrideHandler

* Add recovery to UTXO index

* Add UTXO set override notification

* Fix compilation error

* Fix iterators in UTXO index and fix TestUTXOIndex

* Change Dialing to DEBUG

* Change LogBlock location

* Rename StopNotify to StopNotifying

* Add sanity check

* Add comment

* Remove receiver from serialization functions

Co-authored-by: Elichai Turkel <elichai.turkel@gmail.com>
2021-02-23 16:51:51 +02:00
stasatdaglabs
819ec9f2a7
Pass fromOutpoint into GetPruningPointUTXOs instead of offset, so it could Seek over the cursor (#1447)
* Implement TestGetPruningPointUTXOs.

* Fix a bad error message.

* Fix TestGetPruningPointUTXOs for testnet.

* Make sure all the UTXOs are returned in TestGetPruningPointUTXOs.

* Implement BenchmarkGetPruningPointUTXOs.

* Pass fromOutpoint into GetPruningPointUTXOs instead of offset, so it could Seek over the cursor.

* Fix weird benchmark timer calls.

* Remove unnecessary collection of outpointAndUTXOEntryPairs from BenchmarkGetPruningPointUTXOs.

* Fix a comment.
2021-01-25 13:38:59 +02:00
stasatdaglabs
ca04c049ab
When the pruning point moves, update its UTXO set outside of a database transaction (#1444)
* Remove pruningPointUTXOSetStaging and implement UpdatePruningPointUTXOSet.

* Implement StageStartSavingNewPruningPointUTXOSet, HadStartedSavingNewPruningPointUTXOSet, and FinishSavingNewPruningPointUTXOSet.

* Fix a bad return.

* Implement UpdatePruningPointUTXOSetIfRequired.

* Call UpdatePruningPointUTXOSetIfRequired on consensus creation.

* Rename savingNewPruningPointUTXOSetKey to updatingPruningPointUTXOSet.

* Add a log.

* Add calls to runtime.GC() at its start and end.

* Rename a variable.

* Replace calls to runtime.GC to calls to LogMemoryStats.

* Wrap the contents of LogMemoryStats in a log closure.
2021-01-24 14:48:11 +02:00
stasatdaglabs
756f40c59a
Sync pruning point UTXO sets incrementally instead of all at once (#1431)
* Replaced the content of MsgIBDRootUTXOSetChunk with pairs of outpoint-utxo entry pairs.

* Rename utxoIter to utxoIterator.

* Add a big stinky TODO on an assert.

* Replace pruningStore staging with a UTXO set iterator.

* Reimplement receiveAndInsertIBDRootUTXOSet.

* Extract OutpointAndUTXOEntryPairsToDomainOutpointAndUTXOEntryPairs into domainconverters.go.

* Pass the outpoint and utxy entry pairs to the pruning store.

* Implement InsertCandidatePruningPointUTXOs.

* Implement ClearCandidatePruningPointUTXOs.

* Implement UpdateCandidatePruningPointMultiset.

* Use the candidate pruning point multiset in updatePruningPoint.

* Implement CandidatePruningPointUTXOIterator.

* Use the pruning point utxo set iterator for StageVirtualUTXOSet.

* Defer ClearCandidatePruningPointUTXOs.

* Implement OverwriteVirtualUTXOSet.

* Implement CommitCandidatePruningPointUTXOSet.

* Implement BeginOverwritingVirtualUTXOSet and FinishOverwritingVirtualUTXOSet.

* Implement overwriteVirtualUTXOSetAndCommitPruningPointUTXOSet.

* Rename ClearCandidatePruningPointUTXOs to ClearCandidatePruningPointData.

* Add missing methods to dbManager.

* Implement PruningPointUTXOs.

* Implement RecoverUTXOIfRequired.

* Delete the utxoserialization package.

* Fix compilation errors in TestValidateAndInsertPruningPoint.

* Switch order of operations in the if statements in PruningPointUTXOs so that Next() wouldn't be unnecessarily called.

* Fix missing pruning point utxo set staging and bad slice length.

* Fix no default multiset in InsertCandidatePruningPointUTXOs.

* Make go vet happy.

* Rename candidateXXX to importedXXX.

* Do some more renaming.

* Rename some more.

* Fix bad MsgIBDRootNotFound logic.

* Fix an error message.

* Simplify receiveIBDRootBlock.

* Fix error message in receiveAndInsertIBDRootUTXOSet.

* Do some more renaming.

* Fix merge errors.

* Fix a bug caused by calling iterator.First() unnecessarily.

* Remove databaseContext from stores and don't use a transaction in ClearXXX functions.

* Simplify receiveAndInsertIBDRootUTXOSet.

* Fix offset count in PruningPointUTXOs().

* Fix readOnlyUTXOIteratorWithDiff.First().

* Split handleRequestIBDRootUTXOSetAndBlockFlow into smaller methods.

* Rename IbdRootNotFound to UnexpectedPruningPoint.

* Rename requestIBDRootHash to requestPruningPointHash.

* Rename IBDRootHash to PruningPointHash.

* Rename RequestIBDRootUTXOSetAndBlock to RequestPruningPointUTXOSetAndBlock.

* Rename IBDRootUTXOSetChunk to PruningPointUTXOSetChunk.

* Rename RequestNextIBDRootUTXOSetChunk to RequestNextPruningPointUTXOSetChunk.

* Rename DoneIBDRootUTXOSetChunks to DonePruningPointUTXOSetChunks.

* Rename remaining references to IBD root.

* Fix an error message.

* Add a check for HadStartedImportingPruningPointUTXOSet in commitVirtualUTXODiff.

* Add a check for HadStartedImportingPruningPointUTXOSet in ImportPruningPointUTXOSetIntoVirtualUTXOSet.

* Move FinishImportingPruningPointUTXOSet closer to HadStartedImportingPruningPointUTXOSet.

* Remove reference to pruningStore in utxoSetIterator.

* Pointerify utxoSetIterator receivers.

* Fix bad insert in CommitImportedPruningPointUTXOSet.

* Rename commitImportedPruningPointUTXOSetAll to applyImportedPruningPointUTXOSet.

* Simplify PruningPointUTXOs.

* Add populateTransactionWithUTXOEntriesFromUTXOSet.

* Fix a TODO comment.

* Rename InsertImportedPruningPointUTXOs to AppendImportedPruningPointUTXOs.

* Extract handleRequestPruningPointUTXOSetAndBlockMessage to a separate method.

* Rename stuff in readOnlyUTXOIteratorWithDiff.First().

* Address toAddIterator in readOnlyUTXOIteratorWithDiff.First().

* Call First() before any full iteration on ReadOnlyUTXOSetIterator.

* Call First() before any full iteration on a database Cursor.

* Put StartImportingPruningPointUTXOSet inside the pruning point transaction.

* Make serializeOutpoint and serializeUTXOEntry free functions in pruningStore.

* Fix readOnlyUTXOIteratorWithDiff.First().

* Fix bad validations in importPruningPoint.

* Remove superfluous call to validateBlockTransactionsAgainstPastUTXO.
2021-01-21 17:24:52 +02:00
Ori Newman
729e3db145
Pruning calculation changes (#1250)
* 1) Calculate pruning point incrementally
2) Add IsValidPruningPoint to pruning manager and consensus
3) Use reachability children for selected child iterator

* Add request IBD root hash flow

* Fix UpdatePruningPointByVirtual and IsValidPruningPoint

* Regenerate messages.pb.go

* Make the pruning point the earliest chain block with finality interval higher than the previous pruning point

* Fix merge errors
2020-12-23 11:37:39 +02:00
Elichai Turkel
6a46cb2be6
[NOD-1503] Pruning Manager (#994)
* Update pruningmanager interface

* Add a ProtoUTXOSet to hashserialization

* Update miningmanager with all the necessary stores and managers

* Implement mining manager

* Prune P.AC not in V.Past

* PruningManager fix all review comments
2020-11-04 10:29:45 +02:00
Ori Newman
14fbe50636
[NOD-1493] Implement serialization in data stores (#983)
* [NOD-1493] Implement serialization in data stores

* [NOD-1493] Remove redundant functions

* [NOD-1493] Use bluesAnticoneSizesToDBBluesAnticoneSizes inside BlockGHOSTDAGDataToDBBlockGHOSTDAGData
2020-10-29 17:39:35 +02:00
Ori Newman
be56fb7e8b
[NOD-1488] Get rid of dbaccess (#973)
* [NOD-1488] Get rid of dbaccess

* [NOD-1488] Rename dbwrapper to dbmanager

* [NOD-1488] Create DBWriter interface

* [NOD-1488] Fix block header store

* [NOD-1488] Rename dbwrapper.go to dbmanager.go
2020-10-28 11:34:06 +02:00
stasatdaglabs
97b5b0b875
[NOD-1416] Implement BlockProcessor. (#969)
* [NOD-1416] Add entry/exit logs to all the functions.

* [NOD-1416] Build some scaffolding inside BlockProcessor.

* [NOD-1416] Implement selectParentsForNewBlock.

* [NOD-1416] Implement validateBlock.

* [NOD-1476] Fix merge errors.

* [NOD-1416] Move buildBlock and validateAndInsertBlock to separate files.

* [NOD-1416] Begin implementing buildBlock.

* [NOD-1416] Implement newBlockDifficulty.

* [NOD-1416] Add skeletons for the rest of the buildBlock functions.

* [NOD-1416] Implement newBlockUTXOCommitment.

* [NOD-1416] Implement newBlockAcceptedIDMerkleRoot.

* [NOD-1416] Implement newBlockHashMerkleRoot.

* [NOD-1416] Fix bad function call.

* [NOD-1416] Implement validateHeaderAndProofOfWork and validateBody.

* [NOD-1416] Use ValidateProofOfWorkAndDifficulty.

* [NOD-1416] Finish validateAndInsertBlock.

* [NOD-1416] Implement newBlockHashMerkleRoot.

* [NOD-1416] Implement newBlockAcceptedIDMerkleRoot.

* [NOD-1416] Fix a comment.

* [NOD-1416] Implement newBlockCoinbaseTransaction.

* [NOD-1416] Add VirtualBlockHash.

* [NOD-1416] Add ParentHashes and SelectedParent to VirtualData().

* [NOD-1416] Make go vet happy.

* [NOD-1416] Implement discardAllChanges.

* [NOD-1416] Implement commitAllChanges.

* [NOD-1416] Fix factory.

* [NOD-1416] Make go vet happy.

* [NOD-1416] Format factory.

* [NOD-1416] Pass transactionsWithCoinbase to buildHeader.

* [NOD-1416] Call VirtualData() from buildHeader.

* [NOD-1416] Fix a typo.

* [NOD-1416] Fix in-out-of-context/header-body confusion.

* [NOD-1416] Extract LogAndMeasureExecutionTime.

* [NOD-1416] Add a comment about LogAndMeasureExecutionTime.

* [NOD-1416] Simplify discardAllChanges and commitAllChanges.

* [NOD-1416] If in-context validations fail, discard all changes and store the block with StatusInvalid.

* [NOD-1416] Add a comment above Store.

* [NOD-1416] Use errors.As instead of errors.Is.
2020-10-27 17:24:15 +02:00
stasatdaglabs
45882343e6
[NOD-1475] Implement stage/discard/commit functionality for data structures (#962)
* [NOD-1475] Add Stage, Discard, and Commit methods to all stores.

* [NOD-1475] Simplify interfaces for processes.

* [NOD-1475] Fix GHOSTDAGManager.

* [NOD-1475] Simplify ChooseSelectedParent.

* [NOD-1475] Remove errors from Stage functions.

* [NOD-1475] Add IsStaged to all data structures.

* [NOD-1475] Remove isDisqualified from CalculateConsensusStateChanges.

* [NOD-1475] Add dependency from ConsensusStateManager to BlockStatusStore.

* [NOD-1475] Fix a comment.

* [NOD-1475] Add ReachabilityReindexRoot to reachabilityDataStore.

* [NOD-1475] Fix a comment.

* [NOD-1475] Rename IsStaged to IsAnythingStaged.
2020-10-21 12:37:22 +03:00
stasatdaglabs
a96a5fd2ef
[NOD-1462] Simplify consensus external API (#958)
* [NOD-1461] Change the external api interface to not having anything besides DomainTransactions and DomainBlocks.

* [NOD-1462] Move external api types to a separate package.

* [NOD-1462] Clarify which model we're using in miningmanager.

* [NOD-1462] Extract coinbase data to its own struct.

* [NOD-1462] Add a comment above CoinbaseData.

* [NOD-1462] Fix the comment above CoinbaseData.
2020-10-19 17:59:04 +03:00
stasatdaglabs
81a10e9f89
[NOD-1458] Make further design changes (#956)
* [NOD-1458] Rename RestoreUTXOSet to RestorePastUTXOSet.

* [NOD-1458] Make CalculateAcceptanceDataAndMultiset take BlockGHOSTDAGData and nothing else.

* [NOD-1458] Make ConsensusStateStore's Update take ConsensusStateChanges instead of just UTXODiff.

* [NOD-1458] Add Tips() to ConsensusStateStore.

* [NOD-1458] Make all implementation structs private.

* [NOD-1458] Remove BlockAtDepth and add highHash to ChainBlockAtBlueScore.

* [NOD-1458] Rename CalculateAcceptanceDataAndMultiset to CalculateAcceptanceDataAndUTXOMultiset.

* [NOD-1458] Add a dependency to GHOSTDAGManager from ConsensusStateManager.

* [NOD-1458] Add ChooseSelectedParent to GHOSTDAGManager.

* [NOD-1458] Add DifficultyManager.

* [NOD-1458] Add PastMedianTimeManager.

* [NOD-1458] Add Hash() to Multiset.

* [NOD-1458] Add a dependency to ghostdagManager from blockProcessor.

* [NOD-1458] Add errors to all interfaces that need them.

* [NOD-1458] Uppercasify types in comments.

* [NOD-1458] Fix a bad comment.

* [NOD-1458] Fix a comment.

* [NOD-1458] Rename ChainBlockAtBlueScore to HighestChainBlockBelowBlueScore.

* [NOD-1458] Replace BlockAndTransactionValidator with an anonymous interface.
2020-10-18 12:34:00 +03:00
stasatdaglabs
4f36accd81
[NOD-1413] Make some additional interface changes (#954)
* [NOD-1413] Remove /cmd/addblock

* [NOD-1413] Define and implement TransactionValidator.

* [NOD-1413] Make changes to ConsensusStateManager's interface.

* [NOD-1413] Make changes to PruningManager's interface.

* [NOD-1413] Make changes to DAGTraversalManager's interface.

* [NOD-1413] Make changes to MultisetStore's interface.

* [NOD-1413] Make changes to UTXODiffStore's interface.

* [NOD-1413] Make changes to UTXODiffStore's interface harder.

* [NOD-1413] Make changes to AcceptanceDataStore's interface harder.

* [NOD-1413] Make changes to PruningStore's interface.

* [NOD-1413] Delete BlockIndex.

* [NOD-1413] Add FeeDataStore.

* [NOD-1413] Update BlockMessageStore's interface.

* [NOD-1413] Fix interface violations.

* [NOD-1413] Add FeeDataStore to BlockProcessor.

* [NOD-1413] Make go vet happy.

* [NOD-1413] Add missing fields to ConsensusStateChanges.

* [NOD-1413] Add another missing field to ConsensusStateChanges.

* [NOD-1413] Add a reference to blockStore in consensusStateManager.

* [NOD-1413] Add missing methods to UTXODiffStore.

* [NOD-1413] Rename pruningPointStore to pruningStore everywhere.

* [NOD-1413] Remove superfluous parameters from CalculateConsensusStateChanges.

* [NOD-1413] Add missing dependencies to PruningManager.

* [NOD-1413] Remove implementation-y functions from TransactionValidator's interface.

* [NOD-1413] Make go vet happy.

* [NOD-1413] Add a couple of methods to DAGTopologyManager.

* [NOD-1413] Fix a typo in a file name.

* [NOD-1413] Remove non-interface functions from Validator.
2020-10-13 17:55:31 +03:00