157 Commits

Author SHA1 Message Date
stasatdaglabs
bb244706ea
[NOD-1543] Optimize the performance of Count() in BlockHeaderStore and BlockStore (#1109)
* [NOD-1543] Optimize Count() in BlockHeaderStore.

* [NOD-1543] Optimize Count() in BlockStore.

* [NOD-1543] Fix commitCount.

* [NOD-1543] Explicitly initialize count to 0.
2020-11-18 16:35:32 +02:00
Ori Newman
8500acd86b
[NOD-1548] Remove PoW check from tests (#1105)
* [NOD-1548] Add TestDifficulty and remove PoW check from tests

* [NOD-1548] Add TestSkipProofOfWork

* [NOD-1548] Remove TestDifficulty
2020-11-18 10:27:29 +02:00
Ori Newman
60c24d8dea Fix TestBlueBlockWindow (#1098)
* Fix TestBlueBlockWindow

* Add comments
2020-11-17 16:00:16 +02:00
Mike Zak
7224d58940 [NOD-1532] Add comments 2020-11-17 16:00:16 +02:00
stasatdaglabs
48d8137604 [NOD-1538] Implement GetBlockCount. 2020-11-17 16:00:16 +02:00
Mike Zak
a1fa17d872 [NOD-1532] Add DiscardAllStores to TestConsensus 2020-11-17 16:00:16 +02:00
Ori Newman
fc5e39f6cc [NOD-1535] fix reachability test (#1061)
* Revert "[NOD-1500] Delete integration tests"

This reverts commit fcb57a206690a884fa6afb69d5d493282954a8bf.

* [NOD-1518] hashserialization -> consenusserialization

* [NOD-1518] Fix add genesis to virtual

* [NOD-1518] Fix a bug in SerializeCoinbasePayload.

* [NOD-1518] Fix a loop error and make pastMedianTime behave correctly everywhere on genesis.

* [NOD-1518] Fix another bug and an infinite loop.

* [NOD-1518] Fix uninitialized slice.

* [NOD-1518] Fix bad should-commit checks and another infinite loop.

* [NOD-1518] Fix nil serialization.

* [NOD-1518] Rename blockHash to currentBlockHash.

* [NOD-1518] Move the check whether stagedVirtualUTXOSet != nil to the top of commitVirtualUTXODiff.

* [NOD-1518] Simplify utxoDiffStore.Commit.

* [NOD-1518] Unextract resolveBlockStatusAndCheckFinality.

* [NOD-1518] Move no-transactions logic into CalculateIDMerkleRoot.

* [NOD-1518] Remove redundant is-staged check.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Don't write anything if utxoDiffChild is nil.

* [NOD-1518] Stage virtualAcceptanceData and virtualMultiset.

* [NOD-1518] Fix bugs in getBlockTemplate and submitBlock.

* [NOD-1518] Fix bad validation order in validateHeaderInContext.

* [NOD-1518] Fix bug in Next().

* [NOD-1518] Fix nil dereference of subnetworks in AddressCache.

* [NOD-1518] Fix multisetStore.Get returning a pointer to a multiset that is changed in place.

* [NOD-1518] Break on genesis in countSubtrees.

* [NOD-1518] Fix createBlockLocator.

* [NOD-1518] Fix MsgTxToDomainTransaction.

* [NOD-1518] Set MaxTxVersion to 1.

* [NOD-1518] Fix missing error handling, bug in MsgTxToDomainTransaction, and bad subnetwork equality check.

* [NOD-1518] Fix bug in hasUTXOByOutpointFromStagedVirtualUTXODiff.

* [NOD-1518] Remove irrelevant comments.

* [NOD-1518] Generate transactions with sufficient fee in tx_relay_test.

* [NOD-1518] Fix broken RPC handlers.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Fix bad exists check in restorePastUTXO and missing genesis check in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Add a comment.

* [NOD-1518] Use a regular mutex instead of a read-write mutex in consensus to avoid dealing with sneaky not-actually-read functions.

* [NOD-1518] Fix a deadlock in GetVirtualSelectedParent.

* [NOD-1518] Fix missing handler registration for CmdHeader.

* [NOD-1518] Fix processHeader calling OnNewBlock and LogBlock. Also fix conversion errors in IBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad Command() in MsgIBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad SyncStateMissingUTXOSet logic in resolveSyncState.

* [NOD-1518] Rename mode to syncState.

* [NOD-1518] Fix headers-only blocks coming in after the consensus thinks it's synced.

* [NOD-1518] Fix selectedChildIterator.Next not ignoring virtual, infinite loop in HashSet.Length().

* [NOD-1518] Fix not-properly wrapped IBD blocks.

* [NOD-1518] Fix bad conversion in RequestIBDBlocks.

* [NOD-1518] Fix bad string for CmdRequestHeaders.

* [NOD-1518] Fix bad string for CmdDoneHeaders.

* [NOD-1518] Fix bad Command() for MsgIBDRootNotFound.

* [NOD-1518] Fix bad areHeaderTipsSyncedMaxTimeDifference value.

* [NOD-1518] Add missing string for CmdRequestIBDBlocks.

* [NOD-1518] Fix bad check for SyncStateMissingBlockBodies.

* [NOD-1518] Fix bad timeout durations in tests.

* [NOD-1518] Fix IBD blocks not calling OnNewBlock.

* [NOD-1518] Change when IBD finishes.

* [NOD-1518] Properly clone utxoDiffChild.

* [NOD-1535] Fix reachability tests

* [NOD-1518] Fix merge errors.

* [NOD-1518] Move call to LogBlock to into OnNewBlock.

* [NOD-1518] Return "not implemented" in unimplemented RPC handlers.

* [NOD-1518] Extract cloning of hashes to a method over DomainHash.

* [NOD-1518] Use isHeaderOnlyBlock.

* [NOD-1518] Use constants.TransactionVersion.

* [NOD-1518] Break immediately if we reached the virtual in SelectedChildIterator.

* [NOD-1518] Don't stage nil utxoDiffChild.

* [NOD-1518] Properly check the genesis hash in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Explain why we break on current == nil in countSubtrees.

* [NOD-1518] Add a comment explaining why we check against StatusValid in resolveSyncState.

* [NOD-1535] Add external reachability tests

* [NOD-1535] Fix reachability tests and fix related bugs

* [NOD-1535] Add setters fox reindex slack and window

* [NOD-1535] Remove redundant line

* [NOD-1535] Add comment

* [NOD-1535] Fix comments

* [NOD-1535] Rename DBReader->DatabaseContext

* [NOD-1535] Check that reindex root is changed

* [NOD-1535] Fix calculateNewTips

Co-authored-by: Mike Zak <feanorr@gmail.com>
Co-authored-by: stasatdaglabs <stas@daglabs.com>
2020-11-17 16:00:16 +02:00
Svarog
8ccf381fc7 [NOD-1532] csm unit tests (#1059)
* Revert "[NOD-1500] Delete integration tests"

This reverts commit fcb57a206690a884fa6afb69d5d493282954a8bf.

* [NOD-1518] hashserialization -> consenusserialization

* [NOD-1518] Fix add genesis to virtual

* [NOD-1518] Fix a bug in SerializeCoinbasePayload.

* [NOD-1518] Fix a loop error and make pastMedianTime behave correctly everywhere on genesis.

* [NOD-1518] Fix another bug and an infinite loop.

* [NOD-1518] Fix uninitialized slice.

* [NOD-1518] Fix bad should-commit checks and another infinite loop.

* [NOD-1518] Fix nil serialization.

* [NOD-1518] Rename blockHash to currentBlockHash.

* [NOD-1518] Move the check whether stagedVirtualUTXOSet != nil to the top of commitVirtualUTXODiff.

* [NOD-1518] Simplify utxoDiffStore.Commit.

* [NOD-1518] Unextract resolveBlockStatusAndCheckFinality.

* [NOD-1518] Move no-transactions logic into CalculateIDMerkleRoot.

* [NOD-1518] Remove redundant is-staged check.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Don't write anything if utxoDiffChild is nil.

* [NOD-1518] Stage virtualAcceptanceData and virtualMultiset.

* [NOD-1518] Fix bugs in getBlockTemplate and submitBlock.

* [NOD-1518] Fix bad validation order in validateHeaderInContext.

* [NOD-1518] Fix bug in Next().

* [NOD-1518] Fix nil dereference of subnetworks in AddressCache.

* [NOD-1518] Fix multisetStore.Get returning a pointer to a multiset that is changed in place.

* [NOD-1518] Break on genesis in countSubtrees.

* [NOD-1518] Fix createBlockLocator.

* [NOD-1518] Fix MsgTxToDomainTransaction.

* [NOD-1518] Set MaxTxVersion to 1.

* [NOD-1518] Fix missing error handling, bug in MsgTxToDomainTransaction, and bad subnetwork equality check.

* [NOD-1518] Fix bug in hasUTXOByOutpointFromStagedVirtualUTXODiff.

* [NOD-1518] Remove irrelevant comments.

* [NOD-1518] Generate transactions with sufficient fee in tx_relay_test.

* [NOD-1518] Fix broken RPC handlers.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Fix bad exists check in restorePastUTXO and missing genesis check in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Add a comment.

* [NOD-1518] Use a regular mutex instead of a read-write mutex in consensus to avoid dealing with sneaky not-actually-read functions.

* [NOD-1518] Fix a deadlock in GetVirtualSelectedParent.

* [NOD-1518] Fix missing handler registration for CmdHeader.

* [NOD-1518] Fix processHeader calling OnNewBlock and LogBlock. Also fix conversion errors in IBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad Command() in MsgIBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad SyncStateMissingUTXOSet logic in resolveSyncState.

* [NOD-1518] Rename mode to syncState.

* [NOD-1518] Fix headers-only blocks coming in after the consensus thinks it's synced.

* [NOD-1518] Fix selectedChildIterator.Next not ignoring virtual, infinite loop in HashSet.Length().

* [NOD-1518] Fix not-properly wrapped IBD blocks.

* [NOD-1532] Add TestMultiset

* [NOD-1518] Fix bad conversion in RequestIBDBlocks.

* [NOD-1518] Fix bad string for CmdRequestHeaders.

* [NOD-1518] Fix bad string for CmdDoneHeaders.

* [NOD-1518] Fix bad Command() for MsgIBDRootNotFound.

* [NOD-1532] Add TestPastUTXOMultiset

* [NOD-1518] Fix bad areHeaderTipsSyncedMaxTimeDifference value.

* [NOD-1532] Added TestDoubleSpends

* [NOD-1518] Add missing string for CmdRequestIBDBlocks.

* [NOD-1518] Fix bad check for SyncStateMissingBlockBodies.

* [NOD-1518] Fix bad timeout durations in tests.

* [NOD-1518] Fix IBD blocks not calling OnNewBlock.

* [NOD-1518] Change when IBD finishes.

* [NOD-1518] Properly clone utxoDiffChild.

* [NOD-1532] Update hashes of blocks

* [NOD-1532] Fix genesis blocks and a few more bugs

* [NOD-1532] Bugfix: incorrect key passed to dbTx.Put

* [NOD-1532] Make sure there's no nil payloads

* [NOD-1532] Fix AddBlockToVirtual

* [NOD-1532] Update tips and virtualDiffParents properly

* [NOD-1532] Allow nil payload

* [NOD-1532] Check for actual error and not just some RuleError

* [NOD-1532] Get rid of SimpleCoinbaseData and make OpTrueScript P2SH

* [NOD-1532] If coinbaseData is nil - fill in with generic coinbaseData

Co-authored-by: Ori Newman <orinewman1@gmail.com>
Co-authored-by: stasatdaglabs <stas@daglabs.com>
2020-11-17 16:00:16 +02:00
stasatdaglabs
eef5e3768c
[NOD-1518] Fix genesis block insertion and integration tests (#1013)
* Revert "[NOD-1500] Delete integration tests"

This reverts commit fcb57a206690a884fa6afb69d5d493282954a8bf.

* [NOD-1518] hashserialization -> consenusserialization

* [NOD-1518] Fix add genesis to virtual

* [NOD-1518] Fix a bug in SerializeCoinbasePayload.

* [NOD-1518] Fix a loop error and make pastMedianTime behave correctly everywhere on genesis.

* [NOD-1518] Fix another bug and an infinite loop.

* [NOD-1518] Fix uninitialized slice.

* [NOD-1518] Fix bad should-commit checks and another infinite loop.

* [NOD-1518] Fix nil serialization.

* [NOD-1518] Rename blockHash to currentBlockHash.

* [NOD-1518] Move the check whether stagedVirtualUTXOSet != nil to the top of commitVirtualUTXODiff.

* [NOD-1518] Simplify utxoDiffStore.Commit.

* [NOD-1518] Unextract resolveBlockStatusAndCheckFinality.

* [NOD-1518] Move no-transactions logic into CalculateIDMerkleRoot.

* [NOD-1518] Remove redundant is-staged check.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Don't write anything if utxoDiffChild is nil.

* [NOD-1518] Stage virtualAcceptanceData and virtualMultiset.

* [NOD-1518] Fix bugs in getBlockTemplate and submitBlock.

* [NOD-1518] Fix bad validation order in validateHeaderInContext.

* [NOD-1518] Fix bug in Next().

* [NOD-1518] Fix nil dereference of subnetworks in AddressCache.

* [NOD-1518] Fix multisetStore.Get returning a pointer to a multiset that is changed in place.

* [NOD-1518] Break on genesis in countSubtrees.

* [NOD-1518] Fix createBlockLocator.

* [NOD-1518] Fix MsgTxToDomainTransaction.

* [NOD-1518] Set MaxTxVersion to 1.

* [NOD-1518] Fix missing error handling, bug in MsgTxToDomainTransaction, and bad subnetwork equality check.

* [NOD-1518] Fix bug in hasUTXOByOutpointFromStagedVirtualUTXODiff.

* [NOD-1518] Remove irrelevant comments.

* [NOD-1518] Generate transactions with sufficient fee in tx_relay_test.

* [NOD-1518] Fix broken RPC handlers.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Fix bad exists check in restorePastUTXO and missing genesis check in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Add a comment.

* [NOD-1518] Use a regular mutex instead of a read-write mutex in consensus to avoid dealing with sneaky not-actually-read functions.

* [NOD-1518] Fix a deadlock in GetVirtualSelectedParent.

* [NOD-1518] Fix missing handler registration for CmdHeader.

* [NOD-1518] Fix processHeader calling OnNewBlock and LogBlock. Also fix conversion errors in IBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad Command() in MsgIBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad SyncStateMissingUTXOSet logic in resolveSyncState.

* [NOD-1518] Rename mode to syncState.

* [NOD-1518] Fix headers-only blocks coming in after the consensus thinks it's synced.

* [NOD-1518] Fix selectedChildIterator.Next not ignoring virtual, infinite loop in HashSet.Length().

* [NOD-1518] Fix not-properly wrapped IBD blocks.

* [NOD-1518] Fix bad conversion in RequestIBDBlocks.

* [NOD-1518] Fix bad string for CmdRequestHeaders.

* [NOD-1518] Fix bad string for CmdDoneHeaders.

* [NOD-1518] Fix bad Command() for MsgIBDRootNotFound.

* [NOD-1518] Fix bad areHeaderTipsSyncedMaxTimeDifference value.

* [NOD-1518] Add missing string for CmdRequestIBDBlocks.

* [NOD-1518] Fix bad check for SyncStateMissingBlockBodies.

* [NOD-1518] Fix bad timeout durations in tests.

* [NOD-1518] Fix IBD blocks not calling OnNewBlock.

* [NOD-1518] Change when IBD finishes.

* [NOD-1518] Properly clone utxoDiffChild.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Move call to LogBlock to into OnNewBlock.

* [NOD-1518] Return "not implemented" in unimplemented RPC handlers.

* [NOD-1518] Extract cloning of hashes to a method over DomainHash.

* [NOD-1518] Use isHeaderOnlyBlock.

* [NOD-1518] Use constants.TransactionVersion.

* [NOD-1518] Break immediately if we reached the virtual in SelectedChildIterator.

* [NOD-1518] Don't stage nil utxoDiffChild.

* [NOD-1518] Properly check the genesis hash in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Explain why we break on current == nil in countSubtrees.

* [NOD-1518] Add a comment explaining why we check against StatusValid in resolveSyncState.

Co-authored-by: Mike Zak <feanorr@gmail.com>
Co-authored-by: Ori Newman <orinewman1@gmail.com>
2020-11-12 15:19:39 +02:00
Svarog
135ffbd4f2
[NOD-1529] Add getters + AddBlock to TestConsensus (#1025)
* [NOD-1529] Add all stores and processes to consensus, and add access to TestConsensus

* [NOD-1529] Move the getters of TestConsensus to separate file

* [NOD-1529] Add AddBlock to TestConsensus

* [NOD-1529] Update NewTestConsensus to be more all-encompassing

* [NOD-1529] Remove test directory in teardown

* [NOD-1529] Add ForAllNets function

* [NOD-1529] Add comment
2020-11-11 12:31:13 +02:00
Ori Newman
4736213ba4
[NOD-1528] Make data stores copy data on stage (#1020)
* [NOD-1528] Make data stores copy data on stage

* [NOD-1528] Add proto objects to serialize consensus state objects

* [NOD-1528] Fix receiver names

* [NOD-1528] Add copy to block store and utxo diff staging

* [NOD-1528] Return errors where needed
2020-11-10 18:32:42 +02:00
Ori Newman
23c1ea6c31
[NOD-1525] Implement headers first ibd (#1017)
* [NOD-1525] Implement headers first IBD

* [NOD-1525] Fix proto translators

* [NOD-1525] Register missing flows

* [NOD-1525] Rename SyncStateNormal->SyncStateRelay, simplifiy IBD peer selection code and get rid of panic in FinishIBD

* [NOD-1525] Remove redundant methods from interface
2020-11-10 16:14:51 +02:00
Svarog
6db337c8c5
[NOD-1519] Add TestAPI for BuildBlockWithParents (#1011)
* [NOD-1519] Separate BlockBuilder and BlockProcessor

* [NOD-1519] Wire blockBuilder properly + implement buildBlockWithParents

* [NOD-1519] Added testapi package, TestConsensus interface, and TestConsensus factory

* [NOD-1519] Add comments

* [NOD-1519] Separate TestBlockBuilder out of BlockBuilder

* [NOD-1519] TestBlockBuilder should also implement BlockBuilder

* [NOD-1519] Add NewTestConsensus to factory interface
2020-11-09 12:02:42 +02:00
Ori Newman
9a344152aa
[NOD-1517] Properly initialize consensus with Genesis block (#1009)
* [NOD-1517] Properly initialize consensus with Genesis block

* [NOD-1517] Remove redundant AddHeaderTip

* [NOD-1517] Don't return nil from dbHash<->DomainHash converters

* [NOD-1517] Use pointer receivers

* [NOD-1517] Use domain block in dagParams

* [NOD-1517] Remove boolean from SelectedTip

* [NOD-1517] Rename hasHeader to isHeadersOnlyBlock

* [NOD-1517] Add comment

* [NOD-1517] Change genesis version

* [NOD-1517] Rename TestNewFactory->TestNewConsensus
2020-11-08 15:17:20 +02:00
Svarog
281944762d
[NOD-1500] Glue between domain and application (#1007)
* [NOD-1500] Added Domain type and Constructor

* [NOD-1500] Replaced dag+txpool with domain in flowContext

* [NOD-1500] Replaced dag+txpool with domain in flowContext

* [NOD-1500] Converters: domain objects from/to appmessage

* [NOD-1500] Convert hashes to DomainHashes in appmessages

* [NOD-1500] Remove references to daghash in dagconfig

* [NOD-1500] Fixed all appmessage usages of hashes

* [NOD-1500] Update all RPC to use domain

* [NOD-1500] Big chunk of protocol flows re-wired to domain

* [NOD-1500] Finished re-wiring all protocol flows to new Domain

* [NOD-1500] Fix some mempool and kaspaminer compilation errors

* [NOD-1500] Deleted util/{block,tx,daghash} and dbaccess

* [NOD-1500] util.CoinbaseTransactionIndex -> transactionhelper.CoinbaseTransactionIndex

* [NOD-1500] Fix txsigner

* [NOD-1500] Removed all references to util/subnetworkid

* [NOD-1500] Update RpcGetBlock related messages

* [NOD-1500] Many more compilation fixes

* [NOD-1500] Return full list of missing blocks for orphan resolution

* [NOD-1500] Fixed handshake

* [NOD-1500] Fixed flowcontext compilation

* [NOD-1500] Update users of StartIBDIfRequired to handle error

* [NOD-1500] Removed some more fields from RPC

* [NOD-1500] Fix the getBlockTemplate flow

* [NOD-1500] Fix HandleGetCurrentNetwork

* [NOD-1500] Remove redundant code

* [NOD-1500] Remove obsolete notifications

* [NOD-1500] Split MiningManager and Consensus to separate fields in Domain

* [NOD-1500] Update two wrong references to location of txscript

* [NOD-1500] Added comments

* [NOD-1500] Fix some tests

* [NOD-1500] Removed serialization logic from appmessage

* [NOD-1500] Rename database/serialization/messages.proto to dbobjects.proto

* [NOD-1500] Delete integration tests

* [NOD-1500] Remove txsort

* [NOD-1500] Fix tiny bug

* [NOD-1500] Remove rogue dependancy on bchd

* [NOD-1500] Some stylistic fixes
2020-11-08 11:55:54 +02:00
stasatdaglabs
d0fc728c23
[NOD-1514] Allow full block validation during IBD. (#1004) 2020-11-05 12:18:16 +02:00
stasatdaglabs
1c710daf98
[NOD-1501] Finish implementing GetSyncInfo (#1002)
* [NOD-1501] Rename IsBlockInHeaderPruningPointFutureAndVirtualPast to IsBlockInHeaderPruningPointFuture.

* [NOD-1501] Create syncinfo.go.

* [NOD-1501] Implement resolveSyncState.

* [NOD-1501] Fix ChooseSelectedParent.

* [NOD-1501] Fix merge errors.

* [NOD-1501] Finish implementing getSyncState.

* [NOD-1501] Fix bad equality check.

* [NOD-1501] Fix merge errors.

* [NOD-1501] Pass targetTimePerBlock as int64 milliseconds.
2020-11-05 11:50:59 +02:00
Ori Newman
5566aaf95a
[NOD-1512] Implement utxo deserialization (#1003)
* [NOD-1512] Implement UTXO set deserialization

* [NOD-1512] Remove redundant file

* [NOD-1512] Don't use big endian for serialization

* [NOD-1512] Use Read/Write element

* [NOD-1512] Unexport ReadElement

* [NOD-1512] Fix StageVirtualUTXOSet

* [NOD-1512] Get rid of dagParams in consensusStateManager

* [NOD-1512] Get rid of dagParams in consensusStateManager
2020-11-05 10:59:49 +02:00
Ori Newman
52c73d3a08
[NOD-1511] Implement missingBlockBodyHashes (#1000)
* [NOD-1511] Implement missingBlockBodyHashes

* [NOD-1511] Rename selectedparentiterator.go to blockiterator.go

* [NOD-1511] Fix condition

* [NOD-1511] Simplify missingBlocks logic
2020-11-05 10:11:42 +02:00
Ori Newman
baf8d25656
[NOD-1510] Update reachability reindex root based on header tips selected tip (#1001) 2020-11-04 16:55:59 +02:00
Ori Newman
2eb0d946e5
[NOD-1509] Implement StageVirtualUTXOSet (#999)
* [NOD-1509] Implement StageVirtualUTXODiff

* [NOD-1509] Fix HasUTXOByOutpoint, get rid of database.ErrNotFound, and fix errors
2020-11-04 16:30:08 +02:00
Elichai Turkel
6a46cb2be6
[NOD-1503] Pruning Manager (#994)
* Update pruningmanager interface

* Add a ProtoUTXOSet to hashserialization

* Update miningmanager with all the necessary stores and managers

* Implement mining manager

* Prune P.AC not in V.Past

* PruningManager fix all review comments
2020-11-04 10:29:45 +02:00
Ori Newman
7f2ef708a6
[NOD-1506] Implement SetPruningPointUTXOSet (#996)
* [NOD-1506] Implement SetPruningPointUTXOSet

* [NOD-1506] Rename ErrHeaderlessBlockInIBD->ErrMissingBlockHeaderInIBD

* [NOD-1506] Change virtualHeaderHash
2020-11-03 18:24:45 +02:00
stasatdaglabs
ca9161024f
[NOD-1501] Add SyncManager-related interfaces (#995)
* [NOD-1501] Add logAndMeasureExecutionTime to SyncManager methods.

* [NOD-1501] Implement antiPastHashesBetween.

* [NOD-1501] Implement createBlockLocator.

* [NOD-1501] Implement findNextBlockLocatorBoundaries.

* [NOD-1501] Rename IsBlockHeaderInPruningPointFutureAndVirtualPast to IsBlockInHeaderPruningPointFutureAndVirtualPast.

* [NOD-1501] Add GetSyncInfo.

* [NOD-1501] Make go vet happy.

* [NOD-1501] Rename sync states.

* [NOD-1501] Move maxHashesInGetHashesBetween to antipast.go.

* [NOD-1501] Rename maxHashesInAntiPastHashesBetween.

* [NOD-1501] Implement LowestChainBlockAboveOrEqualToBlueScore.

* [NOD-1501] Fix bad variable name.

* [NOD-1501] Fix LowestChainBlockAboveOrEqualToBlueScore.

* [NOD-1501] Clarify LowestChainBlockAboveOrEqualToBlueScore.
2020-11-03 17:08:52 +02:00
Svarog
8dc246a2a7
[NOD-1498] Consensus State Store (#992)
* [NOD-1420] Start working on ConsensusStateManager. Might be redundant due to recent changes

* [NOD-1420] Convert model to externalapi in utxo_algerbra helpers

* [NOD-1420] Add UTXO-diff algebra

* [NOD-1420] Prepare skeleton of calculateAcceptanceDataAndMultiset

* [NOD-1420] Added skeleton for AddBlockToVirtual

* [NOD-1420] Implement PopulateTransactionWithUTXOEntries

* [NOD-1420] Implement restorePastUTXO

* [NOD-1420] Implement finality check

* [NOD-1420] Move handling of tips to consensusStateManager

* [NOD-1420] Implement calculateAcceptanceDataAndMultiset

* [NOD-1420] Start implementing resolveBlockStatus

* [NOD-1420] Implement resolveBlockStatus

* [NOD-1420] Update related fields in end of resolveSingleBlockStatus

* [NOD-1420] Start working on selectVirtualParents

* [NOD-1420] Implemented BlockHeap

* [NOD-1420] Implement selectVirtualParents

* [NOD-1420] Implement updateVirtual

* [NOD-1420] Added comments where they were missing

* [NOD-1420] Place all consensusStateManager functions in correct files

* [NOD-1420] Return the missing outpoints from populateTransactionWithUTXOEntriesFromVirtualOrDiff

* [NOD-1420] Outpoint.ID -> TransactionID

* [NOD-1420] Fix Stringer tests

* [NOD-1420] Copy hash.FromString into utils

* [NOD-1420] SetParents should return an error

* [NOD-1420] Remove all reachabilityManager references from consensusStateManager

* [NOD-1420] Remove VirtualData. Get the info from the stores where needed

* [NOD-1420] Invert parameters to IsAncestorOf

* [NOD-1420] Use model.AcceptanceData

* [NOD-1420] Don't return accumulatedMassBefore in error cases

* [NOD-1420] Don't expect store functions to return nil when the requested data was found - instead add HasXXX functions

* [NOD-1420] addTransactionToMultiset sets isCoinbase properly

* [NOD-1420] expected hash string length is externalapi.DomainHashSize * 2

* [NOD-1420] Rename reachabilityTree -> reachabilityManager + updateReindexRoot if isNextVirtualSelectedParent

* [NOD-1420] ValidateCoinbaseTransaction in csm.verifyAndBuildUTXO

* [NOD-1420] Re-write HAsUTXODiffChild

* [NOD-1420] delete past_utxo.go.bak

* [NOD-1420] Implement validateCoinbaseTransaction in CSM

* [NOD-1420] Imlemented missing functionality in ValidateTransactionAndPopulateWithConsensusData

* [NOD-1420] Moved merge depth logic to MergeDepthManager

* [NOD-1420] Add logs

* [NOD-1498] Implement tips-related methods of consensusStateStore

* [NOD-1498] Implement consensusStateStore virtualDiffParents functionality

* [NOD-1498] Implement ConsensusStateStore UTXO-Set part

* [NOD-1498] Implement rest of consensusStateStore methods

* [NOD-1498] Use io.ReadFull instead of r.Read

* [NOD-1498] Added comments

* [NOD-1498] Move utxo serialization to protobufs

* [NOD-1498] Add comments

* [NOD-1498] Minor fixes in ConsensusStateStore

* [NOD-1498] Use empty bucket key + simplify serializeUTXOEntry
2020-11-03 14:42:26 +02:00
Ori Newman
c7f2de73df
[NOD-1502] Implement RestorePastUTXOSetIterator (#993)
* [NOD-1502] Implement RestorePastUTXOSetIterator

* [NOD-1502] Rename newUtxoSetIterator->newUTXOSetIterator
2020-11-02 18:36:55 +02:00
Ori Newman
2abd4a274b
[NOD-1496] Implement headers only verification (#987)
* [NOD-1496] Implement headers only verification

* [NOD-1496] Add checkParentsExist

* [NOD-1496] Stage block statuses in block processor

* [NOD-1496] Rename AddBlock->AddHeaderTip

* [NOD-1496] Return early from validateAndInsertBlock on header only and put ValidateProofOfWorkAndDifficulty inside validateBlock
2020-11-02 16:30:59 +02:00
Svarog
c5707f64dc
[NOD-1420] Implement consensusStateManager (#985)
* [NOD-1420] Start working on ConsensusStateManager. Might be redundant due to recent changes

* [NOD-1420] Convert model to externalapi in utxo_algerbra helpers

* [NOD-1420] Add UTXO-diff algebra

* [NOD-1420] Prepare skeleton of calculateAcceptanceDataAndMultiset

* [NOD-1420] Added skeleton for AddBlockToVirtual

* [NOD-1420] Implement PopulateTransactionWithUTXOEntries

* [NOD-1420] Implement restorePastUTXO

* [NOD-1420] Implement finality check

* [NOD-1420] Move handling of tips to consensusStateManager

* [NOD-1420] Implement calculateAcceptanceDataAndMultiset

* [NOD-1420] Start implementing resolveBlockStatus

* [NOD-1420] Implement resolveBlockStatus

* [NOD-1420] Update related fields in end of resolveSingleBlockStatus

* [NOD-1420] Start working on selectVirtualParents

* [NOD-1420] Implemented BlockHeap

* [NOD-1420] Implement selectVirtualParents

* [NOD-1420] Implement updateVirtual

* [NOD-1420] Added comments where they were missing

* [NOD-1420] Place all consensusStateManager functions in correct files

* [NOD-1420] Return the missing outpoints from populateTransactionWithUTXOEntriesFromVirtualOrDiff

* [NOD-1420] Outpoint.ID -> TransactionID

* [NOD-1420] Fix Stringer tests

* [NOD-1420] Copy hash.FromString into utils

* [NOD-1420] SetParents should return an error

* [NOD-1420] Remove all reachabilityManager references from consensusStateManager

* [NOD-1420] Remove VirtualData. Get the info from the stores where needed

* [NOD-1420] Invert parameters to IsAncestorOf

* [NOD-1420] Use model.AcceptanceData

* [NOD-1420] Don't return accumulatedMassBefore in error cases

* [NOD-1420] Don't expect store functions to return nil when the requested data was found - instead add HasXXX functions

* [NOD-1420] addTransactionToMultiset sets isCoinbase properly

* [NOD-1420] expected hash string length is externalapi.DomainHashSize * 2

* [NOD-1420] Rename reachabilityTree -> reachabilityManager + updateReindexRoot if isNextVirtualSelectedParent

* [NOD-1420] ValidateCoinbaseTransaction in csm.verifyAndBuildUTXO

* [NOD-1420] Re-write HAsUTXODiffChild

* [NOD-1420] delete past_utxo.go.bak

* [NOD-1420] Implement validateCoinbaseTransaction in CSM

* [NOD-1420] Imlemented missing functionality in ValidateTransactionAndPopulateWithConsensusData

* [NOD-1420] Moved merge depth logic to MergeDepthManager

* [NOD-1420] Add logs
2020-11-02 16:18:53 +02:00
stasatdaglabs
62bb841e89
[NOD-1497] Add additional methods to consensus' external API (#991)
* [NOD-1497] Add missing APIs.

* [NOD-1497] Rename some new APIs.

* [NOD-1497] Implement getBlock and getBlockHeader.

* [NOD-1497] Implement getPruningPointUTXOSet.

* [NOD-1497] Implement getSelectedParent.

* [NOD-1497] Implement getBlockInfo.

* [NOD-1497] Fix merge errors.

* [NOD-1497] Implement syncManager-related functions in consensus.

* [NOD-1497] Implement SetPruningPointUTXOSet in consensus.

* [NOD-1497] Add dependency from syncManager to dagTraversalManager.

* [NOD-1497] Move IsBlockHeaderInPruningPointFutureAndVirtualPast to syncManager.

* [NOD-1497] Rename lowHigh to lowHash.
2020-11-02 13:24:03 +02:00
stasatdaglabs
23cccb6396
[NOD-1497] Add additional methods to consensus external API (#989)
* [NOD-1497] Add missing APIs.

* [NOD-1497] Rename some new APIs.

* [NOD-1497] Add fields to BlockInfo.

* [NOD-1497] Add comments over BlockInfo and BlockLocator.

* [NOD-1497] Rename GetSelectedParent to GetVirtualSelectedParent.

* [NOD-1497] Add SetPruningPointUTXOSet.

* [NOD-1497] Rename GetHashesAbovePruningPoint to GetMissingBlockBodyHashes.

* [NOD-1497] Fix rename error.
2020-11-02 12:05:33 +02:00
Ori Newman
c59adaa4db
[NOD-1494] Remove transactions from data store delete (#984)
* [NOD-1494] Remove transactions from data store delete

* [NOD-1494] Remove redundant underscores
2020-10-29 18:15:11 +02:00
Ori Newman
14fbe50636
[NOD-1493] Implement serialization in data stores (#983)
* [NOD-1493] Implement serialization in data stores

* [NOD-1493] Remove redundant functions

* [NOD-1493] Use bluesAnticoneSizesToDBBluesAnticoneSizes inside BlockGHOSTDAGDataToDBBlockGHOSTDAGData
2020-10-29 17:39:35 +02:00
Elichai Turkel
d3ede3a46f
Add new ErrMissingTxOut and ErrInvalidTransactionsInNewBlock errors (#972)
* Add new ErrMissingTxOut error

* Add tests for ruleError wrapping

* Update consensus to use new ErrMissingTxOut type where appropriate

* Add new ErrInvalidTransactionsInNewBlock error

* Add wrapping tests for ErrInvalidTransactionsInNewBlock

* Fix Review suggestions

* Fix broken serialization(add pointer redirection)
2020-10-29 16:59:00 +02:00
Elichai Turkel
971d50b684
[NOD-1418] Implement DAG Traversal (#953)
* Implement DAG Traversal

* Update the DAGTraversalManager interface
2020-10-29 16:48:41 +02:00
stasatdaglabs
126e2e49bb
[NOD-1493] Implement serialization/deserialization inside BlockHeaderStore (#979)
* [NOD-1492] Rename dbmanager to database.

* [NOD-1492] Write messages.proto for DbBlock and DbTransaction.

* [NOD-1492] Implement serializeHeader.

* [NOD-1492] Implement deserializeHeader.
2020-10-29 11:15:14 +02:00
stasatdaglabs
c88266afed
[NOD-1492] Implement GHOSTDAGDataStore, MultisetStore, PruningStore, ReachabilityDataStore, and UTXODiffStore (#977)
* [NOD-1492] Implement GHOSTDAGDataStore.

* [NOD-1492] Implement MultisetStore.

* [NOD-1492] Implement PruningStore.

* [NOD-1492] Implement ReachabilityDataStore.

* [NOD-1492] Implement UTXODiffStore.

* [NOD-1492] Pluralize the multiset bucket name.

* [NOD-1492] In PruningPoint and PruningPointSerializedUTXOSet, don't use IsStaged.

* [NOD-1492] Leave pruning point serialization/deserialization for future implementation.

* [NOD-1492] Leave reachability reindex root serialization/deserialization for future implementation.

* [NOD-1492] Leave utxo diff child serialization/deserialization for future implementation.

* [NOD-1492] Add Serialize() to Multiset.

* [NOD-1492] Also check serializedUTXOSetStaging in IsStaged.

* [NOD-1492] Also check utxoDiffChildStaging in IsStaged.

* [NOD-1492] Fix UTXODiffStore.Delete.
2020-10-28 17:13:14 +02:00
Ori Newman
7402f3fb0e
[NOD-1492] Implement some data stores (#978)
* [NOD-1492] Implement some data stores

* [NOD-1492] Remove pointers to acceptance data

* [NOD-1492] Fix receiver names

* [NOD-1492] Implement delete for acceptanceDataStore

* [NOD-1492] In blockRelationStore rename IsAnythingStaged to IsStaged

* [NOD-1492] Rename bucket name
2020-10-28 16:30:55 +02:00
Ori Newman
eae8bce941
[NOD-1491] Implement block headers store (#976)
* [NOD-1491] Implement block headers store

* [NOD-1491] Don't commit transaction and delete from staging too
2020-10-28 14:34:00 +02:00
Ori Newman
8c0275421a
[NOD-1490] Implement database manager (#975) 2020-10-28 12:59:34 +02:00
Ori Newman
a436b30ebf
[NOD-1417] Implement reachability (#964)
* [NOD-1417] Implement reachability

* [NOD-1417] Rename package name

* [NOD-1417] Add UpdateReindexRoot to interface api

* [NOD-1417] Remove redundant type

* [NOD-1417] Rename reachabilityTreeManager/reachabilityTree to reachabilityManager

* [NOD-1417] Fix typo

* [NOD-1417] Remove redundant copyright message

* [NOD-1417] Fix comment
2020-10-28 12:19:50 +02:00
Ori Newman
be56fb7e8b
[NOD-1488] Get rid of dbaccess (#973)
* [NOD-1488] Get rid of dbaccess

* [NOD-1488] Rename dbwrapper to dbmanager

* [NOD-1488] Create DBWriter interface

* [NOD-1488] Fix block header store

* [NOD-1488] Rename dbwrapper.go to dbmanager.go
2020-10-28 11:34:06 +02:00
stasatdaglabs
4fbe130592
[NOD-1489] Add BlockHeaderStore (#974)
* [NOD-1489] Add BlockHeaderStore.

* [NOD-1489] Use BlockHeaderStore.
2020-10-28 10:35:18 +02:00
Ori Newman
ed6d8243ef
[NOD-1487] Implement dagtopology's IsAncestorOfAny and IsInSelectedParentChainOf (#971)
* [NOD-1487] Implement dagtopology's IsAncestorOfAny and IsInSelectedParentChainOf

* [NOD-1487] Fix IsInSelectedParentChainOf to use reachabilityTree
2020-10-27 17:46:30 +02:00
Ori Newman
03790ad8a2
[NOD-1469] Implement past median time (#968)
* [NOD-1469] Implement past median time

* [NOD-1469] Move BlueWindow to DAGTraversalManager
2020-10-27 17:45:47 +02:00
stasatdaglabs
97b5b0b875
[NOD-1416] Implement BlockProcessor. (#969)
* [NOD-1416] Add entry/exit logs to all the functions.

* [NOD-1416] Build some scaffolding inside BlockProcessor.

* [NOD-1416] Implement selectParentsForNewBlock.

* [NOD-1416] Implement validateBlock.

* [NOD-1476] Fix merge errors.

* [NOD-1416] Move buildBlock and validateAndInsertBlock to separate files.

* [NOD-1416] Begin implementing buildBlock.

* [NOD-1416] Implement newBlockDifficulty.

* [NOD-1416] Add skeletons for the rest of the buildBlock functions.

* [NOD-1416] Implement newBlockUTXOCommitment.

* [NOD-1416] Implement newBlockAcceptedIDMerkleRoot.

* [NOD-1416] Implement newBlockHashMerkleRoot.

* [NOD-1416] Fix bad function call.

* [NOD-1416] Implement validateHeaderAndProofOfWork and validateBody.

* [NOD-1416] Use ValidateProofOfWorkAndDifficulty.

* [NOD-1416] Finish validateAndInsertBlock.

* [NOD-1416] Implement newBlockHashMerkleRoot.

* [NOD-1416] Implement newBlockAcceptedIDMerkleRoot.

* [NOD-1416] Fix a comment.

* [NOD-1416] Implement newBlockCoinbaseTransaction.

* [NOD-1416] Add VirtualBlockHash.

* [NOD-1416] Add ParentHashes and SelectedParent to VirtualData().

* [NOD-1416] Make go vet happy.

* [NOD-1416] Implement discardAllChanges.

* [NOD-1416] Implement commitAllChanges.

* [NOD-1416] Fix factory.

* [NOD-1416] Make go vet happy.

* [NOD-1416] Format factory.

* [NOD-1416] Pass transactionsWithCoinbase to buildHeader.

* [NOD-1416] Call VirtualData() from buildHeader.

* [NOD-1416] Fix a typo.

* [NOD-1416] Fix in-out-of-context/header-body confusion.

* [NOD-1416] Extract LogAndMeasureExecutionTime.

* [NOD-1416] Add a comment about LogAndMeasureExecutionTime.

* [NOD-1416] Simplify discardAllChanges and commitAllChanges.

* [NOD-1416] If in-context validations fail, discard all changes and store the block with StatusInvalid.

* [NOD-1416] Add a comment above Store.

* [NOD-1416] Use errors.As instead of errors.Is.
2020-10-27 17:24:15 +02:00
Ori Newman
aeb4b96560
[NOD-1451] Implement Validators (#966)
* [NOD-1451] Implement block validator

* [NOD-1451] Implement block validator

* [NOD-1451] Fix merge errors

* [NOD-1451] Implement block validator

* [NOD-1451] Implement checkTransactionInIsolation

* [NOD-1451] Copy txscript to validator

* [NOD-1451] Change txscript to new design

* [NOD-1451] Add checkTransactionInContext

* [NOD-1451] Add checkBlockSize

* [NOD-1451] Add error handling

* [NOD-1451] Implement checkTransactionInContext

* [NOD-1451] Add checkTransactionMass placeholder

* [NOD-1451] Finish validators

* [NOD-1451] Add comments and stringers

* [NOD-1451] Return model.TransactionValidator interface

* [NOD-1451] Premake rule errors for each "code"

* [NOD-1451] Populate transaction mass

* [NOD-1451] Renmae functions

* [NOD-1451] Always use skipPow=false

* [NOD-1451] Renames

* [NOD-1451] Remove redundant types from WriteElement

* [NOD-1451] Fix error message

* [NOD-1451] Add checkTransactionPayload

* [NOD-1451] Add ValidateProofOfWorkAndDifficulty to block validator interface

* [NOD-1451] Move stringers to model

* [NOD-1451] Fix error message
2020-10-26 17:33:39 +02:00
stasatdaglabs
b413760136
[NOD-1476] Make further design changes (#965)
* [NOD-1476] Add dependency to BlockRelationStore in BlockProcessor.

* [NOD-1476] Add dependency to BlockStatusStore in BlockValidator.

* [NOD-1476] Add dependency to GHOSTDAGManager in BlockValidator.

* [NOD-1476] Rename CalculateConsensusStateChanges to AddBlockToVirtual.

* [NOD-1476] Remove RestoreDiffFromVirtual.

* [NOD-1476] Remove RestorePastUTXOSet.

* [NOD-1476] Add dependency to GHOSTDAGDataStore in ConsensusStateManager.

* [NOD-1476] Rename CalculateAcceptanceDataAndUTXOMultiset to just CalculateAcceptanceData.

* [NOD-1476] Remove UTXODiffManager and add dependencies to AcceptanceManager.

* [NOD-1476] Rename CalculateAcceptanceData to CalculateAcceptanceDataAndMultiset.

* [NOD-1476] Add dependency to DAGTopologyManager from ConsensusStateManager.

* [NOD-1476] Add dependency to BlockStore from ConsensusStateManager.

* [NOD-1476] Add dependency to PruningManager from ConsensusStateManager.

* [NOD-1476] Remove unnecessary stuff from ConsensusStateChanges.

* [NOD-1476] Add dependency to UTXODiffStore from ConsensusStateManager.

* [NOD-1476] Add tips to BlockRelationsStore.

* [NOD-1476] Add dependency to BlockRelationsStore from ConsensusStateManager.

* [NOD-1476] Remove Tips() from ConsensusStateStore.

* [NOD-1476] Remove acceptanceManager.

* [NOD-1476] Remove irrelevant functions out of ConsensusStateManager.
2020-10-25 15:19:20 +02:00
stasatdaglabs
45882343e6
[NOD-1475] Implement stage/discard/commit functionality for data structures (#962)
* [NOD-1475] Add Stage, Discard, and Commit methods to all stores.

* [NOD-1475] Simplify interfaces for processes.

* [NOD-1475] Fix GHOSTDAGManager.

* [NOD-1475] Simplify ChooseSelectedParent.

* [NOD-1475] Remove errors from Stage functions.

* [NOD-1475] Add IsStaged to all data structures.

* [NOD-1475] Remove isDisqualified from CalculateConsensusStateChanges.

* [NOD-1475] Add dependency from ConsensusStateManager to BlockStatusStore.

* [NOD-1475] Fix a comment.

* [NOD-1475] Add ReachabilityReindexRoot to reachabilityDataStore.

* [NOD-1475] Fix a comment.

* [NOD-1475] Rename IsStaged to IsAnythingStaged.
2020-10-21 12:37:22 +03:00
stasatdaglabs
8c63835971
[NOD-1461] Make further design changes (#959)
* [NOD-1461] Split blockValidator and TransactionValidator.

* [NOD-1461] Remove feeDataStore.

* [NOD-1461] Move tips out of ConsensusStateManager and into DAGTopologyManager.

* [NOD-1461] Add UTXODiffManager.

* [NOD-1461] Add RestoreDiffFromVirtual.

* [NOD-1461] Add AcceptanceManager.

* [NOD-1461] Replace SetTips with AddTip.

* [NOD-1461] Fix merge errors.

* [NOD-1461] Rename CoinbaseData to DomainCoinbaseData.
2020-10-20 09:35:58 +03:00
stasatdaglabs
a96a5fd2ef
[NOD-1462] Simplify consensus external API (#958)
* [NOD-1461] Change the external api interface to not having anything besides DomainTransactions and DomainBlocks.

* [NOD-1462] Move external api types to a separate package.

* [NOD-1462] Clarify which model we're using in miningmanager.

* [NOD-1462] Extract coinbase data to its own struct.

* [NOD-1462] Add a comment above CoinbaseData.

* [NOD-1462] Fix the comment above CoinbaseData.
2020-10-19 17:59:04 +03:00