* Improve output of non-critical protocol errors to avoid user panic
* Add log messages at the end of IBD with headers proof
* Found a case where this was falsely triggered due to the wrong equality test
* add p2p v5 which is currently identical to v4
* set all internal imports to v5
* wip
* set default version to 5
* protobuf gen for new ibd chain locator
* wire for new ibd chain locator types
* new ibd shared block algo -- only basic test passing
* address the case where pruning points disagree, now both IBD tests pass
* protobuf gen for new past diff request message
* wire for new request past diff message
* handle and flow for new request past diff message - logic unimplemented yet
* implement ibd sync past diff of relay and selected tip
* go fmt
* remove unused methods
* missed one err check
* addressing simple comments
* apply the traversal limit logic and sort headers
* rename pastdiff -> anticone
* apply Don't relay blocks in virtual anticone #1970 to v5
* go fmt
* Fixed minor comments
* Limit the number of chain negotiation restarts
* Pruning headers p2p basic structure
* Remove headers-first
* Fix consensus tests except TestValidateAndInsertPruningPointWithSideBlocks and TestValidateAndInsertImportedPruningPoint
* Add virtual genesis
* Implement PruningPointAndItsAnticoneWithMetaData
* Start fixing TestValidateAndInsertImportedPruningPoint
* Fix TestValidateAndInsertImportedPruningPoint
* Fix BlockWindow
* Update p2p and gRPC
* Fix all tests except TestHandleRelayInvs
* Delete TestHandleRelayInvs parts that cover the old IBD flow
* Fix lint errors
* Add p2p_request_ibd_blocks.go
* Clean code
* Make MsgBlockWithMetaData implement its own representation
* Remove redundant check if highest share block is below the pruning point
* Fix TestCheckLockTimeVerifyConditionedByAbsoluteTimeWithWrongLockTime
* Fix comments, errors ane names
* Fix window size to the real value
* Check reindex root after each block at TestUpdateReindexRoot
* Remove irrelevant check
* Renames and comments
* Remove redundant argument from sendGetBlockLocator
* Don't delete staging on non-recoverable errors
* Renames and comments
* Remove redundant code
* Commit changes inside ResolveVirtual
* Add comment to IsRecoverableError
* Remove blocksWithMetaDataGHOSTDAGDataStore
* Increase windows pagefile
* Move DeleteStagingConsensus outside of defer
* Get rid of mustAccepted in receiveBlockWithMetaData
* Ban on invalid pruning point
* Rename interface_datastructures_daawindowstore.go to interface_datastructures_blocks_with_meta_data_daa_window_store.go
* * Change GetVirtualSelectedParentChainFromBlockResponseMessage and VirtualSelectedParentChainChangedNotificationMessage to show only added block hashes
* Remove ResolveVirtual
* Use externalapi.ConsensusWrapper inside MiningManager
* Fix pruningmanager.blockwithmetadata
* Set pruning point selected child when importing the pruning point UTXO set
* Change virtual genesis hash
* replace the selected parent with virtual genesis on removePrunedBlocksFromGHOSTDAGData
* Get rid of low hash in block locators
* Remove +1 from everywhere we use difficultyAdjustmentWindowSize and increase the default value by one
* Add comments about consensus wrapper
* Don't use separate staging area when resolving resolveBlockStatus
* Fix netsync stability test
* Fix checkResolveVirtual
* Rename ConsensusWrapper->ConsensusReference
* Get rid of blockHeapNode
* Add comment to defaultDifficultyAdjustmentWindowSize
* Add SelectedChild to DAGTraversalManager
* Remove redundant copy
* Rename blockWindowHeap->calculateBlockWindowHeap
* Move isVirtualGenesisOnlyParent to utils
* Change BlockWithMetaData->BlockWithTrustedData
* Get rid of maxReasonLength
* Split IBD to 100 blocks each time
* Fix a bug in calculateBlockWindowHeap
* Switch to trusted data when encountering virtual genesis in blockWithTrustedData
* Move ConsensusReference to domain
* Update ConsensusReference comment
* Add comment
* Rename shouldNotAddGenesis->skipAddingGenesis
* Make use of maxBlocks instead of maxBlueScoreDifference in antiPastHashesBetween
* Make use of maxBlocks instead of maxBlueScoreDifference in antiPastHashesBetween
* Make use of maxBlocks instead of maxBlueScoreDifference in antiPastHashesBetween
* Make use of maxBlocks instead of maxBlueScoreDifference in antiPastHashesBetween
* Make use of maxBlocks instead of maxBlueScoreDifference in antiPastHashesBetween
Co-authored-by: Ori Newman <orinewman1@gmail.com>
* Change mergeSet to be ordered topologically.
* Add special condition for genesis.
* Add check that the coinbase is validated.
* Change names of variables(old: chainHash, blueHash).
* Fix the DAG diagram in the comment above the function.
* Fix variables names.
Co-authored-by: tal <tal@daglabs.com>
Co-authored-by: Ori Newman <orinewman1@gmail.com>
* Add StagingArea struct
* Implemented staging areas in blockStore
* Move blockStagingShard to separate folder
* Apply staging shard to acceptanceDataStore
* Update blockHeaderStore with StagingArea
* Add StagingArea to BlockRelationStore
* Add StagingArea to blockStatusStore
* Add StagingArea to consensusStateStore
* Add StagingArea to daaBlocksStore
* Add StagingArea to finalityStore
* Add StagingArea to ghostdagDataStore
* Add StagingArea to headersSelectedChainStore and headersSelectedTipStore
* Add StagingArea to multisetStore
* Add StagingArea to pruningStore
* Add StagingArea to reachabilityDataStore
* Add StagingArea to utxoDiffStore
* Fix forgotten compilation error
* Update reachability manager and some more things with StagingArea
* Add StagingArea to dagTopologyManager, and some more
* Add StagingArea to GHOSTDAGManager, and some more
* Add StagingArea to difficultyManager, and some more
* Add StagingArea to dagTraversalManager, and some more
* Add StagingArea to headerTipsManager, and some more
* Add StagingArea to constnsusStateManager, pastMedianTimeManager
* Add StagingArea to transactionValidator
* Add StagingArea to finalityManager
* Add StagingArea to mergeDepthManager
* Add StagingArea to pruningManager
* Add StagingArea to rest of ValidateAndInsertBlock
* Add StagingArea to blockValidator
* Add StagingArea to coinbaseManager
* Add StagingArea to syncManager
* Add StagingArea to blockBuilder
* Update consensus with StagingArea
* Add StagingArea to ghostdag2
* Fix remaining compilation errors
* Update names of stagingShards
* Fix forgotten stagingArea passing
* Mark stagingShard.isCommited = true once commited
* Move isStaged to stagingShard, so that it's available without going through store
* Make blockHeaderStore count be avilable from stagingShard
* Fix remaining forgotten stagingArea passing
* commitAllChanges should call dbTx.Commit in the end
* Fix all tests tests in blockValidator
* Fix all tests in consensusStateManager and some more
* Fix all tests in pruningManager
* Add many missing stagingAreas in tests
* Fix many tests
* Fix most of all other tests
* Fix ghostdag_test.go
* Add comment to StagingArea
* Make list of StagingShards an array
* Add comment to StagingShardID
* Make sure all staging shards are pointer-receiver
* Undo bucket rename in block_store
* Typo: isCommited -> isCommitted
* Add comment explaining why stagingArea.shards is an array
* Remove Subsystems map and replace with RegisterSubSystem
* Clean up the logger
* Fix LOGFLAGS and make LongFile work correctly
* Parallelize the logger backend
* More logger cleanup
* Initialize and close the logger backend wherever it's needed
* Move the location where the backend is closed, also print the log if it panics while writing
* Add TestMain to reachability manager tests to preserve the same log level
* Fix review comments
Co-authored-by: Svarog <feanorr@gmail.com>
* Add Close() function to all the iterators
* Add defer iterator.Close() whenever we open an iterator
* Add isClosed to all iterators and panic/return error if used after closing
Co-authored-by: Svarog <feanorr@gmail.com>
* Make antiPastHashesBetween return blocks sorted in ghostdag-order
* Return sortedMergeSet instead of blueMergeSet
* Invert the order of parameters of IsAncestorOf
* Add RenderDAGToDot to TestConsensus
* Add HighHash explicitly, unless lowHash == highHash
* Use Equal instead of == when comparing hashes
* Fixed TestSyncManager_GetHashesBetween
* Fix tests
* findHighHashAccordingToMaxBlueScoreDifference: don't start looking if the whole thing fits
* Handle a missed error
* Remove redundant call to RenderToDot
* Fix bug in findHighHashAccordingToMaxBlueScoreDifference
* Don't error out if antiPastHashesBetween have 2 blocks with the same blue score
* Prepend lowHash to RPC GetBlocks request
* Add a test for GetHashesBetween
* Add a test for GetBlocks RPC call
* Update antipast.go
Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
* Remove BlockHexes from GetBlocks request and response
* Add GetBlocks RPC
* Include the selectedTip's anticone in GetBlocks
* Add Anticone to fakeRelayInvsContext
* Include verbose data only if it was requested + Add comments to HandleGetBlocks
* Allow antiPastHashesBetween to receive unrelated low and high hashes
* Convert to/from protowire GetBlocksResponse with no verbose data correctly
* Removed NextLowHash
* Update GetBlocks in rpc_client
* Validate in consensus.Anticone that blockHash exists
Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
* Split deletePastBlocks into sub-routines
* Remove SelectedParentIterator, and refactor SelectedChildIterator to support First and Error
* Implement PruneAllBlocks
* Prune all blocks in the store
* Prune only blocks that are below the pruning point
* Defer call onEnd of LogAndMeasureExecutionTime
* Handle a forgotten error
* Minor style fixes
* Allow GetMissingBlockBodyHashes return an empty list if the missing blocks were requested before IBD start
* Add link to issue in comment about error to be fixed
* Prevent a race condition in findHighestSharedBlockHash where we get headersSelectedTip and then pass it as highHash to GetBlockLocator, without locking consensus
* Restart findHighestSharedBlockHash if lowHash or highHash are no longer in selectedParentChain
* Test for specifically ErrBlockNotInSelectedParentChain instead of database NotFound error
* Fix TestCreateHeadersSelectedChainBlockLocator
Co-authored-by: Ori Newman <orinewman1@gmail.com>
* Add selected chain store and optimize block locator with it
* Fix build error
* Fix comments
* Fix IsStaged
* Rename CalculateSelectedParentChainChanges to CalculateChainPath and SelectedParentChainChanges->SelectedChainPath
* Use binary.LittleEndian directly to allow compiler optimizations
* Remove boolean from HeadersSelectedChainStore interface
* Prevent endless loop in block locator
* Make sure block locator doesn't include a hash lower than the lowHash in
the block locator
* Use finalityStore to optimize LowestChainBlockAboveOrEqualToBlueScore
* Remove the limit on the amount of hashes returned from antiPastHashesBetween.
* Guard against requests with a non-existing block hash.
* Move missing-block-hash guards to consensus.go.
* Ban a peer that doesn't send us all the requested headers during IBD.
* Extract blockHeap.ToSlice.
* Re-request headers in requestHeaders if we didn't receive the highHash.
* 1) Calculate pruning point incrementally
2) Add IsValidPruningPoint to pruning manager and consensus
3) Use reachability children for selected child iterator
* Add request IBD root hash flow
* Fix UpdatePruningPointByVirtual and IsValidPruningPoint
* Regenerate messages.pb.go
* Make the pruning point the earliest chain block with finality interval higher than the previous pruning point
* Fix merge errors
* [NOD-1575] Implement Clone and Equal for all model types
* [NOD-1575] Add assertion for transaction ID equality
* [NOD-1575] Use DomainTransaction.Equal to compare to expected coinbase transaction
* [NOD-1575] Add TestDomainBlockHeader_Clone
* [NOD-1575] Don't clone nil values
* [NOD-1575] Add type assertions
* [NOD-1575] Don't clone nil values
* [NOD-1575] Add missing Equals
* [NOD-1575] Add length checks
* [NOD-1575] Update comment
* [NOD-1575] Check length for TransactionAcceptanceData
* [NOD-1575] Explicitly clone nils where needed
* [NOD-1575] Clone tx id
* [NOD-1575] Flip condition
* Nod 1576 make coverage tests for equal clone inside model externalapi (#1177)
* [NOD-1576] Make coverage tests for equal and clone inside model and externalapi
* Some formatting and naming fixes
* Made transactionToCompare type exported
* Added some tests and made some changes to the tests code
* No changes made
* Some formatting and naming changes made
* Made better test coverage for externalapi clone and equal functions
* Changed expected result for two cases
* Added equal and clone functions tests for ghostdag and utxodiff
* Added tests
* [NOD-1576] Implement reachabilitydata equal/clone unit tests
* [NOD-1576] Full coverage of reachabilitydata equal/clone unit tests
* Made changes and handling panic to transaction_equal_clone_test.go and formating of utxodiff_equal_clone_test.go
* Added recoverForEqual2 for handling panic to transaction_equal_clone_test.go
* [NOD-1576] Full coverage of transaction equal unit test
* [NOD-1576] Add expects panic
* [NOD-1576] Allow composites in go vet
* [NOD-1576] Code review fixes (#1223)
* [NOD-1576] Code review fixes
* [NOD-1576] Code review fixes part 2
* [NOD-1576] Fix wrong name
Co-authored-by: karim1king <karimkaspersky@yahoo.com>
Co-authored-by: Ori Newman <orinewman1@gmail.com>
Co-authored-by: Karim <karim1king@users.noreply.github.com>
* Fix merge errors
* Use Equal where possible
* Use Equal where possible
* Use Equal where possible
Co-authored-by: andrey-hash <74914043+andrey-hash@users.noreply.github.com>
Co-authored-by: karim1king <karimkaspersky@yahoo.com>
Co-authored-by: Karim <karim1king@users.noreply.github.com>
* Get rid of insertMode
* Rename AddBlockToVirtual->AddBlock
* When F is not in the future of P, enforce finality with P and not with F.
* Don't allow blocks with invalid parents or with missing block body
* Check finality violation before checking block status
* Implement CalculateIndependentPruningPoint
* Move checkBlockStatus to validateBlock
* Add ValidateBlock to block processor interface
* Adjust SetPruningPoint to the new IBD flow
* Add pruning store to CSM's constructor
* Flip wrong condition on AddHeaderTip
* Fix func (hts *headerSelectedTipStore) Has
* Fix block stage order
* Call to ValidateBodyInContext from validatePostProofOfWork
* Enable overrideDAGParams
* Update log
* Rename SetPruningPoint to ValidateAndInsertPruningPoint and move most of its logic inside block processor
* Rename hasValidatedHeader->hasValidatedOnlyHeader
* Fix typo
* Name return values for fetchMissingUTXOSet
* Add comment
* Return ErrMissingParents when block body is missing
* Add logs and comments
* Fix merge error
* Fix pruning point calculation to be by virtual selected parent
* Replace CalculateIndependentPruningPoint to CalculatePruningPointByHeaderSelectedTip
* Fix isAwaitingUTXOSet to check pruning point by headers
* Change isAwaitingUTXOSet indication
* Remove IsBlockInHeaderPruningPointFuture from BlockInfo
* Fix LowestChainBlockAboveOrEqualToBlueScore
* Add validateNewPruningPointTransactions
* Add validateNewPruningAgainstPastUTXO
* Rename set_pruning_utxo_set.go to update_pruning_utxo_set.go
* Check missing block body hashes by missing block instead of status
* Validate pruning point against past UTXO with the pruning point as block hash
* Remove virtualHeaderHash
* Fix comment
* Fix imports
* [NOD-1579] Remove selected tip hash messages.
* [NOD-1579] Start moving IBD stuff into blockrelay.
* [NOD-1579] Rename relaytransactions to transactionrelay.
* [NOD-1579] Move IBD files into blockrelay.
* [NOD-1579] Remove flow stuff from ibd.go.
* [NOD-1579] Bring back IsInIBD().
* [NOD-1579] Simplify block relay flow.
* [NOD-1579] Check orphan pool for missing parents to avoid unnecessary processing.
* [NOD-1579] Implement processOrphan.
* [NOD-1579] Implement addToOrphanSetAndRequestMissingParents.
* [NOD-1579] Fix TestIBD.
* [NOD-1579] Implement isBlockInOrphanResolutionRange.
* [NOD-1579] Implement limited block locators.
* [NOD-1579] Add some comments.
* [NOD-1579] Specifically check for StatusHeaderOnly in blockrelay.
* [NOD-1579] Simplify runIBDIfNotRunning.
* [NOD-1579] Don't run IBD if it is already running.
* [NOD-1579] Fix a comment.
* [NOD-1579] Rename mode to syncInfo.
* [NOD-1579] Simplify validateAndInsertBlock.
* [NOD-1579] Fix bad SyncStateSynced condition.
* [NOD-1579] Implement validateAgainstSyncStateAndResolveInsertMode.
* [NOD-1579] Use insertModeHeader.
* [NOD-1579] Add logs to TrySetIBDRunning and UnsetIBDRunning.
* [NOD-1579] Implement and use dequeueIncomingMessageAndSkipInvs.
* [NOD-1579] Fix a log.
* [NOD-1579] Fix a bug in createBlockLocator.
* [NOD-1579] Rename a variable.
* [NOD-1579] Fix a slew of bugs in missingBlockBodyHashes and selectedChildIterator.
* [NOD-1579] Fix bad chunk size in syncMissingBlockBodies.
* [NOD-1579] Remove maxOrphanBlueScoreDiff.
* [NOD-1579] Fix merge errors.
* [NOD-1579] Remove a debug log.
* [NOD-1579] Add logs.
* [NOD-1579] Make various go quality tools happy.
* [NOD-1579] Fix a typo in a variable name.
* [NOD-1579] Fix full blocks over header-only blocks not failing the missing-parents validation.
* [NOD-1579] Add an error log about a condition that should never happen.
* [NOD-1579] Check all antiPast hashes instead of just the lowHash's anticone to filter for header-only blocks.
* [NOD-1579] Remove the nil stuff from GetBlockLocator.
* [NOD-1579] Remove superfluous condition in handleRelayInvsFlow.start().
* [NOD-1579] Return a boolean from requestBlock instead of comparing to nil.
* [NOD-1579] Fix a bad log.Debugf.
* [NOD-1579] Remove a redundant check.
* [NOD-1579] Change an info log to a warning log.
* [NOD-1579] Move OnNewBlock out of relayBlock.
* [NOD-1579] Remove redundant exists check from runIBDIfNotRunning.
* [NOD-1579] Fix bad call to OnNewBlock.
* [NOD-1579] Remove an impossible check.
* [NOD-1579] Added a log.
* [NOD-1579] Rename insertModeBlockWithoutUpdatingVirtual to insertModeBlockBody.
* [NOD-1579] Add a check for duplicate headers.
* [NOD-1579] Added a comment.
* [NOD-1579] Tighten a stop condition.
* [NOD-1579] Simplify a log.
* [NOD-1579] Clarify a log.
* [NOD-1579] Move a log.
* Revert "[NOD-1500] Delete integration tests"
This reverts commit fcb57a206690a884fa6afb69d5d493282954a8bf.
* [NOD-1518] hashserialization -> consenusserialization
* [NOD-1518] Fix add genesis to virtual
* [NOD-1518] Fix a bug in SerializeCoinbasePayload.
* [NOD-1518] Fix a loop error and make pastMedianTime behave correctly everywhere on genesis.
* [NOD-1518] Fix another bug and an infinite loop.
* [NOD-1518] Fix uninitialized slice.
* [NOD-1518] Fix bad should-commit checks and another infinite loop.
* [NOD-1518] Fix nil serialization.
* [NOD-1518] Rename blockHash to currentBlockHash.
* [NOD-1518] Move the check whether stagedVirtualUTXOSet != nil to the top of commitVirtualUTXODiff.
* [NOD-1518] Simplify utxoDiffStore.Commit.
* [NOD-1518] Unextract resolveBlockStatusAndCheckFinality.
* [NOD-1518] Move no-transactions logic into CalculateIDMerkleRoot.
* [NOD-1518] Remove redundant is-staged check.
* [NOD-1518] Fix merge errors.
* [NOD-1518] Don't write anything if utxoDiffChild is nil.
* [NOD-1518] Stage virtualAcceptanceData and virtualMultiset.
* [NOD-1518] Fix bugs in getBlockTemplate and submitBlock.
* [NOD-1518] Fix bad validation order in validateHeaderInContext.
* [NOD-1518] Fix bug in Next().
* [NOD-1518] Fix nil dereference of subnetworks in AddressCache.
* [NOD-1518] Fix multisetStore.Get returning a pointer to a multiset that is changed in place.
* [NOD-1518] Break on genesis in countSubtrees.
* [NOD-1518] Fix createBlockLocator.
* [NOD-1518] Fix MsgTxToDomainTransaction.
* [NOD-1518] Set MaxTxVersion to 1.
* [NOD-1518] Fix missing error handling, bug in MsgTxToDomainTransaction, and bad subnetwork equality check.
* [NOD-1518] Fix bug in hasUTXOByOutpointFromStagedVirtualUTXODiff.
* [NOD-1518] Remove irrelevant comments.
* [NOD-1518] Generate transactions with sufficient fee in tx_relay_test.
* [NOD-1518] Fix broken RPC handlers.
* [NOD-1518] Fix merge errors.
* [NOD-1518] Fix bad exists check in restorePastUTXO and missing genesis check in CalculatePastUTXOAndAcceptanceData.
* [NOD-1518] Add a comment.
* [NOD-1518] Use a regular mutex instead of a read-write mutex in consensus to avoid dealing with sneaky not-actually-read functions.
* [NOD-1518] Fix a deadlock in GetVirtualSelectedParent.
* [NOD-1518] Fix missing handler registration for CmdHeader.
* [NOD-1518] Fix processHeader calling OnNewBlock and LogBlock. Also fix conversion errors in IBDRootUTXOSetAndBlock.
* [NOD-1518] Fix bad Command() in MsgIBDRootUTXOSetAndBlock.
* [NOD-1518] Fix bad SyncStateMissingUTXOSet logic in resolveSyncState.
* [NOD-1518] Rename mode to syncState.
* [NOD-1518] Fix headers-only blocks coming in after the consensus thinks it's synced.
* [NOD-1518] Fix selectedChildIterator.Next not ignoring virtual, infinite loop in HashSet.Length().
* [NOD-1518] Fix not-properly wrapped IBD blocks.
* [NOD-1518] Fix bad conversion in RequestIBDBlocks.
* [NOD-1518] Fix bad string for CmdRequestHeaders.
* [NOD-1518] Fix bad string for CmdDoneHeaders.
* [NOD-1518] Fix bad Command() for MsgIBDRootNotFound.
* [NOD-1518] Fix bad areHeaderTipsSyncedMaxTimeDifference value.
* [NOD-1518] Add missing string for CmdRequestIBDBlocks.
* [NOD-1518] Fix bad check for SyncStateMissingBlockBodies.
* [NOD-1518] Fix bad timeout durations in tests.
* [NOD-1518] Fix IBD blocks not calling OnNewBlock.
* [NOD-1518] Change when IBD finishes.
* [NOD-1518] Properly clone utxoDiffChild.
* [NOD-1518] Fix merge errors.
* [NOD-1518] Move call to LogBlock to into OnNewBlock.
* [NOD-1518] Return "not implemented" in unimplemented RPC handlers.
* [NOD-1518] Extract cloning of hashes to a method over DomainHash.
* [NOD-1518] Use isHeaderOnlyBlock.
* [NOD-1518] Use constants.TransactionVersion.
* [NOD-1518] Break immediately if we reached the virtual in SelectedChildIterator.
* [NOD-1518] Don't stage nil utxoDiffChild.
* [NOD-1518] Properly check the genesis hash in CalculatePastUTXOAndAcceptanceData.
* [NOD-1518] Explain why we break on current == nil in countSubtrees.
* [NOD-1518] Add a comment explaining why we check against StatusValid in resolveSyncState.
Co-authored-by: Mike Zak <feanorr@gmail.com>
Co-authored-by: Ori Newman <orinewman1@gmail.com>
