9 Commits

Author SHA1 Message Date
stasatdaglabs
756f40c59a
Sync pruning point UTXO sets incrementally instead of all at once (#1431)
* Replaced the content of MsgIBDRootUTXOSetChunk with pairs of outpoint-utxo entry pairs.

* Rename utxoIter to utxoIterator.

* Add a big stinky TODO on an assert.

* Replace pruningStore staging with a UTXO set iterator.

* Reimplement receiveAndInsertIBDRootUTXOSet.

* Extract OutpointAndUTXOEntryPairsToDomainOutpointAndUTXOEntryPairs into domainconverters.go.

* Pass the outpoint and utxy entry pairs to the pruning store.

* Implement InsertCandidatePruningPointUTXOs.

* Implement ClearCandidatePruningPointUTXOs.

* Implement UpdateCandidatePruningPointMultiset.

* Use the candidate pruning point multiset in updatePruningPoint.

* Implement CandidatePruningPointUTXOIterator.

* Use the pruning point utxo set iterator for StageVirtualUTXOSet.

* Defer ClearCandidatePruningPointUTXOs.

* Implement OverwriteVirtualUTXOSet.

* Implement CommitCandidatePruningPointUTXOSet.

* Implement BeginOverwritingVirtualUTXOSet and FinishOverwritingVirtualUTXOSet.

* Implement overwriteVirtualUTXOSetAndCommitPruningPointUTXOSet.

* Rename ClearCandidatePruningPointUTXOs to ClearCandidatePruningPointData.

* Add missing methods to dbManager.

* Implement PruningPointUTXOs.

* Implement RecoverUTXOIfRequired.

* Delete the utxoserialization package.

* Fix compilation errors in TestValidateAndInsertPruningPoint.

* Switch order of operations in the if statements in PruningPointUTXOs so that Next() wouldn't be unnecessarily called.

* Fix missing pruning point utxo set staging and bad slice length.

* Fix no default multiset in InsertCandidatePruningPointUTXOs.

* Make go vet happy.

* Rename candidateXXX to importedXXX.

* Do some more renaming.

* Rename some more.

* Fix bad MsgIBDRootNotFound logic.

* Fix an error message.

* Simplify receiveIBDRootBlock.

* Fix error message in receiveAndInsertIBDRootUTXOSet.

* Do some more renaming.

* Fix merge errors.

* Fix a bug caused by calling iterator.First() unnecessarily.

* Remove databaseContext from stores and don't use a transaction in ClearXXX functions.

* Simplify receiveAndInsertIBDRootUTXOSet.

* Fix offset count in PruningPointUTXOs().

* Fix readOnlyUTXOIteratorWithDiff.First().

* Split handleRequestIBDRootUTXOSetAndBlockFlow into smaller methods.

* Rename IbdRootNotFound to UnexpectedPruningPoint.

* Rename requestIBDRootHash to requestPruningPointHash.

* Rename IBDRootHash to PruningPointHash.

* Rename RequestIBDRootUTXOSetAndBlock to RequestPruningPointUTXOSetAndBlock.

* Rename IBDRootUTXOSetChunk to PruningPointUTXOSetChunk.

* Rename RequestNextIBDRootUTXOSetChunk to RequestNextPruningPointUTXOSetChunk.

* Rename DoneIBDRootUTXOSetChunks to DonePruningPointUTXOSetChunks.

* Rename remaining references to IBD root.

* Fix an error message.

* Add a check for HadStartedImportingPruningPointUTXOSet in commitVirtualUTXODiff.

* Add a check for HadStartedImportingPruningPointUTXOSet in ImportPruningPointUTXOSetIntoVirtualUTXOSet.

* Move FinishImportingPruningPointUTXOSet closer to HadStartedImportingPruningPointUTXOSet.

* Remove reference to pruningStore in utxoSetIterator.

* Pointerify utxoSetIterator receivers.

* Fix bad insert in CommitImportedPruningPointUTXOSet.

* Rename commitImportedPruningPointUTXOSetAll to applyImportedPruningPointUTXOSet.

* Simplify PruningPointUTXOs.

* Add populateTransactionWithUTXOEntriesFromUTXOSet.

* Fix a TODO comment.

* Rename InsertImportedPruningPointUTXOs to AppendImportedPruningPointUTXOs.

* Extract handleRequestPruningPointUTXOSetAndBlockMessage to a separate method.

* Rename stuff in readOnlyUTXOIteratorWithDiff.First().

* Address toAddIterator in readOnlyUTXOIteratorWithDiff.First().

* Call First() before any full iteration on ReadOnlyUTXOSetIterator.

* Call First() before any full iteration on a database Cursor.

* Put StartImportingPruningPointUTXOSet inside the pruning point transaction.

* Make serializeOutpoint and serializeUTXOEntry free functions in pruningStore.

* Fix readOnlyUTXOIteratorWithDiff.First().

* Fix bad validations in importPruningPoint.

* Remove superfluous call to validateBlockTransactionsAgainstPastUTXO.
2021-01-21 17:24:52 +02:00
talelbaz
8a309a7d2a
Upgradability mechanisms script version (#1313)
* ''

* ''

* ''

* Changes genesis block version to 0.

* a

* a

* All tests are done.

* All tests passed for changed block version from int32 to uint16

* Adds validation of rejecting blocks with unknown versions.

* Changes txn version from int32 to uint16.

* .

* Adds comments to exported functions.

* Change functions name from ConvertFromRpcScriptPubKeyToRPCScriptPubKey to ConvertFromAppMsgRPCScriptPubKeyToRPCScriptPubKey and from ConvertFromRPCScriptPubKeyToRpcScriptPubKey to ConvertFromRPCScriptPubKeyToAppMsgRPCScriptPubKey

* change comment to "ScriptPublicKey represents a Kaspad ScriptPublicKey"

* delete part (tx.Version < 0) that cannot be exist on the if statement.

* Revert protobuf version.

* Fix a comment.

* Fix a comment.

* Rename a variable.

* Rename a variable.

* Remove a const.

* Rename a type.

* Rename a field.

* Rename a field.

* Remove commented-out code.

* Remove dangerous nil case in DomainTransactionOutput.Clone().

* Remove a constant.

* Fix a string.

* Fix wrong totalScriptPubKeySize in transactionMassStandalonePart.

* Remove a constant.

* Remove an unused error.

* Fix a serialization error.

* Specify version types to be uint16 explicitly.

* Use constants.ScriptPublicKeyVersion.

* Fix a bad test.

* Remove some whitespace.

* Add a case to utxoEntry.Equal().

* Rename scriptPubKey to scriptPublicKey.

* Remove a TODO.

* Rename constants.

* Rename a variable.

* Add version to parseShortForm.

Co-authored-by: tal <tal@daglabs.com>
Co-authored-by: stasatdaglabs <stas@daglabs.com>
2021-01-05 17:50:09 +02:00
Ori Newman
49b6cc6038
Add mutable and immutable header interfaces (#1305)
* Add mutable and immutable header interfaces

* Fix ShouldMine()

* Remove false comment

* Fix Equal signature

* Fix Equal implementation
2020-12-29 13:55:17 +02:00
Svarog
05941a76e7
Make DomainHash and TransactionID read-only structs (#1282)
* Increase size of reachability cache

* Change DomainHash to struct with unexported hashArray

* Fixing compilation errors stemming from new DomainHash structure

* Remove obsolete Read/WriteElement methods in appmessage

* Fix all tests

* Fix all tests

* Add comments

* A few renamings

* go mod tidy
2020-12-24 16:15:23 +02:00
stasatdaglabs
053bb351b5
[NOD-1597] Implement a UTXO index (#1221)
* [NOD-1579] Rename AcceptedTxIDs to AcceptedTransactionIDs.

* [NOD-1579] Add InsertBlockResult to ValidateAndInsertBlock results.

* [NOD-1593] Rename InsertBlockResult to BlockInsertionResult.

* [NOD-1593] Add SelectedParentChainChanges to AddBlockToVirtual's result.

* [NOD-1593] Implement findSelectedParentChainChanges.

* [NOD-1593] Implement TestFindSelectedParentChainChanges.

* [NOD-1593] Fix a string.

* [NOD-1593] Finish implementing TestFindSelectedParentChainChanges.

* [NOD-1593] Fix merge errors.

* [NOD-1597] Begin implementing UTXOIndex.

* [NOD-1597] Connect UTXOIndex to RPC.

* [NOD-1597] Connect Consensus to UTXOIndex.

* [NOD-1597] Add AcceptanceData to BlockInfo.

* [NOD-1597] Implement UTXOIndex.Update().

* [NOD-1597] Implement add(), remove(), and discard() in utxoIndexStore.

* [NOD-1597] Add error cases to add() and remove().

* [NOD-1597] Add special cases to add() and remove().

* [NOD-1597] Implement commit.

* [NOD-1597] Add a mutex around UTXOIndex.Update().

* [NOD-1597] Return changes to the UTXO from Update().

* [NOD-1597] Add NotifyUTXOsChangedRequestMessage and related structs.

* [NOD-1597] Implement HandleNotifyUTXOsChanged.

* [NOD-1597] Begin implementing TestUTXOIndex.

* [NOD-1597] Implement RegisterForUTXOsChangedNotifications.

* [NOD-1597] Fix bad transaction.ID usage.

* [NOD-1597] Implement convertUTXOChangesToUTXOsChangedNotification.

* [NOD-1597] Make UTXOsChangedNotificationMessage.Removed UTXOsByAddressesEntry instead of just RPCOutpoint so that the client can discern which address was the UTXO removed for.

* [NOD-1597] Collect outpoints in TestUTXOIndex.

* [NOD-1597] Rename RPC stuff.

* [NOD-1597] Add messages for GetUTXOsByAddresses.

* [NOD-1597] Implement HandleGetUTXOsByAddresses.

* [NOD-1597] Implement GetUTXOsByAddresses.

* [NOD-1597] Implement UTXOs().

* [NOD-1597] Implement getUTXOOutpointEntryPairs().

* [NOD-1597] Expand TestUTXOIndex.

* [NOD-1597] Convert SubmitTransaction to use RPCTransaction instead of MsgTx.

* [NOD-1597] Finish implementing TestUTXOIndex.

* [NOD-1597] Add messages for GetVirtualSelectedParentBlueScore.

* [NOD-1597] Implement HandleGetVirtualSelectedParentBlueScore and GetVirtualSelectedParentBlueScore.

* [NOD-1597] Implement TestVirtualSelectedParentBlueScore.

* [NOD-1597] Implement NotifyVirtualSelectedParentBlueScoreChanged.

* [NOD-1597] Expand TestVirtualSelectedParentBlueScore.

* [NOD-1597] Implement notifyVirtualSelectedParentBlueScoreChanged.

* [NOD-1597] Make go lint happy.

* [NOD-1593] Fix merge errors.

* [NOD-1593] Rename findSelectedParentChainChanges to calculateSelectedParentChainChanges.

* [NOD-1593] Expand TestCalculateSelectedParentChainChanges.

* [NOD-1597] Add logs to utxoindex.go.

* [NOD-1597] Add logs to utxoindex/store.go.

* [NOD-1597] Add logs to RPCManager.NotifyXXX functions.

* [NOD-1597] Ignore transactions that aren't accepted.

* [NOD-1597] Use GetBlockAcceptanceData instead of GetBlockInfo.

* [NOD-1597] Convert scriptPublicKey to string directly, instead of using hex.

* [NOD-1597] Add a comment.

* [NOD-1597] Guard against calling utxoindex methods when utxoindex is turned off.

* [NOD-1597] Add lock to UTXOs.

* [NOD-1597] Guard against calls to getUTXOOutpointEntryPairs when staging isn't empty.
2020-12-20 17:24:56 +02:00
stasatdaglabs
eef5e3768c
[NOD-1518] Fix genesis block insertion and integration tests (#1013)
* Revert "[NOD-1500] Delete integration tests"

This reverts commit fcb57a206690a884fa6afb69d5d493282954a8bf.

* [NOD-1518] hashserialization -> consenusserialization

* [NOD-1518] Fix add genesis to virtual

* [NOD-1518] Fix a bug in SerializeCoinbasePayload.

* [NOD-1518] Fix a loop error and make pastMedianTime behave correctly everywhere on genesis.

* [NOD-1518] Fix another bug and an infinite loop.

* [NOD-1518] Fix uninitialized slice.

* [NOD-1518] Fix bad should-commit checks and another infinite loop.

* [NOD-1518] Fix nil serialization.

* [NOD-1518] Rename blockHash to currentBlockHash.

* [NOD-1518] Move the check whether stagedVirtualUTXOSet != nil to the top of commitVirtualUTXODiff.

* [NOD-1518] Simplify utxoDiffStore.Commit.

* [NOD-1518] Unextract resolveBlockStatusAndCheckFinality.

* [NOD-1518] Move no-transactions logic into CalculateIDMerkleRoot.

* [NOD-1518] Remove redundant is-staged check.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Don't write anything if utxoDiffChild is nil.

* [NOD-1518] Stage virtualAcceptanceData and virtualMultiset.

* [NOD-1518] Fix bugs in getBlockTemplate and submitBlock.

* [NOD-1518] Fix bad validation order in validateHeaderInContext.

* [NOD-1518] Fix bug in Next().

* [NOD-1518] Fix nil dereference of subnetworks in AddressCache.

* [NOD-1518] Fix multisetStore.Get returning a pointer to a multiset that is changed in place.

* [NOD-1518] Break on genesis in countSubtrees.

* [NOD-1518] Fix createBlockLocator.

* [NOD-1518] Fix MsgTxToDomainTransaction.

* [NOD-1518] Set MaxTxVersion to 1.

* [NOD-1518] Fix missing error handling, bug in MsgTxToDomainTransaction, and bad subnetwork equality check.

* [NOD-1518] Fix bug in hasUTXOByOutpointFromStagedVirtualUTXODiff.

* [NOD-1518] Remove irrelevant comments.

* [NOD-1518] Generate transactions with sufficient fee in tx_relay_test.

* [NOD-1518] Fix broken RPC handlers.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Fix bad exists check in restorePastUTXO and missing genesis check in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Add a comment.

* [NOD-1518] Use a regular mutex instead of a read-write mutex in consensus to avoid dealing with sneaky not-actually-read functions.

* [NOD-1518] Fix a deadlock in GetVirtualSelectedParent.

* [NOD-1518] Fix missing handler registration for CmdHeader.

* [NOD-1518] Fix processHeader calling OnNewBlock and LogBlock. Also fix conversion errors in IBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad Command() in MsgIBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad SyncStateMissingUTXOSet logic in resolveSyncState.

* [NOD-1518] Rename mode to syncState.

* [NOD-1518] Fix headers-only blocks coming in after the consensus thinks it's synced.

* [NOD-1518] Fix selectedChildIterator.Next not ignoring virtual, infinite loop in HashSet.Length().

* [NOD-1518] Fix not-properly wrapped IBD blocks.

* [NOD-1518] Fix bad conversion in RequestIBDBlocks.

* [NOD-1518] Fix bad string for CmdRequestHeaders.

* [NOD-1518] Fix bad string for CmdDoneHeaders.

* [NOD-1518] Fix bad Command() for MsgIBDRootNotFound.

* [NOD-1518] Fix bad areHeaderTipsSyncedMaxTimeDifference value.

* [NOD-1518] Add missing string for CmdRequestIBDBlocks.

* [NOD-1518] Fix bad check for SyncStateMissingBlockBodies.

* [NOD-1518] Fix bad timeout durations in tests.

* [NOD-1518] Fix IBD blocks not calling OnNewBlock.

* [NOD-1518] Change when IBD finishes.

* [NOD-1518] Properly clone utxoDiffChild.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Move call to LogBlock to into OnNewBlock.

* [NOD-1518] Return "not implemented" in unimplemented RPC handlers.

* [NOD-1518] Extract cloning of hashes to a method over DomainHash.

* [NOD-1518] Use isHeaderOnlyBlock.

* [NOD-1518] Use constants.TransactionVersion.

* [NOD-1518] Break immediately if we reached the virtual in SelectedChildIterator.

* [NOD-1518] Don't stage nil utxoDiffChild.

* [NOD-1518] Properly check the genesis hash in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Explain why we break on current == nil in countSubtrees.

* [NOD-1518] Add a comment explaining why we check against StatusValid in resolveSyncState.

Co-authored-by: Mike Zak <feanorr@gmail.com>
Co-authored-by: Ori Newman <orinewman1@gmail.com>
2020-11-12 15:19:39 +02:00
Svarog
37fbdcb453
[NOD-1526] Restore txscript tests (#1019)
* [NOD-1526] Fix compilation errors

* [NOD-1526] Make MsgTx.PayloadHash non-pointer

* [NOD-1526] Fixed many tests

* [NOD-1526] Fix reference_test.go

* [NOD-1526] Removed last instances of appmessage in consensus

* [NOD-1526] No need to check for subnetwork
2020-11-12 10:22:17 +02:00
Ori Newman
23c1ea6c31
[NOD-1525] Implement headers first ibd (#1017)
* [NOD-1525] Implement headers first IBD

* [NOD-1525] Fix proto translators

* [NOD-1525] Register missing flows

* [NOD-1525] Rename SyncStateNormal->SyncStateRelay, simplifiy IBD peer selection code and get rid of panic in FinishIBD

* [NOD-1525] Remove redundant methods from interface
2020-11-10 16:14:51 +02:00
Svarog
281944762d
[NOD-1500] Glue between domain and application (#1007)
* [NOD-1500] Added Domain type and Constructor

* [NOD-1500] Replaced dag+txpool with domain in flowContext

* [NOD-1500] Replaced dag+txpool with domain in flowContext

* [NOD-1500] Converters: domain objects from/to appmessage

* [NOD-1500] Convert hashes to DomainHashes in appmessages

* [NOD-1500] Remove references to daghash in dagconfig

* [NOD-1500] Fixed all appmessage usages of hashes

* [NOD-1500] Update all RPC to use domain

* [NOD-1500] Big chunk of protocol flows re-wired to domain

* [NOD-1500] Finished re-wiring all protocol flows to new Domain

* [NOD-1500] Fix some mempool and kaspaminer compilation errors

* [NOD-1500] Deleted util/{block,tx,daghash} and dbaccess

* [NOD-1500] util.CoinbaseTransactionIndex -> transactionhelper.CoinbaseTransactionIndex

* [NOD-1500] Fix txsigner

* [NOD-1500] Removed all references to util/subnetworkid

* [NOD-1500] Update RpcGetBlock related messages

* [NOD-1500] Many more compilation fixes

* [NOD-1500] Return full list of missing blocks for orphan resolution

* [NOD-1500] Fixed handshake

* [NOD-1500] Fixed flowcontext compilation

* [NOD-1500] Update users of StartIBDIfRequired to handle error

* [NOD-1500] Removed some more fields from RPC

* [NOD-1500] Fix the getBlockTemplate flow

* [NOD-1500] Fix HandleGetCurrentNetwork

* [NOD-1500] Remove redundant code

* [NOD-1500] Remove obsolete notifications

* [NOD-1500] Split MiningManager and Consensus to separate fields in Domain

* [NOD-1500] Update two wrong references to location of txscript

* [NOD-1500] Added comments

* [NOD-1500] Fix some tests

* [NOD-1500] Removed serialization logic from appmessage

* [NOD-1500] Rename database/serialization/messages.proto to dbobjects.proto

* [NOD-1500] Delete integration tests

* [NOD-1500] Remove txsort

* [NOD-1500] Fix tiny bug

* [NOD-1500] Remove rogue dependancy on bchd

* [NOD-1500] Some stylistic fixes
2020-11-08 11:55:54 +02:00