218 Commits

Author SHA1 Message Date
Ori Newman
b04292c97a Don't server parallel PruningPointAndItsAnticoneRequests 2021-11-22 09:56:30 +02:00
Ori Newman
765dd170e4
Optimizations and header size reduce hardfork (#1853)
* Modify DefaultTimeout to 120 seconds

A temporary workaround for nodes having trouble to sync (currently the download of pruning point related data during IBD takes more than 30 seconds)

* Cache existence in reachability store

* Cache block level in the header

* Fix IBD indication on submit block

* Add hardForkOmitGenesisFromParentsDAAScore logic

* Fix NumThreads bug in the wallet

* Get rid of ParentsAtLevel header method

* Fix a bug in BuildPruningPointProof

* Increase race detector timeout

* Add cache to BuildPruningPointProof

* Add comments and temp comment out go vet

* Fix ParentsAtLevel

* Dont fill empty parents

* Change HardForkOmitGenesisFromParentsDAAScore in fast netsync test

* Add --allow-submit-block-when-not-synced in stability tests

* Fix TestPruning

* Return fast tests

* Fix off by one error on kaspawallet

* Fetch only one block with trusted data at a time

* Update fork DAA score

* Don't ban for unexpected message type

* Fix tests

Co-authored-by: Michael Sutton <mikisiton2@gmail.com>
Co-authored-by: Ori Newman <>
2021-11-22 09:00:39 +02:00
Ori Newman
9b81f5145e
Increase p2p msg size and reset utxo after ibd (#1847)
* Don't build unnecessary binaries

* Reset UTXO index after IBD

* Enlarge max msg size to 1gb

* Fix UTXO chunks logic

* Revert UTXO set override change

* Fix sendPruningPointUTXOSet

* Increase tests timeout to 20 minutes

Co-authored-by: Kaspa Profiler <>
2021-11-11 09:27:07 +02:00
Ori Newman
ff72568d6b
Fix pruning point anticone order (#1836)
* Send pruning point anticone in topological order
Fix a UTXO pagination bug
Lengthen the stabilization time for the last DAA test

* Extend "sudden hash rate drop" test length to 45 minutes

Co-authored-by: Kaspa Profiler <>
2021-11-07 08:21:34 +02:00
Ori Newman
99aaacd649
Check blue score before requesting a pruning proof (#1835)
* Check blue score before requesting a pruning proof

* BuildPruningPointProof should return empty proof if the pruning point is genesis

* Don't fail many-tips if kaspad exits ungracefully
2021-10-31 12:48:18 +02:00
stasatdaglabs
77a344cc29
In IBD, validate the timestamps of the headers of the pruning point and selected tip (#1829)
* Implement validatePruningPointFutureHeaderTimestamps.

* Fix TestIBDWithPruning.

* Fix wrong logic.

* Add a comment.

* Fix a comment.

* Fix a variable name.

* Add a commment

* Fix TestIBDWithPruning.

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-10-30 20:32:49 +03:00
stasatdaglabs
3dbc42b4f7
Implement the new block subsidy function (#1830)
* Replace the old blockSubsidy parameters with the new ones.

* Return subsidyGenesisReward if blockHash is the genesis hash.

* Traverse a block's past for the subsidy calculation.

* Partially implement SubsidyStore.

* Refer to SubsidyStore from CoinbaseManager.

* Wrap calcBlockSubsidy in getBlockSubsidy, which first checks the database.

* Fix finalityStore not calling GenerateShardingID.

* Implement calculateAveragePastSubsidy.

* Implement calculateMergeSetSubsidySum.

* Implement calculateSubsidyRandomVariable.

* Implement calcBlockSubsidy.

* Add a TODO about floats.

* Update the calcBlockSubsidy TODO.

* Use binary.LittleEndian in calculateSubsidyRandomVariable.

* Fix bad range in calculateSubsidyRandomVariable.

* Replace float64 with big.Rat everywhere except for subsidyRandomVariable.

* Fix a nil dereference.

* Use a random walk to approximate the normal distribution.

* In order to avoid unsupported fractional results from powInt64, flip the numerator and the denominator manually.

* Set standardDeviation to 0.25, MaxSompi to 10_000_000_000 * SompiPerKaspa and defaultSubsidyGenesisReward to 1_000.

* Set the standard deviation to 0.2.

* Use a binomial distribution instead of trying to estimate the normal distribution.

* Change some values around.

* Clamp the block subsidy.

* Remove the fake duplicate constants in the util package.

* Reduce MaxSompi to only 100m Kaspa to avoid hitting the uint64 ceiling.

* Lower MaxSompi further to avoid new and exciting ways for the uint64 ceiling to be hit.

* Remove debug logs.

* Fix a couple of failing tests.

* Fix TestBlockWindow.

* Fix limitTransactionCount sometimes crashing on index-out-of-bounds.

* In TrustedDataDataDAABlock, replace BlockHeader with DomainBlock

* In calculateAveragePastSubsidy, use blockWindow instead of doing a BFS manually.

* Remove the reference to DAGTopologyManager in coinbaseManager.

* Add subsidy to the coinbase payload.

* Get rid of the subsidy store and extract subsidies out of coinbase transactions.

* Keep a blockWindow amount of blocks under the virtual for IBD purposes.

* Manually remove the virtual genesis from the merge set.

* Fix simnet genesis.

* Fix TestPruning.

* Fix TestCheckBlockIsNotPruned.

* Fix TestBlockWindow.

* Fix TestCalculateSignatureHashSchnorr.

* Fix TestCalculateSignatureHashECDSA.

* Fix serializing the wrong value into the coinbase payload.

* Rename coinbaseOutputForBlueBlock to coinbaseOutputAndSubsidyForBlueBlock.

* Add a TODO about optimizing trusted data DAA window blocks.

* Expand on a comment in TestCheckBlockIsNotPruned.

* In calcBlockSubsidy, divide the big.Int numerator by the big.Int denominator instead of converting to float64.

* Clarify a comment.

* Rename SubsidyMinGenesisReward to MinSubsidy.

* Properly handle trusted data blocks in calculateMergeSetSubsidySum.

* Use the first two bytes of the selected parent's hash for randomness instead of math/rand.

* Restore maxSompi to what it used to be.

* Fix TestPruning.

* Fix TestAmountCreation.

* Fix TestBlockWindow.

* Fix TestAmountUnitConversions.

* Increase the timeout in many-tips to 30 minutes.

* Check coinbase subsidy for every block

* Re-rename functions

* Use shift instead of powInt64 to determine subsidyRandom

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-10-30 10:16:47 +03:00
Ori Newman
5dbb1da84b
Implement pruning point proof (#1832)
* Calculate GHOSTDAG, reachability etc for each level

* Don't preallocate cache for dag stores except level 0 and reduce the number of connections in the integration test to 32

* Reduce the number of connections in the integration test to 16

* Increase page file

* BuildPruningPointProof

* BuildPruningPointProof

* Add PruningProofManager

* Implement ApplyPruningPointProof

* Add prefix and fix blockAtDepth and fill headersByLevel

* Some bug fixes

* Include all relevant blocks for each level in the proof

* Fix syncAndValidatePruningPointProof to return the right block hash

* Fix block window

* Fix isAncestorOfPruningPoint

* Ban for rule errors on pruning proof

* Find common ancestor for blockAtDepthMAtNextLevel

* Use pruning proof in TestValidateAndInsertImportedPruningPoint

* stage status and finality point for proof blocks

* Uncomment golint

* Change test timeouts

* Calculate merge set for ApplyPruningPointProof

* Increase test timeout

* Add better caching for daa window store

* Return to default timeout

* Add ErrPruningProofMissesBlocksBelowPruningPoint

* Add errDAAWindowBlockNotFound

* Force connection loop next iteration on connection manager stop

* Revert to Test64IncomingConnections

* Remove BlockAtDepth from DAGTraversalManager

* numBullies->16

* Set page file size to 8gb

* Increase p2p max message size

* Test64IncomingConnections->Test16IncomingConnections

* Add comment for PruningProofM

* Add comment in `func (c *ConnectionManager) Stop()`

* Rename isAncestorOfPruningPoint->isAncestorOfSelectedTip

* Revert page file to 16gb

* Improve ExpectedHeaderPruningPoint perf

* Fix comment

* Revert "Improve ExpectedHeaderPruningPoint perf"

This reverts commit bca1080e7140c78d510f51bbea858ae280c2f38e.

* Don't test windows
2021-10-26 09:48:27 +03:00
stasatdaglabs
0053ee788d
Use the BlueWork declared in an orphan block's header instead of requesting it explicitly from the peer that sent us the orphan (#1828) 2021-09-13 13:13:03 +03:00
stasatdaglabs
af7e7de247
Add PruningPointProof to the P2P protocol (#1825)
* Add PruningPointProof to externalapi.

* Add BuildPruningPointProof and ValidatePruningPointProof to Consensus.

* Add the pruning point proof to the protocol.

* Add the pruning point proof to the wire package.

* Add PruningPointBlueWork.

* Make go vet happy.

* Properly initialize PruningPointProof in consensus.go.

* Validate pruning point blue work.

* Populate PruningPointBlueWork with the actual blue work of the pruning point.

* Revert "Populate PruningPointBlueWork with the actual blue work of the pruning point."

This reverts commit f2a9829998a8161ca24c259f971bd144084f0488.

* Revert "Validate pruning point blue work."

This reverts commit c6a90c5d2c5137657484d49ae8f33609aeb50235.

* Revert "Properly initialize PruningPointProof in consensus.go."

This reverts commit 9391574bbf711ab14eaf0c7700ae4d4b2e782e9d.

* Revert "Add PruningPointBlueWork."

This reverts commit 48182f652a5ff5048017d2c26640a1ffec79bc60.

* Fix PruningPointProof and MsgPruningPointProof to be two-dimensional.

* Fix wire PruningPointProof to be two-dimensional.
2021-09-05 17:20:15 +03:00
Ori Newman
02a08902a7
Fix current pruning point index cache (#1824)
* Fix ps.currentPruningPointIndexCache

* Remove redundant dependency from block builder

* Fix typo
2021-09-05 07:37:25 +03:00
Ori Newman
d9bc94a2a8
Replace header finality point with pruning point and enforce finality rules on IBD with headers proof (#1823)
* Replace header finality point with pruning point

* Fix TestTransactionAcceptance

* Fix pruning candidate

* Store all past pruning points

* Pass pruning points on IBD

* Add blue score to block header

* Simplify ArePruningPointsInValidChain

* Fix static check errors

* Fix genesis

* Renames and text fixing

* Use ExpectedHeaderPruningPoint in block builder

* Fix TestCheckPruningPointViolation
2021-08-31 08:01:48 +03:00
stasatdaglabs
837dac68b5
Update block headers to include multiple levels of parent blocks (#1822)
* Replace the old parents in the block header with BlockLevelParents.

* Begin fixing compilation errors.

* Implement database serialization for block level parents.

* Implement p2p serialization for block level parents.

* Implement rpc serialization for block level parents.

* Add DirectParents() to the block header interface.

* Use DirectParents() instead of Parents() in some places.

* Revert test_block_builder.go.

* Add block level parents to hash serialization.

* Use the zero hash for genesis finality points.

* Fix failing tests.

* Fix a variable name.

* Update headerEstimatedSerializedSize.

* Add comments in blocklevelparents.go.

* Fix the rpc-stability stability test.

* Change the field number for `parents` fields in p2p.proto and rpc.proto.

* Remove MsgBlockHeader::NumParentBlocks.
2021-08-24 12:06:39 +03:00
stasatdaglabs
d922ee1be2
Add header commitments for DAA score, blue work, and finality points (#1817)
* Add DAAScore, BlueWork, and FinalityPoint to externalapi.BlockHeader.

* Add DAAScore, BlueWork, and FinalityPoint to NewImmutableBlockHeader and fix compilation errors.

* Add DAAScore, BlueWork, and FinalityPoint to protowire header types and fix failing tests.

* Check for header DAA score in validateDifficulty.

* Add DAA score to buildBlock.

* Fix failing tests.

* Add a blue work check in validateDifficultyDAAAndBlueWork.

* Add blue work to buildBlock and fix failing tests.

* Add finality point validation to ValidateHeaderInContext.

* Fix genesis blocks' finality points.

* Add finalityPoint to blockBuilder.

* Fix tests that failed due to missing reachability data.

* Make blockBuilder use VirtualFinalityPoint instead of directly calling FinalityPoint with the virtual hash.

* Don't validate the finality point for blocks with trusted data.

* Add debug logs.

* Skip finality point validation for block whose finality points are the virtual genesis.

* Revert "Add debug logs."

This reverts commit 3c18f519ccbb382f86f63904dbb1c4cd6bc68b00.

* Move checkDAAScore and checkBlueWork to validateBlockHeaderInContext.

* Add checkCoinbaseBlueScore to validateBodyInContext.

* Fix failing tests.

* Add DAAScore, blueWork, and finalityPoint to blocks' hashes.

* Generate new genesis blocks.

* Fix failing tests.

* In BuildUTXOInvalidBlock, get the bits from StageDAADataAndReturnRequiredDifficulty instead of calling RequiredDifficulty separately.
2021-08-12 13:25:00 +03:00
Ori Newman
f26a7fdedf
Return headers first (#1806)
* Return headers first

* Delete TestHandleRelayInvs

* resolve virtual only after IBD

* Fix ResolveVirtual

* Fix comments and variable names
2021-07-27 17:07:29 +03:00
Ori Newman
d207888b67
Implement pruned headers node (#1787)
* Pruning headers p2p basic structure

* Remove headers-first

* Fix consensus tests except TestValidateAndInsertPruningPointWithSideBlocks and TestValidateAndInsertImportedPruningPoint

* Add virtual genesis

* Implement PruningPointAndItsAnticoneWithMetaData

* Start fixing TestValidateAndInsertImportedPruningPoint

* Fix TestValidateAndInsertImportedPruningPoint

* Fix BlockWindow

* Update p2p and gRPC

* Fix all tests except TestHandleRelayInvs

* Delete TestHandleRelayInvs parts that cover the old IBD flow

* Fix lint errors

* Add p2p_request_ibd_blocks.go

* Clean code

* Make MsgBlockWithMetaData implement its own representation

* Remove redundant check if highest share block is below the pruning point

* Fix TestCheckLockTimeVerifyConditionedByAbsoluteTimeWithWrongLockTime

* Fix comments, errors ane names

* Fix window size to the real value

* Check reindex root after each block at TestUpdateReindexRoot

* Remove irrelevant check

* Renames and comments

* Remove redundant argument from sendGetBlockLocator

* Don't delete staging on non-recoverable errors

* Renames and comments

* Remove redundant code

* Commit changes inside ResolveVirtual

* Add comment to IsRecoverableError

* Remove blocksWithMetaDataGHOSTDAGDataStore

* Increase windows pagefile

* Move DeleteStagingConsensus outside of defer

* Get rid of mustAccepted in receiveBlockWithMetaData

* Ban on invalid pruning point

* Rename interface_datastructures_daawindowstore.go to interface_datastructures_blocks_with_meta_data_daa_window_store.go

* * Change GetVirtualSelectedParentChainFromBlockResponseMessage and VirtualSelectedParentChainChangedNotificationMessage to show only added block hashes
*  Remove ResolveVirtual
* Use externalapi.ConsensusWrapper inside MiningManager
* Fix pruningmanager.blockwithmetadata

* Set pruning point selected child when importing the pruning point UTXO set

* Change virtual genesis hash

* replace the selected parent with virtual genesis on removePrunedBlocksFromGHOSTDAGData

* Get rid of low hash in block locators

* Remove +1 from everywhere we use difficultyAdjustmentWindowSize and increase the default value by one

* Add comments about consensus wrapper

* Don't use separate staging area when resolving resolveBlockStatus

* Fix netsync stability test

* Fix checkResolveVirtual

* Rename ConsensusWrapper->ConsensusReference

* Get rid of blockHeapNode

* Add comment to defaultDifficultyAdjustmentWindowSize

* Add SelectedChild to DAGTraversalManager

* Remove redundant copy

* Rename blockWindowHeap->calculateBlockWindowHeap

* Move isVirtualGenesisOnlyParent to utils

* Change BlockWithMetaData->BlockWithTrustedData

* Get rid of maxReasonLength

* Split IBD to 100 blocks each time

* Fix a bug in calculateBlockWindowHeap

* Switch to trusted data when encountering virtual genesis in blockWithTrustedData

* Move ConsensusReference to domain

* Update ConsensusReference comment

* Add comment

* Rename shouldNotAddGenesis->skipAddingGenesis
2021-07-26 12:24:07 +03:00
stasatdaglabs
38e2ee1b43
Change the log level of the transaction propagation log from Info to Debug. (#1804) 2021-07-19 10:30:29 +03:00
Svarog
60e7a8ebed
Make transaction propagation much more frequent (#1799)
* Make transaction propagation much more frequent

* Update f.transactionIDsToPropagate after brodacasting
2021-07-14 16:06:24 +03:00
Svarog
369a3bac09
Limit block mass instead of merge set limit + Introduce SigOpCount to TransactionInput (#1790)
* Update constants

* Add to transaction SigOpCount

* Update mass calculation, and move it from InContext to InIsolation

* Update block validation accordingly

* Add SigOpCount validation during TransactionInContext

* Remove checking of mass vs maxMassAcceptedByBlock from consensusStateManager

* Update mining manager with latest changes

* Add SigOpCount to MsgTx.Copy()

* Fix initTestTransactionAcceptanceDataForClone

* Fix all tests in transaction_equal_clone_test.go

* Fix TestBlockMass

* Fix tests in transactionvalidator package

* Add SigOpCount to sighash

* Fix TestPruningDepth

* Fix problems in libkaspawalelt

* Fix integration tests

* Fix CalculateSignatureHash tests

* Remove remaining places talking about block size

* Add sanity check to checkBlockMass to make sure all transactions have their mass filled

* always add own sigOpCount to sigHash

* Update protowire/rpc.md

* Start working on removing any remaining reference to block/tx size

* Update rpc transaction verbose data to include mass rather then size

* Convert verboseData and block size check to mass

* Remove remaining usages of tx size in mempool

* Move transactionEstimatedSerializedSize to transactionvalidator

* Add PopulateMass to fakeRelayInvsContext

* Move PopulateMass to beggining of ValidateAndInsertTransaction + fix in it

* Assign mass a new number for backward-compatibility
2021-07-14 14:21:57 +03:00
stasatdaglabs
a4d241c30a
Batch transaction inv messages (#1788)
* Implement EnqueueTransactionIDsForPropagation.

* Fix tests.

* Fix TxRelayTest.

* Add a log for transaction propagation.

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-07-07 17:54:57 +03:00
Constantine Bitensky
e3a6d9e49a
Added RPC connection server version checking, fixes https://github.com/kaspanet/kaspad/issues/1047 (#1783)
Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-07-06 15:10:35 +03:00
Svarog
069ee26e84
Adds name to route, and writes it in every error message (#1777)
* Adds name to route, and writes it in every error message

* Update all calls with route name

* Fixed a few missed points

Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-07-04 14:40:27 +03:00
Svarog
61aa15fd61
Update lastRebroadcastTime when we are rebroadcasting + Add some logs to mempool (#1776)
* Update lastRebroadcastTime when we are rebroadcasting

* Add some logs to mempool

Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-07-04 12:08:44 +03:00
cbitensky
2f7a1395e7
Make use of maxBlocks instead of maxBlueScoreDifference in antiPastHashesBetween (#1772)
* Make use of maxBlocks instead of maxBlueScoreDifference in antiPastHashesBetween

* Make use of maxBlocks instead of maxBlueScoreDifference in antiPastHashesBetween

* Make use of maxBlocks instead of maxBlueScoreDifference in antiPastHashesBetween

* Make use of maxBlocks instead of maxBlueScoreDifference in antiPastHashesBetween

* Make use of maxBlocks instead of maxBlueScoreDifference in antiPastHashesBetween

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-06-30 17:21:22 +03:00
Svarog
798c5fab7d
Add allowOrphans to rpcclient.SubmitTransaction (#1765) 2021-06-24 15:23:10 +03:00
Svarog
c13a4d90ed
Mempool redesign (#1752)
* Added model and stubs for all main methods

* Add constructors to all main objects

* Implement BlockCandidateTransactions

* implement expireOldTransactions and expireOrphanTransactions

* Rename isHighPriority to neverExpires

* Add stub for checkDoubleSpends

* Revert "Rename isHighPriority to neverExpires"

This reverts commit b2da9a4a00c02fb380d2518cf54fa16257bd8423.

* Imeplement transactionsOrderedByFeeRate

* Orphan maps should be idToOrphan

* Add error.go to mempool

* Invert the condition for banning when mempool rejects a transaction

* Move all model objects to model package

* Implement getParentsInPool

* Implemented mempoolUTXOSet.addTransaction

* Implement removeTransaction, remove sanity checks

* Implemented mempoolUTXOSet.checkDoubleSpends

* Implemented removeOrphan

* Implement removeOrphan

* Implement maybeAddOrphan and AddOrphan

* Implemented processOrphansAfterAcceptedTransaction

* Implement transactionsPool.addTransaction

* Implement RemoveTransaction

* If a transaction was removed from the mempool - update it's redeemers in orphan pool as well

* Use maximumOrphanTransactionCount

* Add allowOrphans to ValidateAndInsertTransaction stub

* Implement validateTransaction functions

* Implement fillInputs

* Implement ValidateAndInsertTransaction

* Implement HandleNewBlockTransactions

* Implement missing mempool interface methods

* Add comments to exported functions

* Call ValidateTransactionInIsolation where needed

* Implement RevalidateHighPriorityTransactions

* Rewire kaspad to use new mempool, and fix compilation errors

* Update rebroadcast logic to use new structure

* Handle non-standard transaction errors properly

* Add mutex to mempool

* bugfix: GetTransaction panics when ok is false

* properly calculate targetBlocksPerSecond in config.go

* Fix various lint errors and tests

* Fix expected text in test for duplicate transactions

* Skip the coinbase transaction in HandleNewBlockTransactions

* Unorphan the correct transactions

* Call ValidateTransactionAndPopulateWithConsensusData on unorphanTransaction

* Re-apply policy_test as check_transactions_standard_test

* removeTransaction: Remove redeemers in orphan pool as well

* Remove redundant check for uint64 < 0

* Export and rename isDust -> IsTransactionOutputDust to allow usage by rothschild

* Add allowOrphan to SubmitTransaction RPC request

* Remove all implementation from mempool.go

* tidy go mod

* Don't pass acceptedOrphans to handleNewBlockTransactions

* Use t.Errorf in stead of t.Fatalf

* Remove minimum relay fee from TestDust, as it's no longer configurable

* Add separate VirtualDAASCore method for faster retrieval where it's repeated multiple times

* Broadcast all transactions that were accepted

* Don't re-use GetVirtualDAAScore in GetVirtualInfo - this causes a deadlock

* Use real transaction count, and not Orphan

* Get mempool config from outside, incorporating values received from cli

* Use MinRelayFee and MaxOrphanTxs from global kaspad config

* Add explanation for the seemingly redundant check for transaction version in checkTransactionStandard

* Update some comment

* Convert creation of acceptedTransactions to a single line

* Move mempoolUTXOSet out of checkDoubleSpends

* Add test for attempt to insert double spend into mempool

* fillInputs: Skip check for coinbase - it's always false in mempool

* Clarify comment about removeRedeemers when removing random orphan

* Don't remove high-priority transactions in limitTransactionCount

* Use mempool.removeTransaction in limitTransactionCount

* Add mutex comment to handleNewBlockTransactions

* Return error from limitTransactionCount

* Pluralize the map types

* mempoolUTXOSet.removeTransaction: Don't restore utxo if it was not created in mempool

* Don't evacuate from orphanPool high-priority transactions

* Disallow double-spends in orphan pool

* Don't use exported (and locking) methods from inside mempool

* Check for double spends in mempool during revalidateTransaction

* Add checkOrphanDuplicate

* Add orphan to acceptedOrphans, not current

* Add TestHighPriorityTransactions

* Fix off-by-one error in limitTransactionCount

* Add TestRevalidateHighPriorityTransactions

* Remove checkDoubleSpends from revalidateTransaction

* Fix TestRevalidateHighPriorityTransactions

* Move check for MaximumOrphanCount to beggining of maybeAddOrphan

* Rename all map type to singulateToSingularMap

* limitOrphanPool only after the orphan was added

* TestDoubleSpendInMempool: use createChildTxWhenParentTxWasAddedByConsensus instead of createTransactionWithUTXOEntry

* Fix some comments

* Have separate min/max transaction versions for mempool

* Add comment on defaultMaximumOrphanTransactionCount to keep it small as long as we have recursion

* Fix comment

* Rename: createChildTxWhenParentTxWasAddedByConsensus -> createChildTxWhereParentTxWasAddedByConsensus

* Handle error from createChildTxWhereParentTxWasAddedByConsensus

* Rename createChildTxWhereParentTxWasAddedByConsensus -> createChildAndParentTxsAndAddParentToConsensus

* Convert all MaximumXXX constants to uint64

* Add comment

* remove mutex comments
2021-06-23 15:49:20 +03:00
Constantine Bitensky
4ba8b14675
Make GetVirtualSelectedParentBlueScore work properly (#1764) 2021-06-21 15:56:37 +03:00
Constantine Bitensky
319cbce768
Friendly error messages for ban and unban with ip with brackets (#1762) 2021-06-21 11:42:45 +03:00
Svarog
f317f51cdd
Change IBD finished log to specify if completed succesfully or not (#1754)
* Change IBD finished log to specify if complete succesfully or not

* Move log to outside UnsetIBDRunning

* Style inhancement of IBD finished string
2021-06-16 11:13:29 +03:00
Ori Newman
4207c82f5a
Add database prefix (#1750)
* Add prefix to stores

* Add prefix to forgotten stores

* Add a special type for prefix

* Rename transaction->dbTx

* Change error message

* Use countKeyName

* Rename Temporary Consesnsus to Staging

* Add DeleteStagingConsensus to Domain interface

* Add lock to staging consensus

* Make prefix type-safer

* Use ioutil.TempDir instead of t.TempDir
2021-06-15 17:47:17 +03:00
Constantine Bitensky
2ae1b7853f
Added comment to database lock error (#1746)
More readable database lock error string

Co-authored-by: Constantine Bitensky <constantinebitensky@gmail.com>
Co-authored-by: Svarog <feanorr@gmail.com>
2021-06-10 19:49:08 +03:00
Constantine Bitensky
79c74c482b
Get connections from Seeder when no connections left (#1742)
Co-authored-by: Constantine Bitensky <constantinebitensky@gmail.com>
2021-06-08 17:42:13 +03:00
stasatdaglabs
363494ef7a Implement NotifyVirtualDaaScoreChanged (#1737)
* Add notifyVirtualDaaScoreChanged to protowire.

* Add notifyVirtualDaaScoreChanged to the rest of kaspad.

* Add notifyVirtualDaaScoreChanged to the rest of kaspad.

* Test the DAA score notification in TestVirtualSelectedParentBlueScore.

* Rename TestVirtualSelectedParentBlueScore to TestVirtualSelectedParentBlueScoreAndVirtualDAAScore.

(cherry picked from commit 83e631548f206aaf2da5b2103681c715bc250c75)
2021-06-06 16:46:02 +03:00
Svarog
4f52a6de51
Check if err != nil returned from ConnectionManager.Ban() (#1736) 2021-05-31 10:50:12 +03:00
Svarog
4f4a8934e7
Add option to specify blockHash in EstimateNetworkHashesPerSecond (#1731)
* Add BlockHash optional parameter to EstimateNetworkBlockHashesPerSecond

* Allow to pass '-' for optional values in kaspactl

* Solve a division-by-zero in estimateNetworkHashesPerSecond

* Add BlockHash to toAppMessage/fromAppMessage functions

* Rename: topHash -> StartHash

* Return proper error message if provided startHash doesn't deserialize into a hash
2021-05-27 14:59:29 +03:00
Ori Newman
4658f9d05c
Implement BIP 39 and HD wallet features (#1705)
* Naive bip39 with address reuse

* Avoid address reuse in libkaspawallet

* Add wallet daemon

* Use daemon everywhere

* Add forceOverride

* Make CreateUnsignedTransaction endpoint receive amount in sompis

* Collect close UTXOs

* Filter out non-spendable UTXOs from selectUTXOs

* Use different paths for multisig and non multisig

* Fix tests to use non zero path

* Fix multisig cosigner index detection

* Add comments

* Fix dump_unencrypted_data.go according to bip39 and bip32

* Fix wrong derivation path for multisig on wallet creation

* Remove IsSynced endpoint and add validation if wallet is synced for the relevant endpoints

* Rename server address to daemon address

* Fix capacity for extendedPublicKeys

* Use ReadBytes instead of ReadLine

* Add validation when importing

* Increment before using index value, and use it as is

* Save keys file exactly where needed

* Use %+v printErrorAndExit

* Remove redundant consts

* Rnemae collectCloseUTXOs and collectFarUTXOs

* Move typedefs around

* Add comment to addressesToQuery

* Update collectUTXOsFromRecentAddresses comment about locks

* Split collectUTXOs to small functions

* Add sanity check

* Add addEntryToUTXOSet function

* Change validateIsSynced to isSynced

* Simplify createKeyPairsFromFunction logic

* Rename .Sync() to .Save()

* Fix typo

* Create bip39BitSize const

* Add consts to purposes

* Add multisig check for 'send'

* Rename updatedPSTxBytes to partiallySignedTransaction

* Change collectUTXOsFromFarAddresses's comment

* Use setters for last used indexes

* Don't use the pstx acronym

* Fix SetPath

* Remove spaces when reading lines

* Fix walletserver to daemonaddress

* Fix isUTXOSpendable to use DAA score

Co-authored-by: Svarog <feanorr@gmail.com>
2021-05-19 10:03:23 +03:00
Ori Newman
010df3b0d3
Rename IncludeTransactionVerboseData flag to IncludeTransactions (#1717)
* Rename IncludeTransactionVerboseData flag to IncludeTransactions

* Regenerate auto-generated files
2021-05-18 17:40:06 +03:00
Ori Newman
268906a7ce Add VirtualDaaScore to GetBlockDagInfo (#1719)
Co-authored-by: Svarog <feanorr@gmail.com>

(cherry picked from commit 36c56f73bf7b41528c81013d5d0d3bc7d7f6a206)
2021-05-18 17:07:07 +03:00
Ori Newman
9df80957b1 Fix getBlock and getBlocks RPC commands to return blocks and transactions properly (#1716)
* Fix getBlock RPC command to return transactions

* Fix getBlocks RPC command to return transactions and blocks

* Add GetBlockEvenIfHeaderOnly and use it for getBlock and getBlocks

* Implement GetBlockEvenIfHeaderOnly for fakeRelayInvsContext

* Use less nested code

(cherry picked from commit 50fd86e2878a6dcf1399610d8edc146466a5b492)
2021-05-13 16:00:42 +03:00
talelbaz
ab89efe3dc
Update relayTransactions unit tests. (#1623)
* [NOD-1344] relaytransactions: simple unit tests

* [NOD-1344]  Add mid-complexity unit tests for relaytransactions
* Improve TestHandleRelayedTransactionssub tests
* Improve TestHandleRequestedTransactions sub tests

* [NOD-1344]  Fix Simple call test

* [NOD-1344]  Fix tests after redesign

* Divide transactionrelay_test.go to 2 separated tests and updates the tests.

* Changes due to review:change the test files name and the test function names, adds new comments and fix typo.

* Delete an unnecessary comparison to True in the if statement condition.

* Update the branch to v0.11.0-dev.

Co-authored-by: karim1king <karimkaspersky@yahoo.com>
Co-authored-by: tal <tal@daglabs.com>
Co-authored-by: Svarog <feanorr@gmail.com>
2021-05-02 16:16:57 +03:00
stasatdaglabs
1f56a68a28
Add an RPC command: EstimateNetworkHashesPerSecond (#1686)
* Implement EstimateNetworkHashesPerSecond.

* Fix failing tests.

* Add request/response messages to the .proto files.

* Add the EstimateNetworkHashesPerSecond RPC command.

* Add the EstimateNetworkHashesPerSecond RPC client function.

* Add the EstimateNetworkHashesPerSecond RPC command to kaspactl.

* Disallow windowSize lesser than 2.

* Fix wrong scale (milliseconds instead of seconds).

* Handle windowHashes being 0.
2021-04-22 15:18:21 +03:00
Elichai Turkel
83beae4463
Add consensus.Config as a wrapper for dagParams (#1680)
* Add a new consensus.Config wrapper to dagParams

* Update all tests to use consensus.Config
2021-04-19 09:07:34 +03:00
Svarog
acdc59b565
Add version file to database (#1678)
* Add version file to database

* Remove redundant code

* Check for version before opening the database, create version file after

* Create version file before opening the database
2021-04-14 12:35:39 +03:00
stasatdaglabs
dfa24d8353
Implement a stability test for mempool limits (#1647)
* Copy some boilerplate from the other stability tests.

* Fix a copy+paste error in run.sh.

* Copy over some stability test boilerplate go code.

* Run kaspad in the background.

* Catch panics and initialize the RPC client.

* Mine enough blocks to fund filling up the mempool.

* Extract coinbase transactions out of the generated blocks.

* Tidy up a bit.

* Implement submitting transactions.

* Lower the amount of outputs in each transaction.

* Verify that the mempool size has the expected amount of transactions.

* Pregenerate enough funds before submitting the first transaction so that block creation doesn't interfere with the test.

* Empty mempool out by continuously adding blocks to the DAG.

* Handle orphan transactions when overfilling the mempool.

* Increase mempoolSizeLimit to 1m.

* Fix a comment.

* Fix a comment.

* Add mempool-limits to run-slow.sh.

* Rename generateTransactionsWithLotsOfOutputs to generateTransactionsWithMultipleOutputs.

* Rename generateCoinbaseTransaction to mineBlockAndGetCoinbaseTransaction.

* Make generateFundingCoinbaseTransactions return an object instead of store a global variable.

* Convert mempool-limits into a Go test.

* Convert panics to t.Fatalfs.

* Fix a comment.

* Increase mempoolSizeLimit to 1m.

* Run TestMempoolLimits only if RUN_STABILITY_TESTS is set.

* Move the run of mempool-limits in run-slow.sh.

* Add a comment above fundingCoinbaseTransactions.

* Make a couple of stylistic changes.

* Use transactionhelper.CoinbaseTransactionIndex instead of hardcoding 0.

* Make uninteresting errors print %+v instead of %s.

Co-authored-by: Svarog <feanorr@gmail.com>
2021-04-11 16:59:11 +03:00
Svarog
9bb8123391
Don't include selectedParentHash in block verbose data if it's nil + Fix test vectors in rpc-stability (#1668)
* Fix submitBlockRequest in rpc-stability/commands.json

* Don't include selectedParentHash in block verbose data if it's nil (a.k.a. genesis)
2021-04-08 15:34:10 +03:00
stasatdaglabs
2854d91688
Add missing call to broadcastTransactionsAfterBlockAdded (#1639)
* Add missing call to broadcastTransactionsAfterBlockAdded.

* Fix a comment.

* Fix a comment some more.

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-03-31 10:28:02 +03:00
stasatdaglabs
c5b0394bbc
In RPC, use RPCTransactions and RPCBlocks instead of TransactionMessages and BlockMessages (#1609)
* Replace BlockMessage with RpcBlock in rpc.proto.

* Convert everything in kaspad to use RPCBlocks and fix tests.

* Fix compilation errors in stability tests and the miner.

* Update TransactionVerboseData in rpc.proto.

* Update TransactionVerboseData in the rest of kaspad.

* Make golint happy.

* Include RpcTransactionVerboseData in RpcTransaction instead of the other way around.

* Regenerate rpc.pb.go after merge.

* Update appmessage types.

* Update appmessage request and response types.

* Reimplement conversion functions between appmessage.RPCTransaction and protowire.RpcTransaction.

* Extract RpcBlockHeader toAppMessage/fromAppMessage out of RpcBlock.

* Fix compilation errors in getBlock, getBlocks, and submitBlock.

* Fix compilation errors in getMempoolEntry.

* Fix compilation errors in notifyBlockAdded.

* Update verbosedata.go.

* Fix compilation errors in getBlock and getBlocks.

* Fix compilation errors in getBlocks tests.

* Fix conversions between getBlocks message types.

* Fix integration tests.

* Fix a comment.

* Add selectedParent to the verbose block response.
2021-03-30 17:43:02 +03:00
Svarog
f1451406f7
Add support for multiple staging areas (#1633)
* Add StagingArea struct

* Implemented staging areas in blockStore

* Move blockStagingShard to separate folder

* Apply staging shard to acceptanceDataStore

* Update blockHeaderStore with StagingArea

* Add StagingArea to BlockRelationStore

* Add StagingArea to blockStatusStore

* Add StagingArea to consensusStateStore

* Add StagingArea to daaBlocksStore

* Add StagingArea to finalityStore

* Add StagingArea to ghostdagDataStore

* Add StagingArea to headersSelectedChainStore and headersSelectedTipStore

* Add StagingArea to multisetStore

* Add StagingArea to pruningStore

* Add StagingArea to reachabilityDataStore

* Add StagingArea to utxoDiffStore

* Fix forgotten compilation error

* Update reachability manager and some more things with StagingArea

* Add StagingArea to dagTopologyManager, and some more

* Add StagingArea to GHOSTDAGManager, and some more

* Add StagingArea to difficultyManager, and some more

* Add StagingArea to dagTraversalManager, and some more

* Add StagingArea to headerTipsManager, and some more

* Add StagingArea to constnsusStateManager, pastMedianTimeManager

* Add StagingArea to transactionValidator

* Add StagingArea to finalityManager

* Add StagingArea to mergeDepthManager

* Add StagingArea to pruningManager

* Add StagingArea to rest of ValidateAndInsertBlock

* Add StagingArea to blockValidator

* Add StagingArea to coinbaseManager

* Add StagingArea to syncManager

* Add StagingArea to blockBuilder

* Update consensus with StagingArea

* Add StagingArea to ghostdag2

* Fix remaining compilation errors

* Update names of stagingShards

* Fix forgotten stagingArea passing

* Mark stagingShard.isCommited = true once commited

* Move isStaged to stagingShard, so that it's available without going through store

* Make blockHeaderStore count be avilable from stagingShard

* Fix remaining forgotten stagingArea passing

* commitAllChanges should call dbTx.Commit in the end

* Fix all tests tests in blockValidator

* Fix all tests in consensusStateManager and some more

* Fix all tests in pruningManager

* Add many missing stagingAreas in tests

* Fix many tests

* Fix most of all other tests

* Fix ghostdag_test.go

* Add comment to StagingArea

* Make list of StagingShards an array

* Add comment to StagingShardID

* Make sure all staging shards are pointer-receiver

* Undo bucket rename in block_store

* Typo: isCommited -> isCommitted

* Add comment explaining why stagingArea.shards is an array
2021-03-29 10:34:11 +03:00
Svarog
6824be9216
Remove support for ServiceFlags out of DNSSeeder (#1622) 2021-03-18 18:02:57 +02:00
Elichai Turkel
caf251b7a8
Replace the HomeDir flag with a AppDir flag (#1615) 2021-03-17 12:48:38 +02:00