* [NOD-42] Split checkFinalityRules and newFinalityPoint
* [NOD-42] Rename connectToDAG -> addBlock + move anything that is not actually connecting block to DAG out of connectBlock
* [NOD-42] Extract methods from PastUTXO
* [NOD-42] Give names to outputs in verifyAndBuildUTXO and propagate name
changes up the call tree
* [NOD-42] Split loop that creates UTXODiff and updates acceptance data into 2 separate methods
* [NOD-42] Removed from applyUTXOChanges any validation logic, moved in any logic related to updating the DAG and renamed to applyDAGChanges
* [NOD-42] Rename: CheckTransactionInputs -> CheckTransactionInputsAndCalculateFee
* [NOD-42] Revise some comments
* [NOD-42] Removed finalityErr constant - it's not needed
* Multiple chain -> dag corrections in comments
* [NOD-42] Removed redundant declaration of feeData
* [NOD-42] Reworded some comments
* [NOD-42] Rename MultiblockTxsAcceptanceData -> MultiBlockTxsAcceptanceData
* [DEV-378] Added feeAccumulator structures
* [DEV-378] Added logic to create and store fee data when validating a block
* [DEV-378] Renamed feeAccumulator to compactFeeData, and all related entities accordingly
* [DEV-378] Converted MsgTx.TxHash() to pointer to hash and not hash
* [DEV-378] Restructured parameters to buildFeeTransaction and related entities
* [DEV-378] Finished the code that calculates fees for blocks
* [DEV-378] Fix TxIndex after changing the structure of AcceptedTxsData
* [DEV-378] For genesis block: Return empty AcceptedTxsData instead of nil
* [DEV-378] Off-by-one error
* [DEV-378] Length of compactFeeData should be determined by specific method, not
* [DEV-378] Multiple bugfixes in tx fee calculation
* [DEV-378] Calculate fee even if fastAdd, to save feeData
* [DEV-378] use IsEqual instead of == when comparing TxHash
* [DEV-378] txindex: if including block is the new block - don't fetch id from DB
* [DEV-378] Fixed a few typos and made some vars consts
* [DEV-378] Re-organized fee functions, removed redundant functions and constants, and revised a few comments
* [DEV-378] Recovered fmt string changes lost in merge
* [DEV-378] Renamed acceptedTxsData and related types and vars to txsAcceptanceData
* [DEV-378] Some comment fixes
* [DEV-378] Remove redundant .ToString()
* [DEV-376] Changed any instance of %v in format strings with a more specific format token
* [DEV-376] Fixed some more wrong formatting strings + removed redundant
cast
* [DEV-376] Added fmt.Sprintf where it was missing
* [DEV-376] use %s for util.Amount, to invoke .String()
* [DEV-376] Some more fixes in format strings
* [DEV-376] fixed mruinvmap_test to expect the correct behaviour
* [DEV-364] Add fee transactions validation
* [DEV-364] make NextBlockFeeTransactions for creating block templates
* [DEV-364] apply coinbase rules to fee transaction is some cases
* [DEV-364] Add comments
* [DEV-364] put getTXO as separate function
* [DEV-364] Make getParentsFeeData a separate function
* [DEV-364] fix calculateFees
* [DEV-364] force maximum sequence for fee transactions
* [DEV-364] add TestValidateFeeTransactions
* [DEV-364] change fee transaction to be one tx per block rather than one tx for each blue
* [DEV-364] fix tests
* [DEV-364] Use constants instead of inline numbers
* [DEV-361] Create type TxID as alias to daghash.Hash. Use it for transaction IDs
* [DEV-361] Fixed missed renames
* [DEV-361] Removed usage of zeroHash
* [DEV-361] Fixed more missed renames
* [DEV-311] Moved subnetwork storage from directly in DAG to subnetworkStore
* Added gas validation in CheckBlockSanity
* [DEV-311] Add SubnetworkStore to last remaining call for CheckBlockSanity
* [DEV-311] Added subnetworkID to config in TestcheckBlockSanity
* [DEV-311] Moved CheckBlockSanity to be method of BlockDAG, and removed subnetworkStore argument
* [DEV-311] Removed SubnetworkID as parameter to CheckBlockSanity
* [DEV-311] Update gas usage before
* [DEV-311] some chain=>DAG updates in comments
* [DEV-311] Removed remaining dag-specific parameters from checkBlockSanity
* [DEV-339] Handling of dependent transactions in mempool
* [DEV-339] Small fixes after code review
* [DEV-339] Fixed compilation
* [DEV-339] Removed extra loop in addTransaction function
* [DEV-339] Changed addTransaction do not loop on inputs second time
* [DEV-348] Removed a couple of unused methods.
* [DEV-348] Implemented validating incoming transactions for bad partial transactions.
* [DEV-348] Added a (incomplete) filter for propogation of transactions.
* [DEV-348] Implemented filtering inventory by subnetwork.
* [DEV-348] Fixed broken tests.
* [DEV-348] Added test for non-zero payload partial transactions.
* [DEV-348] Added a comment for Config.SubnetworkID.
* [DEV-348] Fixed formatting.
* [DEV-348] Renamed isRemoteTransactionFull to shouldTxBeFull.
* [DEV-348] Added a check for invalid transaction in maybeAcceptTransaction. Added handling for native networks.
* [DEV-348] Fixed formatting.
* [DEV-348] Fixed a bug in transaction validation.
* [DEV-348] Rephrased a comment.
* [DEV-348] Extracted subnetwork compatibility to a method. Wrote a test for it.
* [DEV-348] Removed an unnecessary check over the native subnetwork.
* [DEV-343] Made sub-network registry use subNetworkIDs.
* [DEV-343] Removed an unnecessary clone.
* [DEV-343] Renamed buildSubNetworkID to txToSubNetworkID. Broke out of a loop when it was known that no further processing is required. Handled error cases from dbGetNetwork separately from the "not-found" case/
* [DEV-343] Added an error case in GasLimit() for where the sub-network is nil.
* [DEV-343] Fixed return nil instead of err. Used a better way to check whether we should continue checking accepted transactions.
* [DEV-312] Take in account subnetwork's GAS limit, when adding
transactions to block. Try to do that optimally.
* [DEV-312] Fixed GAS overusage calculation
* [DEV-337] Make sure that a transaction that uses more gas than the total allowed for sub-network
* [DEV-337] Moved transaction GAS check to mempool
* [DEV-337] Added Unit test for gas usage in transaction
* [DEV-337] Fixed build
* [DEV-337] Fixed tests stuff
* [DEV-337] Removed TODO comment
* [DEV-222] Changed type of TxOut.Value and util.Amount to uin64
* [DEV-222] Replaced math.MaxUint64 with 0
* [DEV-222] Fixed comment to reflect uint64 instead of int64
* [DEV-222] Fixed overflow comment
* [DEV-91] add tests for RemoveOrphansByTag
* [DEV-91] Add tests for orphan tx expiration
* [DEV-91] add TestDoubleSpends
* [DEV-91] add TestFetchTransaction
* [DEV-91] added a test for MinHighPriority
* [DEV-91] test unparsable scripts
* [DEV-91] test MaxOrphanTxs=0
* [DEV-91] add TestRemoveTransaction
* [DEV-105] use utxodiff in mempool instead of utxoview
* [DEV-105] get rid of utxoview
* [DEV-105] fix tests to use utxoset
* [DEV-105] remove utxoview type
* [DEV-105] move DagSetup to test_utils.go
* [DEV-105] add comments and add blockdag/test_utils_test.go
* [DEV-105] change checkPoolDoubleSpend to check utxodiff
* [DEV-105] add restoreInputs arg to removeTransaction
* [DEV-91] add restoreInputs arg to removeTransaction in unit tests
* [DEV-105] add restoreInputs arg to removeTransaction
* [DEV-105] give more descriptive names to vars
* [DEV-115] close txChan outside of HandleNewBlock
* [DEV-105] rename DagSetup -> DAGSetup
* [DEV-91] remove IsSpentInDiff
* [DEV-91] fix comment
* [DEV-91] Make IsPushOnlyScript return an error if the script cannot be parsed
* [DEV-91] add more tests for IsPushOnlyScript
* [DEV-91] fix comments
* [DEV-105] use utxodiff in mempool instead of utxoview
* [DEV-105] get rid of utxoview
* [DEV-105] fix tests to use utxoset
* [DEV-105] remove utxoview type
* [DEV-105] move DagSetup to test_utils.go
* [DEV-105] add comments and add blockdag/test_utils_test.go
* [DEV-105] add restoreInputs arg to removeTransaction
* [DEV-105] give more descriptive names to vars
* [DEV-115] close txChan outside of HandleNewBlock
* [DEV-105] rename DagSetup -> DAGSetup
* [DEV-75] Removed fetchEntryByHash, which was no longer used.
* [DEV-75] Removed a few functions in manager.go that weren't used by anything.
* [DEV-75] checkConnectBlock will soon not accept a utxoViewpoint, so removed the call to view.Tip() so that it could be deleted.
* [DEV-75] Got rid of tips in UtxoViewpoint.
* [DEV-75] Added the full UTXO set to the virtual block.
* [DEV-75] Implemented UTXO-wrangling when adding a new block.
* [DEV-75] Added isCoinbase to utxoEntry creation.
* [DEV-75] Added blockHeight to utxoEntry creation.
* Implemented fetching the fullUTXOSet from the database.
* [DEV-75] Simplified DAGState because almost all of the state in it was unnecessary. Also got rid of dbDAGState.
* [DEV-75] Made the process around adding a new block, UTXO-wise, much safer.
* [DEV-75] Implemented melding the virtual UTXO diff to the database.
* [DEV-75] Fixed utxoSet loading from the wrong bucket.
* [DEV-75] Began pruning utxoviewpoint.go. Replaced FetchUtxoEntry with a fullUTXOSet-based GetUTXOEntry.
* [DEV-75] Removed fetchUtxos.
* [DEV-75] Moved GetUTXOEntry into the virtual block.
* [DEV-75] Updated IndexManager to not use utxoViewpoints.
* [DEV-75] Fixed some bad login in restoreUTXO involving nodeDiffs.
* [DEV-75] Got rid of the UTXO spend journal, which wasn't used anywhere.
* [DEV-75] Moved some STXO-related validation logic out of connectToDAG and into checkConnectBlock.
* [DEV-75] Renamed UtxoXxx to UTXOXxx. Removed a bunch of methods that were no longer used by anything.
* [DEV-75] Another Utxo -> UTXO rename.
* [DEV-75] Removed IsModified from UTXOView, which was not used anywhere.
* [DEV-75] Renamed nodeDiff to provisionalNode. Added a bunch of comments.
* [DEV-75] Removed the test for genesis in pastUTXO, since it can never happen.
* [DEV-75] Wrote tests for errors in pastUTXO.
* [DEV-75] Wrote tests for errors in restoreUTXO.
* [DEV-75] Improved testErrorThroughPatching.
* [DEV-75] Wrote tests for errors in verifyAndBuildUTXO.
* [DEV-75] Used TipHashes instead of tips().hashes(), fixed comments in a few places, and brought back an error return that I erroneously removed.
* [DEV-75] Removed UTXO set logs.
* [DEV-75] Recreated add/remove/contains functions for utxoCollection.
* [DEV-75] Changed newUTXOEntry to use an object initializer.
* [DEV-75] Changed the UTXO bucket version to 1.
* Added a comment that clarifies that the index is not initialized before initDAGState is called.
* [DEV-75] Renamed GetVirtualBlock to VirtualBlock.
* [DEV-75] Removed superfluous variables.
* [DEV-75] Combined connectBlockToParents with updateParentsDiffs.
* [DEV-75] Removed another superfluous variable.
* [DEV-75] In pastUTXO, first fetch transactions from the database and only then add them.
* [DEV-75] Reworded the comment for commit().
* [DEV-75] Made all the connectUTXO subfunctions not BlockDAG methods.
* [DEV-75] Updated the comment for connectUTXO.
* [DEV-75] Updated the comment explaining why we're creating provisionalNodes.
* [DEV-75] Removed a couple of unnecessary calls to toProvisionalNode.
* [DEV-75] Renamed connectUTXO to applyUTXOChanges.
* [DEV-75] Replaced allProvisionalNodes with provisionalNodeSet, an object that holds provisionalNodes for this particular call to applyUTXOChanges.
* [DEV-75] Changed createProvisionalNode to accept a boolean "withParents" instead of relying on the caller to supply the node's parents or an empty set.
* [DEV-75] Made most applyUTXOChanges subfunctions be methods on provisionalNode.
* [DEV-75] Fixed a couple of bad comments.
* [DEV-75] Added descriptive error messages to callers of applyUTXOChanges.
* [DEV-75] Fixed weird English.
* [DEV-75] Replaced DAGState with a slice of DAG tip hashes.
* [DEV-75] In createProvisionalNode, changed withParents to withRelatives to avoid certain kinds of attacks.
* [DEV-75] Renamed createdProvisionalNode to newProvisionalNode.
* [DEV-75] Added precalculating the amount of transactions inside a new block's blue set.
* [DEV-75] Pruned unnecessary variable.
* [DEV-97] Moved github.com/daglabs/btcutil into github.com/daglabs/btcd/btcutil.
* [DEV-97] Updated Gopkg.toml to no longer refer to btcutil.
* [DEV-97] Renamed btcutil to util.
This modifies the utxoset in the database and related UtxoViewpoint to
store and work with unspent transaction outputs on a per-output basis
instead of at a transaction level. This was inspired by similar recent
changes in Bitcoin Core.
The primary motivation is to simplify the code, pave the way for a
utxo cache, and generally focus on optimizing runtime performance.
The tradeoff is that this approach does somewhat increase the size of
the serialized utxoset since it means that the transaction hash is
duplicated for each output as a part of the key and some additional
details such as whether the containing transaction is a coinbase and the
block height it was a part of are duplicated in each output.
However, in practice, the size difference isn't all that large, disk
space is relatively cheap, certainly cheaper than memory, and it is much
more important to provide more efficient runtime operation since that is
the ultimate purpose of the daemon.
While performing this conversion, it also simplifies the code to remove
the transaction version information from the utxoset as well as the
spend journal. The logic for only serializing it under certain
circumstances is complicated and it isn't actually used anywhere aside
from the gettxout RPC where it also isn't used by anything important
either. Consequently, this also removes the version field of the
gettxout RPC result.
The utxos in the database are automatically migrated to the new format
with this commit and it is possible to interrupt and resume the
migration process.
Finally, it also updates the tests for the new format and adds a new
function to the tests to convert the old test data to the new format for
convenience. The data has already been converted and updated in the
commit.
An overview of the changes are as follows:
- Remove transaction version from both spent and unspent output entries
- Update utxo serialization format to exclude the version
- Modify the spend journal serialization format
- The old version field is now reserved and always stores zero and
ignores it when reading
- This allows old entries to be used by new code without having to
migrate the entire spend journal
- Remove version field from gettxout RPC result
- Convert UtxoEntry to represent a specific utxo instead of a
transaction with all remaining utxos
- Optimize for memory usage with an eye towards a utxo cache
- Combine details such as whether the txout was contained in a
coinbase, is spent, and is modified into a single packed field of
bit flags
- Align entry fields to eliminate extra padding since ultimately
there will be a lot of these in memory
- Introduce a free list for serializing an outpoint to the database
key format to significantly reduce pressure on the GC
- Update all related functions that previously dealt with transaction
hashes to accept outpoints instead
- Update all callers accordingly
- Only add individually requested outputs from the mempool when
constructing a mempool view
- Modify the spend journal to always store the block height and coinbase
information with every spent txout
- Introduce code to handle fetching the missing information from
another utxo from the same transaction in the event an old style
entry is encountered
- Make use of a database cursor with seek to do this much more
efficiently than testing every possible output
- Always decompress data loaded from the database now that a utxo entry
only consists of a specific output
- Introduce upgrade code to migrate the utxo set to the new format
- Store versions of the utxoset and spend journal buckets
- Allow migration process to be interrupted and resumed
- Update all tests to expect the correct encodings, remove tests that no
longer apply, and add new ones for the new expected behavior
- Convert old tests for the legacy utxo format deserialization code to
test the new function that is used during upgrade
- Update the utxostore test data and add function that was used to
convert it
- Introduce a few new functions on UtxoViewpoint
- AddTxOut for adding an individual txout versus all of them
- addTxOut to handle the common code between the new AddTxOut and
existing AddTxOuts
- RemoveEntry for removing an individual txout
- fetchEntryByHash for fetching any remaining utxo for a given
transaction hash
new txs that it observes. The block manager alerts the fee estimator
of new and orphaned blocks.
Check for invalid state and recreate FeeEstimator if necessary.
This removes the standardness check to reject transactions with a lock
time greater than a maxint32 because the old bitcoind nodes which it was
designed to protect against are no longer valid for other reasons and
thus there are no longer any of them on the network to worry about.
This commit implements the new “weight” metric introduced as part of
the segwit soft-fork. Post-fork activation, rather than limiting the
size of blocks and transactions based purely on serialized size, a new
metric “weight” will instead be used as a way to more accurately
reflect the costs of a tx/block on the system. With blocks constrained
by weight, the maximum block-size increases to ~4MB.
This allows a caller-provided tag to be associated with orphan
transactions. This is useful since the caller can use the tag for
purposes such as keeping track of which peers orphans were first seen
from.
Also, since a parameter is required now anyways, it associates the peer
ID with processed transactions from remote peers.
This move the export for MinHighPriority from the mempool package to the
mining package. This should have been done when the priority
calculation code was moved to the mining package.
This commit adds a new option to the mempool’s policy configuration
which determines which transaction versions should be accepted as
standard.
The default version set by the policy within the server is 2; this
allows accepting transactions which have version 2 enabled in order to
utilize the new sequence locks feature.
This moves the priority-related code from the mempool package to the
mining package and also exports a new constant named UnminedHeight which
takes the place of the old unexported mempoolHeight.
Even though the mempool makes use of the priority code to make decisions
about what it will accept, priority really has to do with mining since
it influences which transactions will end up into a block. This change
also has the side effect of being a step towards enabling separation of
the mining code into its own package which, as previously mentioned,
needs access to the priority calculation code as well.
Finally, the mempoolHeight variable was poorly named since what it
really represents is a transaction that has not been mined into a block
yet. Renaming the variable to more accurately reflect its purpose makes
it clear that it belongs in the mining package which also needs the
definition now as well since the priority calculation code relies on it.
This will also benefit an outstanding PR which needs access to the same
value.
This implements orphan expiration in the mempool such that any orphans
that have not had their ancestors materialize within 15 minutes of their
initial arrival time will be evicted which in turn will remove any other
orphans that attempted to redeem it.
In order to perform the evictions with reasonable efficiency, an
opportunistic scan interval of 5 minutes is used. That is to say that
there is not a hard deadline on the scan interval and instead it runs
when a new orphan is added to the pool if enough time has passed.
The following is an example of running this code against the main
network for around 30 minutes:
23:05:34 2016-10-24 [DBG] TXMP: Expired 3 orphans (remaining: 254)
23:10:38 2016-10-24 [DBG] TXMP: Expired 112 orphans (remaining: 231)
23:15:43 2016-10-24 [DBG] TXMP: Expired 95 orphans (remaining: 206)
23:20:44 2016-10-24 [DBG] TXMP: Expired 90 orphans (remaining: 191)
23:25:51 2016-10-24 [DBG] TXMP: Expired 71 orphans (remaining: 191)
23:30:55 2016-10-24 [DBG] TXMP: Expired 70 orphans (remaining: 105)
23:36:19 2016-10-24 [DBG] TXMP: Expired 55 orphans (remaining: 107)
As can be seen from the above, without orphan expiration on this data
set, the orphan pool would have grown an additional 496 entries.
This modifies the way orphan removal and processing is done to more
aggressively remove orphans that can no longer be valid due to other
transactions being added or removed from the primary transaction pool.
The net effect of these changes is that orphan pool will typically be
much smaller which greatly improves its effectiveness. Previously, it
would typically quickly reach the max allowed worst-case usage and
effectively stay there forever.
The following is a summary of the changes:
- Modify the map that tracks which orphans redeem a given transaction to
instead track by the specific outpoints that are redeemed
- Modify the various orphan removal and processing functions to accept
the full transaction rather than just its hash
- Introduce a new flag on removeOrphans which specifies whether or not
to remove the transactions that redeem the orphan being removed as
well which is necessary since only some paths require it
- Add a new function named removeOrphanDoubleSpends that is invoked
whenever a transaction is added to the main pool and thus the outputs
they spent become concrete spends
- Introduce a new flag on maybeAcceptTransaction which specifies whether
or not duplicate orphans should be rejected since only some paths
require it
- Modify processOrphans as follows:
- Make use of the modified map
- Use newly available flags and logic work more strictly work with tx
chains
- Recursively remove any orphans that also redeem any outputs redeemed
by the accepted transactions
- Several new tests to ensure proper functionality
- Removing an orphan that doesn't exist is removed both when there is
another orphan that redeems it and when there is not
- Removing orphans works properly with orphan chains per the new
remove redeemers flag
- Removal of multi-input orphans that double spend an output when a
concrete redeemer enters the transaction pool
This optimizes the way in which the mempool oprhan map is limited in the
same way the server block manager maps were previously optimized.
Previously the code would read a cryptographically random value large
enough to construct a hash, find the first entry larger than that value,
and evict it.
That approach is quite inefficient and could easily become a
bottleneck when processing transactions due to the need to read from a
source such as /dev/urandom and all of the subsequent hash comparisons.
Luckily, strong cryptographic randomness is not needed here. The primary
intent of limiting the maps is to control memory usage with a secondary
concern of making it difficult for adversaries to force eviction of
specific entries.
Consequently, this changes the code to make use of the pseudorandom
iteration order of Go's maps along with the preimage resistance of the
hashing function to provide the desired functionality. It has
previously been discussed that the specific pseudorandom iteration order
is not guaranteed by the Go spec even though in practice that is how it
is implemented. This is not a concern however because even if the
specific compiler doesn't implement that, the preimage resistance of the
hashing function alone is enough.
The following is a before and after comparison of the function for both
speed and memory allocations:
benchmark old ns/op new ns/op delta
----------------------------------------------------------------
BenchmarkLimitNumOrphans 3727 243 -93.48%
benchmark old allocs new allocs delta
-----------------------------------------------------------------
BenchmarkLimitNumOrphans 4 0 -100.00%
This renames the mempool.Config.RelayNonStd option to AcceptNonStd which
more accurately describes its behavior since the mempool was refactored
into a separate package.
The reasoning for this change is that the mempool is not responsible for
relaying transactions (nor should it be). Its job is to maintain a pool
of unmined transactions that are validated according to consensus and
policy configuration options which are then used to provide a source of
transactions that need to be mined.
Instead, it is the server that is responsible for relaying transactions.
While it is true that the current server code currently only relays txns
that were accepted to the mempool, this does not necessarily have to
be the case. It would be entirely possible (and perhaps even a good
idea as something do in the future), to separate the relay policy from
the mempool acceptance policy (and thus indirectly the mining policy).
This coincides with the mempool only, policy change which enforces
transaction finality according to the median-time-past rather than
blockheader timestamps. The behavior is pre-cursor to full blown BIP
113 consensus deployment, and subsequent activation.
As a result, the TimeSource field in the mempoolConfig is no longer
needed so it has been removed. Additionally, checkTransactionStandard has been
modified to instead take a time.Time as the mempool is no longer explicitly
dependant on a Chain instance.
