* [NOD-492] Split ApiServer to server and syncd.
* [NOD-492] Add missing file.
* [NOD-492] Remove references to --migrate from common config.
* [NOD-492] Move MQTT to the sync daemon.
* [NOD-492] Fix server Dockerfile and create one for syncd.
* [NOD-492] Rename ApiServer to Kasparov.
* [NOD-492] Fix packages.
* [NOD-492] Fix more packages.
* [NOD-492] Fix comments and package names.
* [NOD-492] Move blank import packages to main.
* [NOD-492] Move common logging logic out of individual config.go files.
* [NOD-492] Move database models to a package called dbmodels.
* [NOD-492] Rename models package to apimodels.
* [NOD-487] Implement a mechanism to gracefully shut down after a panic.
* [NOD-487] Fixed bad log.
* [NOD-487] Removed unused import.
* [NOD-487] Convert panic handlers from anonymous functions to methods.
* [NOD-488] Make getBlueBlocksBetween return error if start hash is not in the selected parent chain of stop hash
* [NOD-488] Convert for to while style
* [NOD-479] Separate max outbound connections and max inbound connections
* [NOD-479] Fix merge
* [NOD-479] Renames and add function countinboundPeers
* [NOD-479] Remove redundant check on maximum outbound peers
* [NOD-479] Rename countinboundPeers -> countInboundPeers
* [NOD-428] Required RPC user and password, and do not create a default config file for btcctl if rpc login details were provided
* [NOD-428] Don't check rpc user and password if rpc is disabled
* [NOD-428] Fix error message
* [NOD-484] Fix deadlock between p2p server and sync manager during shutdown.
* [NOD-484] Fix quitWaitGroup.Wait() potentially not waiting in some scenarios.
* [NOD-484] Add a comment explaining quitWaitGroup.
* [NOD-484] Fix typo.
* [NOD-484] Add etc to comment.
* [NOD-443] Immediately request missing parents in API server sync
* [NOD-443] Add rpc client log to api server
* [NOD-443] Fix wrong ordering of pendingHeaders queue
* [NOD-443] Fix error comparision at TestNewHashFromStr
* [NOD-443] Make a separate handleMissingParentHeaders function
* [NOD-443] Change log level
* [NOD-443] Put handleMissingParentHeaders next to handleBlockHeader
* [NOD-443] Make handleBlockAddedMsg function
* [NOD-443] Make reusable missingParentsHashesStr string
* [NOD-443] Remove redundant 's'
* [NOD-443] Refactor to first get all blocks and then add them to database
* [NOD-443] Rename variables and functions, and remove redundant logs
* [NOD-443] Make fetchBlockAndMissingAncestors use block hash as an argument
* [NOD-443] Add log only for first orphan block
* [NOD-443] Fix wrong order of adding blocks to pendingBlocks
* [NOD-443] Write logs for all orphans
* [NOD-443] Log only missing parents that are not already fetched
* [NOD-443] Rename rawVerboseBlockTuple -> rawVerboseBlock
* [NOD-443] Make fetchBlock return *rawVerboseBlock
* [NOD-443] Rename rawVerboseBlock -> rawAndVerboseBlock
* [NOD-472] Don't fetch accepting block and tx confirmations for getBlocks
* [NOD-472] Don't fetch accepting block and tx confirmations in any block verbose result
* [NOD-472] Add stringPointerToString function
* [NOD-382] Add notification for accepted transactions
* [NOD-382] Remove print statement
* [NOD-426] Publish notifications for unaccepted transactions
* [NOD-426] Load DB in controller
* [NOD-426] Remove function name from error message
* [NOD-426] Add input addresses for transactions notifications
* [NOD-426] Remove function name from error message
* [NOD-426] Change method name to accepted transactions
* [NOD-426] Remove newlines
* [NOD-426] Use join instead of separate query
* [NOD-426] Remove new line
* [NOD-447] Fix deadlocks and hanging goroutines
* [NOD-447] Add tests
* [NOD-447] Add unpatch to spawnPatch
* [NOD-447] Don't send to releaseWait if waitingCounter is zero
* [NOD-447] Change waitingCounter to boolean and rename to isReleaseWaitWaiting, change checkIfRunningSpawnsAreLeft to return only one function, and lock critical code related to wg.isReleaseWaitWaiting
* [NOD-447] Rename txConfirmations -> txConfirmationsNoLock, txConfirmationsWithLock -> txConfirmations
* [NOD-447] Add documentation and delete redundant spawn
* [NOD-447] Fix comments
* [NOD-447] Fix comments
* [NOD-445] Added option to mining simulator to get address list from AWS
* [NOD-445] Add support to get miningsimulator addresslist from AWS
* [NOD-445] Added mechanism to update when new servers come online
* [NOD-445] Set config in connectionManager
* [NOD-445] Invert DisableTLS condition in readCert
* [NOD-381] Publish transaction messages to MQTT
* [NOD-381] Remove redundant variable
* [NOD-381] Send payload as string
* [NOD-381] Add Error handling
* [NOD-381] Respond with TransactionResponse
* [NOD-381] Use transactionResponse for notifications
* [NOD-381] Move code to appropriate places
* [NOD-381] Pass raw block instead of txId
* [NOD-381] Add comments to public functions
* [NOD-381] Remove print statement
* [NOD-381] Pass transaction instead of block; Use pointers so default will be nil;
* [NOD-381] Use pointers so value could be nil
* [NOD-381] Change variable name
* [NOD-381] Set QoS to 2
* [NOD-381] Move isConnected to MQTT, so client won't have to worry about it; General code refactors;
* [NOD-448] Change GetBlocksCmd to be able to include both raw and verbose block data.
* [NOD-448] Update sync logic to only make one getBlocks call per page.
* [NOD-448] Make GetBlocks get each block only once.
* [NOD-423] Rename BestBlock to SelectedTip
* [NOD-423] Implement GetSelectedTip RPC command
* [NOD-423] Add help to getSelectedTip command
* [NOD-423] Fix getSelectedTip test
* [NOD-423] Fix tests so they would compile. These tests will need to be rewriten at some point.
* [NOD-423] Make integration test compile. Test need to be revisited
* [NOD-423] Rename variables
* [NOD-423] Change comment s about best block to selected tip.
* [NOD-423] Update comment
* [NOD-423] Change height to bluescore
* [NOD-434] Add the same enqueue/process mechanism as chainChangedMsgs for blockAddedMsgs.
* [NOD-434] Clean up after merge.
* [NOD-434] Implement mechanism for re-requesting missing parent blocks.
* [NOD-434] Fixed bad error message.
* [NOD-434] Split processBlockAddedMsgs.
* [NOD-434] Name return values in canHandleBlockAddedMsg.
* [NOD-434] Rename canHandleBlockAddedMsg to missingParentHashes and fix bad loop break.
* [NOD-434] Rename the variable missingParentHashes to missingHashes.
* [NOD-434] Rename a couple of variables.
* [NOD-434] Rename outerloop to outerLoop.
* [NOD-434] Fix typo and remove superfluous continue.
* [NOD-412] Change Warnf to Errorf where appropriate.
* [NOD-430] Print hashes of missing parents in case can't insert block into DB of API-Server
* [NOD-430] Use continue OUTER_LOOP instead of break
* [NOD-430] Use lowerCamelCase for label
* [NOD-380] Add MQTT to the project.
* [NOD-380] Add MQTT params to config.
* [NOD-380] Implement connecting to an mqtt broker.
* [NOD-380] Fix a comment.
* [NOD-380] Removed unnecessary option.
* [NOD-380] Added comments to MQTT functions.
* [NOD-380] Fix copy+paste error.
* [NOD-380] Make it so that all the mqtt flags must be passed together.
* [NOD-380] Use activeConfig instead of passing it everywhere.
* [NOD-395] Write a test for the diffFromAcceptanceData crash.
* [NOD-395] Converted MultiBlockTxsAcceptanceData into a slice.
* [NOD-395] Fix failing test.
* [NOD-395] Populate multiBlockTxsAcceptanceData bottom-to-top.
* [NOD-395] Add comment to FindAcceptanceData.
* [NOD-395] Remove no-longer relevant note about probability in TestOrderInDiffFromAcceptanceData.
* [NOD-413] Make "Max failed connection attempts reached" less frequent
* [NOD-413] Throttle only certain types of logs.
* [NOD-413] Add a comment for shouldWriteConnFailedLog.
* [NOD-413] Fix lint error.
* [NOD-413] Make ErrNoAddress a special type to support error wrapping.
* [NOD-413] Make throttledConnFailedLogInterval 10 minutes.
* [NOD-413] Move p2p errors into variables.
* [NOD-413] Reorganize throttled stuff to be next to each other.
* [NOD-386] Extract net parsing functionality to a shared place.
* [NOD-386] Add extract ActiveNetParams to cmdconfig
* [NOD-386] Adding comments so go-vet won't shout at me
* [NOD-386] Rename package name to config
* [NOD-386] Rename commandConfig to configFlags
* [NOD-386] Rename function to ResolveNetwork
* [NOD-386] Fix renaming errors
* [NOD-386] Refactor network config to btcd level so APIserver and btcd could use it
* [NOD-386] Refactor network config to config package
* [NOD-386] Move ActiveNetParams to network section
* [NOD-386] Explictly return nil
* [NOD-386] Reuse activeNetParams from netwrok config
* [NOD-386] Set ActiveNetworkFlags instance to be global
* [NOD-386] Remove redundant newline
* [NOD-386] Init ActiveNetParams in address manager test
* [NOD-386] Add dnsseeder network config
* [NOD-386] Use ActiveConfig() method to access configuration
* [NOD-373] Implement Schnorr digital signatures and remove ECDSA (based on code from gcash/bchd)
* [NOD-374] Add new error to list; Update comments.
* [NOD-373] Remove leftovers of verifyMessage RPC command (which was deleted)
* [NOD-373] Remove redundant test, add Schnorr tests, and fix tests where needed
* [NOD-373] Fix tests and remove redundant ones
* [NOD-373] Refactor functions names
* [NOD-373] Remove empty line
* [NOD-373] Fix comments, rename functions to more meaningful names
* [NOD-373] Additional data in nonceRFC6979 should not be nil
* [NOD-373] Refactor function name
* [NOD-373] Add permalinks for links to bchd code
* [NOD-385] Make confirmations be calculated as dag.selectedTip().blueScore - acceptingBlock.blueScore + 2
* [NOD-385] Fix comments
* [NOD-385] Make more explicit check in accepting block for selected tip
* [NOD-385] Put only non accepted transactions in areTxsInBlock
* [NOD-385] fetchSelectedTip only if needed
* [NOD-404] Calculate mass in API server
* [NOD-404] Fix uninitialized maps
* [NOD-404] Use txID instead of prevDBTransactionsOutput.Transaction.TransactionID
* [NOD-398] Change API server type HandlerError to work with errors instead of error strings
* [NOD-398] Rename OriginalError -> Cause and isHandleError -> ok
* [NOD-350] Implement testnet faucet
* [NOD-350] Add JSON annotations to api server response types
* [NOD-350] Fix IP check query, update IP usage with upsert, and make IP a primary key
* [NOD-377] Remove redundant float conversion
* [NOD-377] Change not current database error message
* [NOD-377] change API route from /money_request to /request_money
* [NOD-377] Add a constant for 24 hours
* [NOD-377] Remove redundant call for getWalletUTXOSet()
* [NOD-377] Condition refactoring
* [NOD-377] Fix POST request to API server content type
* [NOD-350] Rename day -> timeBetweenRequests
* [NOD-377] Rename timeBetweenRequests -> minRequestInterval, timeBefore24Hours -> minRequestInterval
* [NOD-350] Rename file responsetypes -> response_types
* [NOD-350] Rename convertTxModelToTxResponse -> convertTxDBModelToTxResponse
* [NOD-350] Explicitly select blue_score in fetchSelectedTipBlueScore
* [NOD-350] Refactor and add comments
* [NOD-350] Make calcFee use MassPerTxByte
* [NOD-350] Convert IP column to varchar(39) to allow ipv6 addresses
* [NOD-350] Add comments to isFundedAndIsChangeOutputRequired
* [NOD-350] Remove approximateConfirmationsForCoinbaseMaturity
* [NOD-350] Fix comments
* [NOD-394] Rename --only-build to --no-run.
* [NOD-394] Allow --rm and --no-build to be run together with no-run.
* [NOD-394] Make --cleanup alias for --rm --no-run --no-build.
* [NOD-394] Fix typo in usage string.
* [NOD-394] Set docker/docker-compose.yaml to use devnet instead of testnet.
* [NOD-376] Made bad unorphaned blocks not reject the original block.
* [NOD-376] Fix wording in a comment.
* [NOD-376] Add a test to make sure that bad child blocks don't invalidate valid parent blocks.
* [NOD-376] Clarify comments and don't check PoW for child block (it's irrelevant for this test case).
* [NOD-368] correct text output of address generator (address instead of public key)
* [NOD-368] add hash160 to output of genaddr
* [NOD-368] change encoding to hexadecimal
* [NOD-368] fix formatting
* change finality interval to 1000 ( ~16.6 minutes interval)
* [NOD-357] define finality interval in dagParams instead of using a constant.
* use dagParams for FinalityInterval instead of constant
* override parameter so test would pass on CI (Jenkins machine runs out of memory if we use 1000)
* formating the code
* [NOD-134] Don't connect to an address from the same 16 CIDR
* [NOD-134] Rename outboundPeerConnected variables
* [NOD-134] Change newConnMtx to newConnReqMtx
* [NOD-338] Recover indexer if it didn't work for a while
* [NOD-338] Recover indexer if it didn't work for a while
* [NOD-338] Recover indexer if it didn't work for a while
* [NOD-338] Add tests and move blockidhash.go to blockdag package
* [NOD-338] Delete index current block id when dropping index, and do some refactoring
* [NOD-338] Change comments
* [NOD-338] Change recover error messages
* [NOD-338] Fix comments
* [NOD-338] Fix comments and fix test name
* [NOD-337] In CheckTransactionSanity, made max mass of transaction to be half of block max mass.
* [NOD-337] Added a comment for MaxMassPerTx.
* [NOD-337] Fixed a couple of comments.
* [NOD-275] Moved getBlockTemplate and related functionality to a separate file.
* [NOD-275] Started moving handlers to separate files.
* [NOD-275] Fixed merge errors.
* [NOD-275] Moved all handlers out of rpcserver.go.
* [NOD-275] Moved non-shared functions out of rpcserver.go.
* [NOD-275] Moved handleGetAllManualNodesInfo to a separate file.
* [NOD-275] Moved handlers out of rpcwebsocket.go to separate files.
* [NOD-275] Fixed import error.
* [NOD-275] Renamed all handler files to include underscores.
* [NOD-275] Moved common rpc helper functions to common.go.
* [NOD-289] Implemented database isCurrent checking and connection.
* [NOD-289] Added GetChainFromBlock to RPCClient.
* [NOD-289] Limited the amount of blocks in GetChainFromBlockResponse.
* [NOD-289] Fixed various issues that were keeping GetChainFromBlocks from working properly.
* [NOD-289] Created blockloop.go.
* [NOD-289] Updated go.mod after merge.
* [NOD-289] Implemented collection of current selected parent chain.
* [NOD-289] Fixed test. Reverted not deleting utxoDiffData from the DB.
* [NOD-289] Implemented GetBlocks.
* [NOD-289] Added comment to BlockHashesFrom.
* [NOD-289] Added GetBlocks to rpcclient.
* [NOD-289] Added verboseBlocks to GetBlocks.
* [NOD-289] Implemented block insertion.
* [NOD-289] Added AUTO_INCREMENT to tables that were missing it.
* [NOD-289] Made gasLimit in subnetwork nullable.
* [NOD-289] Renamed transactions_outputs to transaction_outputs.
* [NOD-289] Fixed weird coinbase behavior in vin.
* [NOD-289] Made collectCurrentBlocks start from the most recent startHash.
* [NOD-289] Added IsChainBlock to GetBlockVerboseResult.
* [NOD-289] Implemented adding a block from onBlockAdded.
* [NOD-289] Added removedParentChainHashes to getChainFromBlock.
* [NOD-289] Implemented updating the selected parent chain from onChainChanged.
* [NOD-289] Implemented some initial logic for updating the UTXO.
* [NOD-289] Fixed merge errors.
* [NOD-326] Fixed some more merge errors.
* [NOD-289] Added error handling for missing required records.
* [NOD-289] Implemented handling removedChainHashes.
* [NOD-289] Implemented handling addedChainBlocks.
* [NOD-289] Fixed incorrect coinbase check.
* [NOD-289] Implemented inserting the transaction output address.
* [NOD-289] Added updating block.IsChainBlock.
* [NOD-289] Split insertBlock into many small functions.
* [NOD-289] Split updateSelectedParentChain into smaller functions.
* [NOD-289] Fixed pointer errors.
* [NOD-289] Fixed a bad exists check.
* [NOD-289] Fixed a couple of small bugs.
* [NOD-289] Fixed a TxID/Hash mixup.
* [NOD-289] Added block/tx mass to getBlockVerboseResponse.
* [NOD-289] Renamed blockLoop.go to sync.go. Added comments.
* [NOD-289] Deleted apiserver README.
* [NOD-289] Fixed golint errors.
* [NOD-289] Renamed findMostRecentBlockHash to findHashOfBluestBlock.
* [NOD-289] Fixed style in syncBlocks and fixed a comment.
* [NOD-289] Copied NewErrorFromDBErrors over from NOD-324.
* [NOD-289] Created a couple of utils to make error handling with gorm slightly less painful.
* [NOD-289] Added error handling for database calls.
* [NOD-289] Fixed some more style/comments.
* [NOD-289] Fixed comments.
* [NOD-289] Renamed TransactionInput.TransactionOutput to TransactionInput.PreviousTransactionOutput.
* [NOD-289] Added a commends about pagination in getBlocks and getChainFromBlock.
* [NOD-289] Removed the coinbase field from Vin.
* [NOD-289] Deferred handling chainChangedMsgs until we have the appropriate data.
* [NOD-289] Optimized queries in updateRemovedChainHashes and updateAddedChainBlocks.
* [NOD-289] Optimized queries in insertBlockParents.
* [NOD-289] Optimized queries in insertTransactionInput.
* [NOD-289] Split Where calls to separate lines.
* [NOD-289] Fixed merge errors.
* [NOD-289] Exited early from insertBlockParents if we're the genesis block.
* [NOD-289] Improved nextChainChangedChan mechanism.
* [NOD-289] Fixed the above sync mechanism a bit.
* [NOD-289] Renamed IsDBRecordNotFoundError to HasDBRecordNotFoundError and IsDBError to HasDBError.
* [NOD-289] Replaced old error handling for db errors with the lovely new stuff.
* [NOD-289] Exited early if we already inserted a block. This saves us checking if a record already exists for some record types.
* [NOD-289] Decoupled syncBlocks from syncSelectedParentChain.
* [NOD-289] Made a comment more explicit.
* [NOD-289] Extracted net resolution to a separate function.
* [NOD-289] Extracted syncing to a separate function.
* [NOD-289] Fixed a comment.
* [NOD-289] Fixed merge erros.
* [NOD-289] Fixed a couple of bugs.
* [NOD-289] Fixed another bug.
* [NOD-289] Extracted ChainChangedMsg conversion to a separate function.
* [NOD-289] Optimized queries in canHandleChainChangedMsg.
* [NOD-289] Moved the sync function closer to its call site.
* [NOD-289] Renamed HasDBRecordNotFoundError to IsDBRecordNotFoundError.
* [NOD-289] Used count instead of first.
* [NOD-289] Renamed address to hexAddress.
* [NOD-315] Created acceptanceindex.go including boilerplate.
* [NOD-315] Disallowed calls to notifyChainChanges and getChainFromBlock if the acceptance index is not on.
* [NOD-315] Implemented the acceptance index.
* [NOD-315] Fixed serialization/deserialization. Added test.
* [NOD-315] Fixed/added comments.
* [NOD-315] Fixed copy/paste errors.
* [NOD-315] Added an empty line for readability.
* [NOD-335] Made it not write a stack trace if the command line flags are wrong.
* [NOD-335] Fixed panic not printing the right error.
* [NOD-335] Removed code duplication.
* [NOD-319] Add query params to api server route handler
* Temp commit
* [NOD-322] Make database.DB a function
* [NOD-322] Move context to be the first parameter in all functions
* [NOD-322] Set db to nil on database.Close()
* [NOD-322] Tidy go.mod/go.sum
* [NOD-323] Move rpc-client to separate package
* [NOD-309] Add controller for POST /transaction
* [NOD-309] Added route for POST /transaction
* [NOD-309] in POST /transaction: Forward reject errors to client
* [NOD-309] Added custom client messages to errors in POST /transaction
* [NOD-309] Use utils.NewInternalServerHandlerError where appropriate
* [NOD-324] Properly handle GORM errors in API server
* [NOD-324] Handle RecordNotFound error in GetBlockByHashHandler
* [NOD-324] Make a separate function for NewErrorFromDBErrors
* [NOD-327] Add --migrate cli flag to API server
* [NOD-327] Change log messages
* [NOD-327] Remove `required` flag from API server RPC CLI arguments
* [NOD-327] Add database version in migrations logs
* [NOD-307] Implement API-Server GET /blocks
* [NOD-307] Implement API-Server GET /blocks
* [NOD-307] Add comments to exported constants
* [NOD-307] Flatten GET query values and check that 'order' value is valid
* [NOD-307] Validate order values in GetBlocksHandler
* [NOD-307] Add convertQueryParamToInt function
* [NOD-322] Make database.DB a function
* [NOD-322] Move context to be the first parameter in all functions
* [NOD-322] Set db to nil on database.Close()
* [NOD-322] Tidy go.mod/go.sum
* [NOD-322] Use http package const + message for StatusInternalServerError
* [NOD-299] Add waitgroup to wait for all `spawn`s to complete before calling teardown
* [NOD-299] Restore spawn on teardown + mark spawn done in the correct thread
* [NOD-303] Implement get transactions by address for API server
* [NOD-304] Implement get transactions by address
* [NOD-304] Implement get transactions by address
* [NOD-304] Use structs for where if possible
* [NOD-304] Auto increment IDs
* [NOD-304] Make defaultGetTransactionsLimit constant
* [NOD-304] Delete db directory
* [NOD-304] change db var to query
* [NOD-304] Extract route handle function from addRoutes
* [NOD-304] Order transactions by ID
* [NOD-304] Add error for passing arrays to GET
* [NOD-303] Implement get transaction by id for api server
* [NOD-303] Make routeParamTxID a constant
* [NOD-303] Change database is not current error.
* [NOD-303] Add ID to TransactionInput and TransactionOutput models
* [NOD-303] Change transactions_outputs table name to transaction_outputs and transactions_inputs to transaction_inputs
* [NOD-303] Add json annotations to transaction response types
* [NOD-303] Split server package
* [NOD-303] Add GetTransactionByHashHandler
* [NOD-303] Add comments to exported functions and variables
* [NOD-303] Put response types in a separate file
* [NOD-303] Rename functions
* [NOD-310] Implement REST server in API server
* [NOD-310] MetaData -> Metadata
* [NOD-310] Make custom context methods instead of custom request functions
* [NOD-310] change "Request ID" prefix to "RID" and convert to apiServerContext with newAPIServerContext everywhere
* [NOD-301] Don't sync with peer if the rendezvous point is below finality
* [NOD-301] Add block hash and peer address for the warn message
* [NOD-301] Fix perrLog.Warnf arguments order
* [NOD-300] If node has invalid ancestor set the according status in blockindex
* [NOD-300] Test that status is also updated for grand child of an invalid block
* [NOD-300] change make(blockSet) to newSet()
* [NOD-292] In accept.go, made dbStoreBlock and flushToDB occur within the same transaction.
* [NOD-292] Implemented processing blocks that were not validated on BTCD start.
* [NOD-292] Fixed processing logic on init. Added a test for it.
* [NOD-292] Fixed some comments.
* [NOD-292] Made unlocks deferred in a couple of places.
* [NOD-292] Made unprocessed block reprocess via ProcessBlock rather than maybeAcceptBlock.
* [NOD-292] Fixed grammar in comment. Added an explanation to TestAcceptingInInit.
* [NOD-292] Split flushToDB into two versions.
* [NOD-292] Fixed a bad assignment.
* [NOD-292] Fixed bad spacing.
* [NOD-297] Fix onChainChanged on rpcclient
* [NOD-285] create gorm models for db (#378)
* [NOD-285] Map API-Server database using GORM
* [NOD-285] Add accepting block to transactions and blocks models, and remove accepting block model
* [NOD-285] Define model relations
* [NOD-285] Fix many to many for Transaction and Block models
* [NOD-285] Remove redundant main file
* [NOD-296] Send SyncMgr.SubmitBlock errors as rpc errors (#381)
* [NOD-296] Send SyncMgr.SubmitBlock errors as rpc errors
* [NOD-296] Add error message prefix
* [NOD-298] Add comments to gorm models (#382)
* [NOD-294] Fix golint in deploy.sh and fix all lint warnings (#380)
* [NOD-294] Fix golint in deploy.sh and fixed all lint errors
* [NOD-294] Fix typos in comments
* [NOD-294] Convert VirtualForTest into alias of *virtualBlock
* [NOD-294] Fixed some more typos in comments
* [NOD-295] Limit the length of GetData to 50 (#383)
* [NOD-295] Fixed bad break condition in addInvsToGetDataMessageFromQueue.
* [NOD-295] Fixed the fix for bad break condition in addInvsToGetDataMessageFromQueue.
* [NOD-295] Made the check for max invs refer to invsNum instead of MaxInvPerGetDataMsg.
* [NOD-297] Fix onChainChanged on rpcclient
* [NOD-286] Implement API-Server base structure (#379)
* [NOD-286] Implement API-Server base structure
* [NOD-286] Add rpc user and password as command line arguments
* [NOD-286] Make log directory a CLI argument
* [NOD-286] Add db login details as CLI arguments
* [NOD-297] Fix onChainChanged on rpcclient and server
* [NOD-297] Fix variables and functions names
* [NOD-297] Fix AcceptedTxIds -> AcceptedTxIDs
* [NOD-286] Implement API-Server base structure
* [NOD-286] Add rpc user and password as command line arguments
* [NOD-286] Make log directory a CLI argument
* [NOD-286] Add db login details as CLI arguments
* [NOD-295] Fixed bad break condition in addInvsToGetDataMessageFromQueue.
* [NOD-295] Fixed the fix for bad break condition in addInvsToGetDataMessageFromQueue.
* [NOD-295] Made the check for max invs refer to invsNum instead of MaxInvPerGetDataMsg.
* [NOD-294] Fix golint in deploy.sh and fixed all lint errors
* [NOD-294] Fix typos in comments
* [NOD-294] Convert VirtualForTest into alias of *virtualBlock
* [NOD-294] Fixed some more typos in comments
* [NOD-285] Map API-Server database using GORM
* [NOD-285] Add accepting block to transactions and blocks models, and remove accepting block model
* [NOD-285] Define model relations
* [NOD-285] Fix many to many for Transaction and Block models
* [NOD-285] Remove redundant main file
* [NOD-280] Create database for API server
* [NOD-280] Rename public_key_script to pk_script
* [NOD-280] Change indexes names
* [NOD-280] Add accepting block to blocks and transactions table and remove accepting_blocks table
* [NOD-280] Add readme
* [NOD-280] Change VARCHAR(32) to CHAR(64)
* [NOD-280] Rename location_in_block to index
* [NOD-270] Added notifyChainChanges and related commands.
* [NOD-270] Added NTChainChanged to blockdag.
* [NOD-270] Implemented collection and sending of ChainChanged notifications.
* [NOD-270] Fixed an improperly named test.
* [NOD-270] Added a test: TestChainChangedNotification.
* [NOD-270] Fixed a couple copy+paste errors.
* [NOD-270] Added a couple of comments for TestChainChangedNotification.
* [NOD-270] Fixed formatting error.
* [NOD-270] Fixed a comment.
* [NOD-270] Uncoupled chain updates inside blockdag from the concept of a notification.
* [NOD-270] Removed intermediary ChainUpdates object from ChainChangedNotificationData.
* [NOD-256] Add error log
* [NOD-256] Add error log
* [NOD-256] Fix typo and comment
* [NOD-256] Remove btclog dir
* [NOD-256] Format project
* [NOD-256] Add error log files
* [NOD-256] Add an option to add a log file to write into to an existing backend logger
* [NOD-256] Get rid of redundant logs initialization
* [NOD-256] rename initLogRotators to initLog
* [NOD-256] Get rid ExampleSignTxOutput and convert ExampleBlockDAG_ProcessBlock to a regular test
* [NOD-256] Show error message if os.Exiting from initLog
* [NOD-241] Implement lower resolution peer rendezvous point discovery
* [NOD-241] Implement lower resolution peer rendezvous point discovery
* [NOD-241] Find exact rendezvous point
* [NOD-241] Find exact rendezvous point
* [NOD-241] Fix tests
* [NOD-241] Remove hash stop from MsgBlockLocator and add tests to MsgBlockLocator and MsgGetBlockLocator
* [NOD-241] Change everywhere startHash to hashStart and change comments
* [NOD-241] Fix locateBlockNodes to stop at hashStop
* [NOD-241] Formatted locatorSummary.
* [NOD-241] Fix node reversal
* [NOD-241] Fix hash start and hash stop order, and don't include startNode in dag.blockLocator
* [NOD-241] rename locateBlockNodes -> getBlueBlocksBetween and add a comment to it
* [NOD-241] change hash start to start hash and hash stop to stop hash
* [NOD-241] Move block locator stuff to a different file
* [NOD-241] Rename msggetblocks.go to msggetblockinvs.go
* [NOD-241] Format project
* [NOD-241] Rename rpcserverSyncManager.LocateHeaders to GetBlueBlocksHeadersBetween
* [NOD-241] Move the logic of finding the highest shared block to OnBlockLocator
* [NOD-241] Rename chainHeight -> nextChainHeight
* [NOD-241] Fix typo in comment
* [NOD-264] Implemented calcTxSelectionValue.
* [NOD-264] Fixed bad subnetworkID in calcTxSelectionValue.
* [NOD-264] Implemented sorting the txDescs by value.
* [NOD-264] Got rid of txPrioItem.
* [NOD-264] Moved transaction selection to a separate file.
* [NOD-264] Renamed the result object to txsForBlockTemplate.
* [NOD-264] Implemented tx selection.
* [NOD-264] Fixed trying to get the gas limit for built-in subnetworks.
* [NOD-264] Wrote comments where appropriate.
* [NOD-264] Moved calcTxSelectionValue to the mining package. (Non-mining nodes shouldn't be forced to calc selection value for every transaction)
* [NOD-264] Wrote a test for selectTxs.
* [NOD-264] Fixed a comment.
* [NOD-264] Fixed misunderstood test.
* [NOD-264] Added zero fee check. Added a couple more tests.
* [NOD-264] Added probabilistic tests. Fixed a couple of bugs in tx selection.
* [NOD-264] Fixed tests with missing fees.
* [NOD-264] Added a test over a range of txs with different gas/mass.
* [NOD-264] Added expected probability to the rest of the test cases.
* [NOD-264] Tightened bounds in probability test.
* [NOD-264] Fixed values in probabily test.
* [NOD-264] Added a comments for alpha and rebalanceThreshold.
* [NOD-264] Fixed a couple of comments, renamed result to txsForBlockTemplate.
* [NOD-264] Removed an irrelevant comment. Changed Tracef to Warnf in some logs.
* [NOD-264] Renamed selectionValue -> txValue.
* [NOD-264] Moved rebalancing to the start of the tx selection loop.
* [NOD-264] Added overflow check for gasUsage.
* [NOD-264] Renamed blockSigOps and blockMass to totalSigOps and totalMass.
* [NOD-264] Removed the need to pass usedCount to reblanaceCandidates. Also relaxed bounds in a test.
* [NOD-264] Split selectTxs into smaller functions. Also relaxed bounds in a test some more.
* [NOD-264] Added a comment for findTx.
* [NOD-264] Ordered candidateTxs by subnetwork instead of txValue.
* [NOD-264] Disallowed zero tx fees in mempool and config. Renamed iterateCandidateTxs to populateTemplateFromCandidates.
* [NOD-264] Changed isFinalizedTransaction log level from Warn to Debug.
* [NOD-264] Removed references to SigOps in txSelection.
* [NOD-264] Removed SigOps validation. Validating mass should suffice.
* [NOD-264] Renamed wasUsed to isMarkedForDeletion.
* [NOD-264] Renamed markCandidateTxUsed to markCandidateTxForDeletion.
* [NOD-264] Made some probabilistic tests less likely to fail when they shouldn't.
* [NOD-264] Added a message warning people about probabilistic tests.
* [NOD-264] Rephrased a comment about rebalanceThreshold.
* [NOD-264] Removed IsCoinBase, CheckTransactionInputsAndCalulateFee, and ValidateTransactionScripts from txSelection.
* [NOD-264] Removed a condition that is no longer relevant.
* [NOD-264] "which's" -> "whose"
* [NOD-264] Removed wasteful preallocations.
* [NOD-264] Fixed a comment referring to "used" transactions.
* [NOD-269] Added a skeleton for getChainFromBlock.
* [NOD-269] Made startHash and includeBlocks optional.
* [NOD-269] Implemented chainBlock collection.
* [NOD-269] Extracted GetBlockVerboseResult building to its own method.
* [NOD-269] Implemented the IncludeBlocks part of GetChainFromBlock.
* [NOD-269] Added a comment for NewGetChainFromBlockCmd.
* [NOD-269] Made IsInSelectedPathChain return an error.
* [NOD-269] Fixed a very wrong comment.
* [NOD-269] Made SelectedPathChain allocate only the required amount of space.
* [NOD-269] Renamed pathChain to parentChain.
* [NOD-269] Split handleGetChainFromBlock to separate functions.
* [NOD-269] Fixed some grammar.
* [NOD-234] Added an IsCurrent check to handleGetBlockTemplate.
* [NOD-234] Removed IsCurrent check from handleGetBlockTemplateRequest. Added an explanation for why we're checking the chainHeight.
* [NOD-234] Added ShouldMineOnGenesis to the IsCurrent check.
* [NOD-234] Flipped && operands to fail fast.
* [NOD-240] Removed references to the wallet in rpcserver.go.
* [NOD-240] Began removing btcwalletxxx.go.
* [NOD-240] Got rid of rpcclient/wallet.go and walletsvrcmds.go.
* [NOD-240] Moved GetBestBlockResult to dagsvrresults.go.
* [NOD-240] Finished removing walletsvrXXX.go.
* [NOD-240] Removed wallet stuff from btcctl.
* [NOD-240] Removed a few last things that I've missed.
* [NOD-255] When orphan blocks arrive from netsync - don't write log unless we are in Debug
* [NOD-255] If there are more than K*2 orphans in pool - report as a potential problem anyway
* [NOD-255] Update comment to explain the K*2 figure
* [NOD-237] Implemented transaction mass.
* [NOD-237] Added transaction mass validation to the mempool.
* [NOD-237] Made blockMaxMassMax not rely on MaxBlockPayload.
* [NOD-237] Added comments describing the new constants in validate.go.
* [NOD-237] Changed the default blockmaxmass to 10,000,000.
* [NOD-237] Fixed a comment that erroneously didn't refer to mass.
* [NOD-237] Added comments to ValidateTxMass and CalcTxMass.
* [NOD-237] Renamed "size" to "byte". Made validateBlockMass exit early if validation fails. Fixed unit names in comments. In CalcTxMass, moved summing of mass to the bottom of the function.
* [NOD-237] Instead of ErrMassTooHigh, renamed ErrBlockTooBig and ErrTxTooBig. Replaced wire.MaxBlockPayload with MaxMassPerBlock.
* [NOD-237] Fixed sanity checks related to block size in commands.
* [NOD-237] To use up less memory during testing, made the mass in the "too big" test come from pkScripts rather than input bytes.
* [NOD-237] Added an overflow check to validateBlockMass.
* [NOD-222] Added constant: UnacceptedBlueScore.
* [NOD-222] Made it so that block transactions always have UnacceptedBlueScore.
* [NOD-222] Implemented updating unaccepted UTXO entries with accepted ones in the virtual.
* [NOD-222] Fixed an unclear comment.
* [NOD-222] Fixed diffFromAcceptanceData not receiving the right blue score.
* [NOD-222] Fixed various issues with the implementation. It appears to work now.
* [NOD-222] Removed debug logs.
* [NOD-222] Fixed tests that relied on utxoCollection.String().
* [NOD-222] Fixed TestChainedTransactions.
* [NOD-222] Fixed tests that relied on GetVirtualFromParentsForTest.
* [NOD-222] Fixed having identical entries in toAdd and toRemove.
* [NOD-222] Fixed logic in diffFrom that I previously broke.
* [NOD-222] Fixed a wrong check.
* [NOD-222] Figured out the magical invocation to make everything work.
* [NOD-222] Fixed blockDB tests.
* [NOD-222] Removed debug method.
* [NOD-222] Fixed comments related to setting coinbase maturity to 0.
* [NOD-222] Fixed a typo in a comment.
* [NOD-222] Added a comment that explains the new addition in GetVirtualFromParentsForTest.
* [NOD-222] Added a comment to DiffUTXOSet.Get().
* [NOD-222] Fixed a nuance in DiffUTXOSet.containsInputs.
* [NOD-222] Replaced nonsense in GetVirtualFromParentsForTest with diffFromAcceptanceData.
* [NOD-222] Renamed newVirtualUTXO -> newVirtualPastUTXO.
* [NOD-222] Fixed a comment.
* [NOD-222] Extracted checking utxoCollection with blueScore to a method.
* [NOD-222] Added tests where the same entry is in both toAdd and toRemove.
* [NOD-222] Used Add/RemoveEntry inside diffFromAcceptedTx.
* [NOD-222] Removed superfluous test for UnacceptedBlueScore.
* [NOD-222] Added/Updated comments.
* [NOD-222] Added tests to TestUTXODiffRules.
* [NOD-222] Added appropriate protection against impossible "from"s in diffFrom.
* [NOD-222] Added a comment explaining why we diffFrom acceptanceData in verifyAndBuildUTXO.
* [NOD-222] Fixed comments and equal() in utxoset.
* [NOD-225] Finalize nodes below finality point
* [NOD-225] finalizeNodesBelowFinalityPoint only if dag.lastFinalityPoint is changed
* [NOD-225] change comment in validateParents
* [NOD-225] add string to ErrInvalidParentsRelation error
* [NOD-225] Change comment in validateParents
* [NOD-225] Change comment in validateParents
* [NOD-225] change comment in validateParents
* [NOD-225] Delete diff data from db directly from finalizeNodesBelowFinalityPoint
* [NOD-225] Refactor updateFinalityPoint
* [NOD-208] Added blockBlueScore to UTXOEntry.
* [NOD-208] Added blueBlockScore to NewUTXOEntry.
* [NOD-208] Fixed compilation errors in policy, utxoset, and dag tests.
* [NOD-208] Changed validateBlockRewardMaturity and CheckTransactionInputsAndCalulateFee to use blueScore.
* [NOD-208] Changed CalcBlockSubsidy to use blueScore.
* [NOD-208] Changed SequenceLockActive to use blueScore.
* [NOD-208] Removed ExtractCoinbaseHeight.
* [NOD-208] Removed reference to block height in ensureNoDuplicateTx.
* [NOD-208] Changed IsFinalizedTransaction to use blueScore.
* [NOD-208] Fixed merge errors.
* [NOD-208] Made UTXOEntry serialization use blueScore.
* [NOD-208] Changed CalcPriority and calcInputValueAge to use blueScore.
* [NOD-208] Changed calcSequenceLock to use blueScore.
* [NOD-208] Removed blockChainHeight from UTXOEntry.
* [NOD-208] Fixed compilation errors in feeEstimator. Fixed a bug in the test pool hardness.
* [NOD-208] Fixed oldestChainBlockWithBlueScoreGreaterThan not handling an extreme case.
* [NOD-208] Fixed TestDiffFromTx.
* [NOD-208] Got rid of priority and support of free transactions.
* [NOD-208] Fixed TestProcessTransaction.
* [NOD-208] Fixed TestTxFeePrioHeap.
* [NOD-208] Fixed TestAddrIndex and TestFeeEstimatorCfg.
* [NOD-208] Removed unused rateLimit parameter from ProcessTransaction.
* [NOD-208] Fixed tests that rely on CreateTxChain.
* [NOD-208] Fixed tests that rely on CreateSignedTxForSubnetwork.
* [NOD-208] Fixed TestFetchTransaction.
* [NOD-208] Fixed TestHandleNewBlock. Fixed HandleNewBlock erroneously processing fee transactions.
* [NOD-208] Fixed TestTxIndexConnectBlock.
* [NOD-208] Removed the use of Height() from the fee estimator.
* [NOD-208] Removed unused methods from rpcwebsocket.go.
* [NOD-208] Removed Height from util.Block.
* [NOD-208] Removed ErrForkTooOld. It doesn't make sense in a DAG.
* [NOD-208] Made blockHeap use blueScore instead of height.
* [NOD-208] Removed fee estimator.
* [NOD-208] Removed DAG.Height.
* [NOD-208] Made TestAncestorErrors test chainHeight instead of height.
* [NOD-208] Fixed a couple of comments that were still speaking about block height.
* [NOD-208] Replaced all uses of HighestTipHash with SelectedTipHash.
* [NOD-208] Remove blockNode highest and some remaining erroneous uses of height.
* [NOD-208] Fixed a couple of comments. Fixed outPoint -> outpoint merge error.
* [NOD-208] Fixed a couple more comments.
* [NOD-208] Used calcMinRequiredTxRelayFee instead of DefaultMinRelayTxFee for mempool tests.
* [NOD-208] Renamed mempool Config BestHeight to DAGChainHeight.
* [NOD-208] Fixed a bug in oldestChainBlockWithBlueScoreGreaterThan. Made calcSequenceLock use the node's selected parent chain rather than the virtual block's.
* [NOD-208] Removed chainHeight from blockNode String().
Renamed checkpointsByHeight to checkpointsByChainHeight and prevCheckpointHeight to prevCheckpointChainHeight.
Removed reference to chainHeight in blockIndexKey.
Fixed comments in dagio.go.
* [NOD-208] Removed indexers/blocklogger.go, as no one was using it.
* [NOD-208] Made blocklogger.go log blueScore instead of height.
* [NOD-208] Fixed typo.
* [NOD-208] Fixed comments, did minor renaming.
* [NOD-208] Made a "common sense" wrapper around sort.Search.
* [NOD-208] Fixed comment in SearchSlice.
* [NOD-213] Add customization to txgen
* [NOD-213] Add fee rate as an argument
* [NOD-213] Don't delay transaction emission if there's no need
* [NOD-213] enqueueTransactions -> queueTransactions
* [NOD-213] reuse delay variable
* [NOD-213] Add ExtractGasLimit function
* [NOD-213] Use time.Ticket in sendTransactionLoop
* [NOD-195] Made fee tx implicit
* [NOD-195] Removed redundant checks for fee transactions
* [NOD-195] Add fee tx data into acceptence data and fee data
* [NOD-195] Fix some tests
* [NOD-195] Update Block100000 with new data
* [NOD-195] Fixed remaining tests
* [NOD-195] Save and load feeTx to/from database
* [NOD-195] Remove DisconnectBlock methods from indexers, since they are not used anywhere
* [NOD-195] Add fee tx to addrindex
* [NOD-195] Don't populate inputs for fee transactions
* [NOD-195] Delete feeTxBucket in removeDAGState
* [NOD-195] Got rid of util.FeeTRansactionIndex
* [NOD-201] Implemented the AddSubnetwork CLI tool.
* [NOD-201] Fixed various bugs in AddSubnetwork.
* [NOD-201] Fixed mempool maybeAcceptTransaction verifying gasLimit for a subnetwork registry transaction.
* [NOD-201] Fixed serialization/deserialization bugs in addrIndex.
* [NOD-201] Fixed BlockConfirmationsByHash not handling the zeroHash.
* [NOD-201] Used btclog instead of go log.
* [NOD-201] Made gasLimit a command-line flag. Made waitForSubnetworkToBecomeAccepted only return an error.
* [NOD-201] Filtered out mempool transactions.
* [NOD-201] Fixed embarrassing typos.
* [NOD-201] Added subnetwork registry tx fee + appropriate cli flag.
* [NOD-201] Skipped TXOs that can't pay for registration.
* [NOD-192] Add method to compute confirmations of a single transaction (#306)
* [NOD-192] Implemented txConfirmations.
* [NOD-192] Renamed acceptedBy -> acceptingBlock and ConfirmationsByHash -> BlockConfirmationsByHash.
* [NOD-194 + NOD-199] Update all JSON-RPC methods to use new methods for computing confirmations + Remove the x1.5 factor when counting confirmations in txgen (#309)
* [NOD-194] Connected JSON-RPC commands with new confirmations logic.
* [NOD-194] Fixed failing tests.
* [NOD-194] Removed x1.5 from isTxMatured.
* [NOD-194] Made isTxMatured panic if it receives nil confirmations.
* [NOD-194] Added isInMempool to RPC methods that require it.
* [NOD-194] Fixed a typo.
* [NOD-194] Made the declaration of isInMempool more clear.
* [NOD-194] Removed some unnecessary complexity from isTxMatured.
* [NOD-193] Update Tx-Index to accomodate correct Confirmations structure (#308)
* [NOD-193] Uploaded BlockID to be uint64 in txIndex and addrIndex.
* [NOD-193] Removed the inclusion of current block transactions to txsAcceptanceData.
* [NOD-193] Implemented writing to the tx index txs with the virtual as the accepting block.
* [NOD-193] Added test for txs accepted by the virtual block.
* [NOD-193] Removed the requirement for subnetwork registry transactions to be accepted.
* [NOD-194] Made in-memory the txsAcceptedByVirtual part of txIndex.
* [NOD-193] Optimized txsAcceptedByVirtual initialization.
* [NOD-193] Fixed weird loop in txsAcceptedByVirtual initialization.
* [NOD-190] Fixed merge errors.
* [NOD-189] Made UTXODiff WithDiff and DiffFrom allocate collections with appropriate sizes.
In mempool HandleNewBlock, Replaced removeTransaction loop with removeTransactions.
* [NOD-189] Removed code duplication between removeTransaction and removeTransactions.
* [NOD-189] Fixed a merge error.
* [NOD-189] Fixed another merge error.
* [NOD-189] Renamed removeRedeemers to removeDependants.
* [NOD-189] Removed superfluous check inside removeTransactionWithDiff.
* [NOD-189] Added a comment to removeTransactions detailing what it optimizes.
* [NOD-189] Added documentation to removeTransactionWithDiff and split it into smaller methods.
* [NOD-172] Port EMCH from bchd
* [NOD-172] Fix hdkeychain.TestErrors and add btcec.TestRecoverCompact
* [NOD-172] Make ECMH immutable
* [NOD-172] Fix gofmt errors
* [NOD-172] Add TestMultiset_NewMultisetFromDataSlice and fix Point to be immutable
* [NOD-172] Fix gofmt errors
* [NOD-172] Add test for checking that the Union of a multiset and its inverse is zero
* [NOD-179] Add ECMH Point to all UTXO-structs
* [NOD-179] Fix utxo set tests
* [NOD-179] Fix mempool tests
* [NOD-179] Remove RemoveTxOuts
* [NOD-179] Move serializeBlockUTXODiffData to the top of the file
* [NOD-179] Fix serializeBlockUTXODiffData comment format
* [NOD-179] Fix AddTx comment and name return values
* [NOD-180] Validate utxo commitments
* [NOD-179] Fix TestAcceptingBlock and TestConfirmations to not use the block hash as phantom break even
* [NOD-180] Fix typo
* [NOD-180] move most of the logic in calcUTXOCommitment to UTXOSet.WithTransactions
* [NOD-180] Optionally return error when a transaction in WithTransactions is double spent
* [NOD-180] Rename allowDoubleSpends to ignoreDoubleSpends
* [NOD-172] Port EMCH from bchd
* [NOD-172] Fix hdkeychain.TestErrors and add btcec.TestRecoverCompact
* [NOD-172] Make ECMH immutable
* [NOD-172] Fix gofmt errors
* [NOD-172] Add TestMultiset_NewMultisetFromDataSlice and fix Point to be immutable
* [NOD-172] Fix gofmt errors
* [NOD-172] Add test for checking that the Union of a multiset and its inverse is zero
* [NOD-179] Add ECMH Point to all UTXO-structs
* [NOD-179] Fix utxo set tests
* [NOD-179] Fix mempool tests
* [NOD-179] Remove RemoveTxOuts
* [NOD-179] Move serializeBlockUTXODiffData to the top of the file
* [NOD-179] Fix serializeBlockUTXODiffData comment format
* [NOD-179] Fix AddTx comment and name return values
* [NOD-191] Added selectedPathChainSlice to virtualBlock.
* [NOD-191] Implemented acceptingBlock().
* [NOD-191] Implemented confirmations().
* [NOD-191] Added selectedPathChainSlice tests to TestSelectedPath.
* [NOD-191] Fixed a bug in acceptingBlock(). Written tests for confirmations().
* [NOD-191] Written tests for acceptingBlock().
* [NOD-191] Added test to make sure that acceptingBlock(tip) returns the virtual block.
* [NOD-191] Added a panic if we somehow feed a childless block that isn't the virtual to acceptingBlock.
* [NOD-191] Fixed comments.
* [NOD-191] Fixed a bug in acceptingBlock. Added red block tests for acceptingBlock.
* [NOD-191] Added red block tests for confirmations.
* [NOD-191] Fixed misleading comment and error message.
* [NOD-175] Added BlockAddedNotificationData and sent it instead of just a block on BlockAdded.
* [NOD-175] Added BFWasUnorphaned and raised it when an unorphaned block was to be accepted.
* [NOD-175] Fixed a typo.
* [NOD-175] Made it so that only the mempool gets updated if we're not current or the block was just now unorphaned.
* [NOD-177] Removed references to idMerkleRoot.
* [NOD-177] Generated new genesis hashes.
* [NOD-177] Generated new blk_ blocks.
* [NOD-177] Fixed TestHaveBlock.
* [NOD-177] Fixed The rest of the tests.
* [NOD-177] Fixed a couple of comments and a duplicate test.
* [NOD-177] Fixed blocks1-256.bz2.
* [NOD-172] Port EMCH from bchd
* [NOD-172] Fix hdkeychain.TestErrors and add btcec.TestRecoverCompact
* [NOD-172] Make ECMH immutable
* [NOD-172] Fix gofmt errors
* [NOD-172] Add TestMultiset_NewMultisetFromDataSlice and fix Point to be immutable
* [NOD-172] Fix gofmt errors
* [NOD-172] Add test for checking that the Union of a multiset and its inverse is zero
* [NOD-164] Added validation routine
* [NOD-167] Extracted acceptedIDMerkleRoot calculation to its own method and implemented NextAcceptedIDMerkleRoot.
* [NOD-164] Fixed TestValidateFeeTransaction.
* [NOD-164] Fixed TestFinality.
* [NOD-164] Fixed blk_ tests.
* [NOD-164] Fixed if -> iff in a comment.
* [NOD-164] Minor style changes in comments.
* [NOD-164] Moved validateAcceptedIDMerkleRoot to before its population with the block's own transactions.
Replaced heavy call to verifyAndBuildUTXO in NextBlockFeeTransaction and NextAcceptedIDMerkleRoot with a call to pastUTXO on the virtual.
* [NOD-164] Fixed erroneous comment.
* [NOD-164] Inserted the logic from buildAndSortAcceptedTxs into calculateAcceptedIDMerkleRoot, since the former was meaningless on its own.
* [NOD-164] Changed looping over txsAcceptanceData instead of over node.blues.
* [NOD-13] Added notify_telegram to deploy.sh.
* [NOD-13] Made a test fail for testing.
* [NOD-13] Added some temporary logging.
* [NOD-13] Wrote a nice message for the bot to send.
* [NOD-13] Made the message nicer.
* [NOD-13] Made the message nicer still.
* [NOD-13] Added the build log as an attachment.
* [NOD-13] Actually added the build log as an attachment.
* [NOD-13] Added a delay to allow the build log to properly flush.
* [NOD-13] Disowned notify_telegram.
* [NOD-13] Disowning doesn't work. Using the "at" command instead.
* [NOD-13] Properly using the at command.
* [NOD-13] Actually properly using the at command.
* [NOD-13] Added a couple of prints to see whether the script is even being called.
* [NOD-13] More printouts...
* [NOD-13] Added a command to start atd if it stopped for some reason.
* [NOD-13] Added slashes in multiline echo command.
* [NOD-13] Added quotes where required and removed debug comments.
* [NOD-13] Revert "[NOD-13] Made a test fail for testing."
This reverts commit 9701e30e
* [NOD-13] Added some comments.
* [NOD-144] Use chainHeight in SelectedAncestor, and update all logic that uses it
* [NOD-144] Moved UnminedHeight to blockdag, and updated all references
* [NOD-96] Convert txid to pointer where possible
* [NOD-96] Make msgTx.TxID return a pointer
* [NOD-96] observedTransaction.id -> observedTransaction.txID
* [NOD-152] Stopped pushBlockMsg from sending tip inv to syncing nodes.
* [NOD-152] Fixed restartSyncIfNeeded not restarting if sync is needed.
* [NOD-152] Removed continueHash, as it is no longer required.
* [NOD-140] Converted DNSSeeder to use btclog.
* [NOD-140] Converted MiningSimulator to use btclog.
* [NOD-140] Converted TxGen to use btclog.
* [NOD-140] Fixed log level in handlePanic in txgen.
* [NOD-140] Renamed logger to log everywhere. Removed superfluous flag-setting to go-log.
* [NOD-139] Made processOrphans not turn an error if one of the parents is still missing.
* [NOD-139] Made addOrphanBlock and removeOrphanBlock process all parents instead of only the selected parent.
* [NOD-139] Made addOrphanBlock remove excess orphans by their timestamp rather than their discovery time. Fixed orphans being added more than once.
* [NOD-139] Simplified removal from slice in removeOrphanBlock.
* [NOD-139] Made check for no-orphans-left come before assignment to prevOrphans.
* [NOD-139] Added Timestamp() to util.Block.
* [NOD-139] Fixed merge errors.
* [NOD-149] Store phantom values in the database
* [NOD-149] Explain when zero hash is used when serializing blockNode
* [NOD-149] make deserializeBlockNode return a blockNode
* [NOD-149] use blockNode initializer instead of lots of assignments
* [NOD-142] Updated util.FastLog2Floor to work on uint64
* [NOD-142] Convert height and chainHeight to uint64
* [NOD-142] A couple fixes in comments of TestFastLog2Floor
* [NOD-142] Make spendableOutOffset uint64 too
* [NOD-131] Allow override of dnsseed by command line or config
* [NOD-131] Moved tor.go from connmgr to util/network, to prevent dependancy loop
* [NOD-131] Typo fix
* [NOD-131] Clarify description for --dnsseed cli flag
* [NOD-131] Removed redundant line that somehow got into go.sum
* [NOD-121] Do not handle transaction inputs for reward transactions in SearchRawTransactions RPC call
* [NOD-121] Do not get transaction inputs for fee transactions in SearchRawTransactions RPC call
* [NOD-66] Created TX generator
* [NOD-66] Created transaction generator
* [NOD-66] Improved TX generator against double spend. Created genaddr utility. Refactored
* [NOD-66] Save chenges before branch switch
* [NOD-66] Use log package instead of fmt
* [NOD-66] Fixed/restored docker files
* [NOD-66] Changed according to new WithLock/NoLock convention
* [NOD-122] Handle each message in rpcclient with a separate goroutine
* [NOD-122] Stop listening to new blocks when not mining
* [NOD-122] Made RPC logging in mining simulator more explicit + some styling enhencement
* [NOD-115] Add timeout to rpcclient requests
* [NOD-115] Add timeout of half a second to mining simulator requests
* [NOD-115] Remove redundant allocation of responseChan
* [NOD-114] Added a minimum address amount GetAddrs.
* [NOD-114] Added smallNetwork intervals for when the network is small.
* [NOD-114] Fixed bad minimum address calculation.
* [NOD-95] Added docker file and scripts to run dev instances for debugging
* [NOD-95] Enabled debugging from dockers
* [NOD-95] Removed redundant entrypoint.sh file
* [NOD-95] Removed multi-stage build artifacts from Dockerfile.dev
* [NOD-95] Add --help to run-dev.sh
* [NOD-82] Added RPC files to the BTCD Docker container.
* [NOD-82] Generated new cert and key files that include *.daglabs.com.
* [NOD-82] Added a dagConfig parameter for accepting unroutable IPs.
* [NOD-83] Modify block parents' children only after validation
* [NOD-83] Creating CheckConnectBlockTemplateWithLock with RLock
* [NOD-83] Put updateParentsChildren inside updateParents
* [NOD-83] create updateParentsDiffs function
* [NOD-58] Replace lastBlock with selected tip in version message (#210)
* [NOD-58] Replace lastBlock with selected tip in version message
* [NOD-58] Fix typo in comment
* [NOD-58] Add mutex to SelectedTipHash
* [NOD-58] Remove redundant comment
* [NOD-58] Remove wantStartingHeight from peerStats
* [NOD-58] Remove lock from SelectedTipHash
* Nod 53 change getheaders message to handle new block locator (#213)
* [NOD-53] Change getheaders message to handle the new block locator mechanism
* [NOD-53] Use heap in locateHeaders
* [NOD-53] Create a constructor for each heap direction
* [NOD-57] Check if a node is synced only by timestamps (#214)
* [NOD-60] implement isSyncCandidate (#218)
* [NOD-60] Implement isSyncCandidate
* [NOD-60] Fix typo
* [NOD-65] Fix netsync related tests and remove fields optionality from… (#220)
* [NOD-65] Fix netsync related tests and remove fields optionality from msgversion
* [NOD-65] gofmt rpcserver.go
* [NOD-65] add missing test for verRelayTxFalse
* [NOD-62] Change getblocks message to handle the new block locator mechanism (#219)
* [NOD-62] Change getblocks message to handle the new block locator mechanism
* [NOD-62] Add locateBlockNodes function
* [NOD-68] Adjust orphan parents requesting for a DAG (#222)
* [NOD-68] Adjust orphan parents requesting for a DAG
* [NOD-68] add sendInvsFromRequestedQueue and trigger it when requested blocks slice is empty, or immediatly if we're not in sync mode
* [NOD-68] Prevent duplicates from entering to state.requestQueue and add wrapping locks to addBlocksToRequestQueue
* [NOD-68] Fix Lock -> Unlock in sendInvsFromRequestedQueue
* [NOD-74] Starts syncing again when the current sync peer is done (#225)
* [NOD-74] Starts syncing again when the current sync peer is done
* [NOD-74] Unlock mtx before netsync is restarted
* [NOD-74] Fix name isSyncPeerFree -> isWaitingForBlocks
* [NOD-75] fixing netsync bugs (#227)
* [NOD-74] Starts syncing again when the current sync peer is done
* [NOD-74] Unlock mtx before netsync is restarted
* [NOD-75] Fixing netsync bugs
* [NOD-80] Request block data from block propagation just after you are… (#231)
* [NOD-80] Request block data from block propagation just after you are current
* [NOD-80] Fix adding to both queues in addInvToRequestQueue
* [NOD-81] Start to mine on top of genesis iff all peers selected tip is genesis (#232)
* [NOD-81] Start to mine on top of genesis only if all of your peers' selected tip is genesis
* [NOD-81] Explain forAllPeers/forAllOutboundPeers shouldContinue behaviour in comments
* [NOD-81] Add forAllInboundPeers and add return values for forAllPeers/forAllOutboundPeers/forAllInboundPeers functions
* [NOD-16] Add pushSet to the BlockHeap type
* [NOD-16] Fixed syntax error
* [NOD-78] Use transfer some read-only functions from dagLock.Lock to dagLock.RLock
* [NOD-78] Make mining.NewBlockTemplate lock the dag up until the point it calls CheckConnectBlockTemplate
* [NOD-78] Removed locking from functions that are concurrency-safe, and added WithLock version to those that are not
* [NOD-78] Remove unused isLockHeld from CalcNextBlockVersion
* [NOD-78] Renamed UTXORLock/UTXORUnlock to RLock/RUnlock
* [NOD-78] Revesed dagUnlocket to isDagLocked
* [NOD-77] AddrManager.Good now updates subnetwork in case it was modified
* [NOD-77] Do not update subnetworkID in updateAddr if address already known
* [NOD-77] Restructure case where ka.tried = true for more readable code
* [NOD-77] Some corrections to comments
* [NOD-77] Fixd typo
* [NOD-70] Added GetBlockTemplate method to rpcclient
* [NOD-70] Basic infrastructure for mining simulator
* [NOD-70] Fix txFees in NewBlockTempalte: include value for fee transaction + don't set fee for coinbase = -totalFees.
* [NOD-70] Added capabilities parameter to Client.GetBlockTemplate call
* [NOD-70] Dirty version of mining simulator complete
* [NOD-70] cleaned up mining simulator
* [NOD-70] Added dockerfile to mining simulator
* [NOD-70] Updated base config path of mining simulator to mining_simulator
* [NOD-70] Remove error return from msgblock.AddTransaction - it never returns one
* [NOD-70] Renamed r -> random
* [NOD-70] Move paths initialization of mining simulator to main
* [NOD-70] Cleaned up mining simulator dockerfile
* [NOD-70] Add '--' to tini argument
* [NOD-56] Created devnet
* [NOD-56] Fixed tests
* [NOD-56] Fixed go vet errors
* [NOD-56] Added TestSolveGenesisBlock function
* [NOD-56] Created command line tool for solving genesis blocks
* [NOD-63] Merge BlockAccepted and BlockConnected notifications into BlockAdded + remove BlockDisconnected notifications
* [NOD-63] Many instances of chain->DAG and similar
* [NOD-63] Some more chian -> DAG
* [NOD-48] Update wire.NewMsgTx to recieve all fields in msgTx
* [NOD-48] Fix all compilation errors resulting from modification of wire.NewMsgTx
* [NOD-48] Calculate payloadHash iff subnetworkID is not native
* [NOD-48] Update all places the instantiate wire.MsgTx to use wire.NewMsgTx
* [NOD-48] Remove 'wire.' calls inside wire package
* [NOD-48] Made newMsgTx with all parameters private, and added a few public functions that take various arguments for all common use-cases
* [NOD-48] Explicitly pass SubnetworkIDNative instead of nil to newMsgTx
* [NOD-48] Remove option to pass nil to newMsgTx
* [NOD-42] Split checkFinalityRules and newFinalityPoint
* [NOD-42] Rename connectToDAG -> addBlock + move anything that is not actually connecting block to DAG out of connectBlock
* [NOD-42] Extract methods from PastUTXO
* [NOD-42] Give names to outputs in verifyAndBuildUTXO and propagate name
changes up the call tree
* [NOD-42] Split loop that creates UTXODiff and updates acceptance data into 2 separate methods
* [NOD-42] Removed from applyUTXOChanges any validation logic, moved in any logic related to updating the DAG and renamed to applyDAGChanges
* [NOD-42] Rename: CheckTransactionInputs -> CheckTransactionInputsAndCalculateFee
* [NOD-42] Revise some comments
* [NOD-42] Removed finalityErr constant - it's not needed
* Multiple chain -> dag corrections in comments
* [NOD-42] Removed redundant declaration of feeData
* [NOD-42] Reworded some comments
* [NOD-42] Rename MultiblockTxsAcceptanceData -> MultiBlockTxsAcceptanceData
* [NOD-26] Zero payload for signature hash calculation
* [NOD-26] Zero payload in shallowCopyTx function
* [NOD-26] Zero payload in calcSignatureHash function
* [NOD-26] Updated comment
* [NOD-46] Immediately associate received address to their subnetwork
* [NOD-46] fix comment
* [NOD-46] Disconnect peer if it sends an MsgAddr with no subnetwork
* [NOD-46] Disconnect peer if it sends an MsgAddr with incorrect subnetwork id
* [NOD-46] add parenthesis to condition
* [NOD-46] change order of conditions
* [DEV-379] Refactored checkBlockContext: Mainly extracted methods and re-organized variable use to minimize clutter
* [DEV-379] Simplify the condition according to which we increment blockCount
* [DEV-379] Move all logic to save new block data to separate method
* [DEV-379] Refactored the checking of finality point
* [DEV-379] Minor styling changes
* [DEV-379] Extracted method in applyUTXOChanges subroutines
* [NOD-41] Moved update of finality point to after block was validated + some minor style fixes
* [NOD-41] call dag.checkFinalityRulesAndGetFinalityPoint(node) even if fastAdd
* [NOD-41] Fix in a comment
* [NOD-41] Some methods of dag that could have been just functions converted to function
* [DEV-363] Added testnet seeders
* [DEV-363] Prepare BTCD to work with DNS seeder
* [DEV-363] Finished BTCD part of DNS seeder
* [DEV-363] Reverted protocol changes
* [DEV-363] Reverted protocol changes
* [DEV-363] Reverted protocol changes and fixed tests
* [DEV-363] Added tests and refactored
* [DEV-363] Small refactoring of p2p changes
* [DEV-363] Update NeedMoreAddresses function to take into account full peers + small refactoring
* [DEV-363] Removed IsDNSSeederPeer flag
* [DEV-363] Fixed comment
* [DEV-378] Added feeAccumulator structures
* [DEV-378] Added logic to create and store fee data when validating a block
* [DEV-378] Renamed feeAccumulator to compactFeeData, and all related entities accordingly
* [DEV-378] Converted MsgTx.TxHash() to pointer to hash and not hash
* [DEV-378] Restructured parameters to buildFeeTransaction and related entities
* [DEV-378] Finished the code that calculates fees for blocks
* [DEV-378] Fix TxIndex after changing the structure of AcceptedTxsData
* [DEV-378] For genesis block: Return empty AcceptedTxsData instead of nil
* [DEV-378] Off-by-one error
* [DEV-378] Length of compactFeeData should be determined by specific method, not
* [DEV-378] Multiple bugfixes in tx fee calculation
* [DEV-378] Calculate fee even if fastAdd, to save feeData
* [DEV-378] use IsEqual instead of == when comparing TxHash
* [DEV-378] txindex: if including block is the new block - don't fetch id from DB
* [DEV-378] Fixed a few typos and made some vars consts
* [DEV-378] Re-organized fee functions, removed redundant functions and constants, and revised a few comments
* [DEV-378] Recovered fmt string changes lost in merge
* [DEV-378] Renamed acceptedTxsData and related types and vars to txsAcceptanceData
* [DEV-378] Some comment fixes
* [DEV-378] Remove redundant .ToString()
* [DEV-376] Changed any instance of %v in format strings with a more specific format token
* [DEV-376] Fixed some more wrong formatting strings + removed redundant
cast
* [DEV-376] Added fmt.Sprintf where it was missing
* [DEV-376] use %s for util.Amount, to invoke .String()
* [DEV-376] Some more fixes in format strings
* [DEV-376] fixed mruinvmap_test to expect the correct behaviour
* [DEV-259] Allow to spend genesis coinbase, and use ProcessBlock to add genesis to the DAG like any other block
* [DEV-259] fix IsCurrent to check genesis timestamp if needed
* [DEV-259] add genesisPastUTXO as separate function
* [DEV-316] Update JSON-RPC API to include new transaction fields
* [DEV-316] Fix tests
* [DEV-316] Add txAcceptedVerbose test with subnetwork, gas and paylaod
* [DEV-364] Add fee transactions validation
* [DEV-364] make NextBlockFeeTransactions for creating block templates
* [DEV-364] apply coinbase rules to fee transaction is some cases
* [DEV-364] Add comments
* [DEV-364] put getTXO as separate function
* [DEV-364] Make getParentsFeeData a separate function
* [DEV-364] fix calculateFees
* [DEV-364] force maximum sequence for fee transactions
* [DEV-364] add TestValidateFeeTransactions
* [DEV-364] change fee transaction to be one tx per block rather than one tx for each blue
* [DEV-364] fix tests
* [DEV-364] Use constants instead of inline numbers
* [DEV-336] Split addresses in address manager by subnetwork id
* [DEV-336] Load DAG with subnetwork from the config file
* [DEV-336] Remove redundant checks in updateAddrNew and updateAddrTried
* [DEV-92] Covered blocknode.go with tests.
* [DEV-92] Added test for blockSet.highest. Fixed a bug in it.
* [DEV-92] Added tests for blockSet.subtract and blockSet.addSet.
* [DEV-92] Covered blockset.go with tests.
* [DEV-92] Got rid of some old stuff related to STXOs.
* [DEV-92] Covered error.go with tests.
* [DEV-92] Covered utxoSet with tests.
* [DEV-92] Fixed formatting.
* [DEV-361] Create type TxID as alias to daghash.Hash. Use it for transaction IDs
* [DEV-361] Fixed missed renames
* [DEV-361] Removed usage of zeroHash
* [DEV-361] Fixed more missed renames
* [DEV-319] Implemented transaction data sending logic.
* [DEV-319] Implemented NotifyNewTransactions command validation.
* [DEV-319] Reduced some duplication in notifyForNewTx.
* [DEV-319] Renamed a parameter for clarity.
* [DEV-319] Added a test for marshalling/unmarshalling the new varient of notifyNewTransactions.
* [DEV-319] Added a check in handleNotifyNewTransactions to avoid unnecessary validation.
* [DEV-319] Added comments to explain the initialization of marshalledJSONVerboseFull and marshalledJSONVerbosePartial.
* [DEV-311] Moved subnetwork storage from directly in DAG to subnetworkStore
* Added gas validation in CheckBlockSanity
* [DEV-311] Add SubnetworkStore to last remaining call for CheckBlockSanity
* [DEV-311] Added subnetworkID to config in TestcheckBlockSanity
* [DEV-311] Moved CheckBlockSanity to be method of BlockDAG, and removed subnetworkStore argument
* [DEV-311] Removed SubnetworkID as parameter to CheckBlockSanity
* [DEV-311] Update gas usage before
* [DEV-311] some chain=>DAG updates in comments
* [DEV-311] Removed remaining dag-specific parameters from checkBlockSanity
* [DEV-339] Handling of dependent transactions in mempool
* [DEV-339] Small fixes after code review
* [DEV-339] Fixed compilation
* [DEV-339] Removed extra loop in addTransaction function
* [DEV-339] Changed addTransaction do not loop on inputs second time
* [DEV-332] Added MerkleProof to MsgBlock and rejected full-node/partial-block type misbehaviors.
* [DEV-332] Fixed merge issues.
* [DEV-332] Got rid of MerkleProof. Turns out we no longer need it.
* [DEV-332] Got rid of NTBlockDisconnected, as no one was ever triggering it. (It was part of reorg)
* [DEV-332] Implemented clearing out the payloads of transactions of outgoing blocks for partial nodes.
* [DEV-332] Extracted ConvertToPartial to its own method. Added a test. Added a condition for converting to a partial block.
* [DEV-332] Fixed bad ConvertToPartial condition.
* [DEV-348] Removed a couple of unused methods.
* [DEV-348] Implemented validating incoming transactions for bad partial transactions.
* [DEV-348] Added a (incomplete) filter for propogation of transactions.
* [DEV-348] Implemented filtering inventory by subnetwork.
* [DEV-348] Fixed broken tests.
* [DEV-348] Added test for non-zero payload partial transactions.
* [DEV-348] Added a comment for Config.SubnetworkID.
* [DEV-348] Fixed formatting.
* [DEV-348] Renamed isRemoteTransactionFull to shouldTxBeFull.
* [DEV-348] Added a check for invalid transaction in maybeAcceptTransaction. Added handling for native networks.
* [DEV-348] Fixed formatting.
* [DEV-348] Fixed a bug in transaction validation.
* [DEV-348] Rephrased a comment.
* [DEV-348] Extracted subnetwork compatibility to a method. Wrote a test for it.
* [DEV-348] Removed an unnecessary check over the native subnetwork.
* [DEV-333] Added subnetwork to local peer version message.
* [DEV-333] Fixed broken references.
* [DEV-333] Added serialization/deserialization of the version message.
* [DEV-333] Added rejection for peers with wrong subnetwork.
* [DEV-333] Fixed bad comment.
* [DEV-333] Renamed ChainParams to DAGParams.
* [DEV-333] Fixed partial nodes disconnecting from full nodes.
* [DEV-333] chain -> DAG
* [DEV-333] Possibly fixed outbound msgVersion rules.
* [DEV-333] Combined the two incompatible subnetwork cases into one if.
* [DEV-333] Reformatted the condition.
* [DEV-338] Remove provisional nodes, and panic in case there's an error after verifying the block is valid
* [DEV-338] Improved deffered blockNode cleanup and cosmetic changes
* [DEV-338] Fixed dag.index.SetStatusFlags for parents + cosmetic changes
* [DEV-338] Fixed build
* [DEV-338] Fixed comments
* [DEV-345] Validate that gas and payload are 0 when required by sub-network
* [DEV-345] Remove check for txOut.Value < 0, since txOut.Value is a uint64
* [DEV-345] Added tests for CheckTransactionSanity
* [DEV-345] Remove checks for Gas and Payload validity in wire.MsgTx.Decode
* [DEV-345] Verify that payload in Gas sub-network is always 8 bytes (uint64).
* [DEV-345] Renamed tstCheck{Script/Rule}Error to check{Script/Rule}Error
* [DEV-345] Improved formatting
* [DEV-343] Made sub-network registry use subNetworkIDs.
* [DEV-343] Removed an unnecessary clone.
* [DEV-343] Renamed buildSubNetworkID to txToSubNetworkID. Broke out of a loop when it was known that no further processing is required. Handled error cases from dbGetNetwork separately from the "not-found" case/
* [DEV-343] Added an error case in GasLimit() for where the sub-network is nil.
* [DEV-343] Fixed return nil instead of err. Used a better way to check whether we should continue checking accepted transactions.
* [DEV-312] Take in account subnetwork's GAS limit, when adding
transactions to block. Try to do that optimally.
* [DEV-312] Fixed GAS overusage calculation
* [DEV-337] Make sure that a transaction that uses more gas than the total allowed for sub-network
* [DEV-337] Moved transaction GAS check to mempool
* [DEV-337] Added Unit test for gas usage in transaction
* [DEV-337] Fixed build
* [DEV-337] Fixed tests stuff
* [DEV-337] Removed TODO comment
* [DEV-326] Worked around a parallelism issue in go test. Fixed a test where a file was being deleted while still open.
* [DEV-326] Worked around a timer-related oddity.
* [DEV-326] Fixed closing a file twice.
* [DEV-326] Using pdb.closed instead of an additional variable.
* [DEV-355] Disallowed mining-related RPC commands from running on any sub-network besides the "all" sub-network.
* [DEV-335] Fixed formatting errors.
* [DEV-335] Fixed some more formatting errors.
* [DEV-335] Changed error type to ErrRPCInvalidRequest and rephrased the error message.
* [DEV-335] Fixed formatting.
* [DEV-335] Removed error from getGenerate.
* [DEV-312] Take in account subnetwork's GAS limit, when adding
transactions to block. Try to do that optimally.
* [DEV-312] Fixed GAS overusage calculation
* [DEV-312] Prepare to merge with master
* [DEV-312] Fix after merging
* [DEV-307] Implemented adding pending registry transactions.
* [DEV-307] Implemented a skeleton for the sub-network registry algorithm.
* [DEV-307] Implemented the serialization/deserialization logic for pending sub-network transactions.
* [DEV-307] Implemented marking sub-network registry transactions as successfully registered.
* [DEV-307] Implemented clearing pending sub-network entries and writing registered sub-networks.
* [DEV-307] Added comments for all the dagio functions.
* [DEV-307] Fixed a couple of bugs and added a test for checking serialization/deserialization of pending sub-network transactions.
* [DEV-307] Implemented getting a registered sub-network's gas limit.
* [DEV-307] Updated sub-network announcement transaction validation rules.
* [DEV-307] Specified what is validated.
* [DEV-307] Added initialization for lastSubNetworkID.
* [DEV-307] Renamed extractValidSubNetworkRegistryTxs to validateAndExtractSubNetworkRegistryTxs to better reflect what it does.
* Dev 303 implement block finality (#139)
* [DEV-303] Implement block finality
* [DEV-303] Add finality tests
* [DEV-303] Make finality tests use maybeAcceptBlock
* [DEV-303] Only check finality rules if we are fastAdd mode
* [DEV-303] replace hasBiggerFinalityScoreThan checks with .finalityScore()
* [DEV-307] Forgot to actually create the sub-network buckets.
* [DEV-307] Wrote a full-flow test for sub-network registry and fixed a couple of bugs that it had uncovered.
* [DEV-307] Took firstSubNetworkID out of dagconfig.Params. It's unnecessary there.
* [DEV-307] Added persistance for dag.lastSubNetworkID.
* [DEV-307] Moved sub-network stuff to their own files. Got rid of firstSubNetworkID. Replaced SubNetworkReservedLast with SubNetworkUnreservedFirst. Added ErrSubNetworkRegistry.
* [DEV-307] Added a newline at the end of dag_test.go.
* [DEV-307] Renamed previousFinalityPoint to initialFinalityPoint.
* [DEV-307] Moved sub-network IO stuff to subnetworks.go and subnetworks_test.go.
* [DEV-307] Merged dbPutRegisteredSubNetworkTx and dbRegisterSubNetwork. Fixed a too-big allocation and a block double-processing bug.
* [DEV-307] Simplified the serialized representation of a slice of transactions.
* [DEV-307] Fixed comments.
* [DEV-331] Disallowed node to run in reserved sub-networks
* [DEV-331] Added test to make sure that constant are not modified without modifying help text
* [DEV-314] Added tests for DisasmPC and DisasmScript
* [DEV-314] Re-wrote TestCheckErrorCondition to cover the whole method
* [DEV-314] Fixed error message
* [DEV-330] Covered all methods in daghash with tests
* [DEV-330] Added tests for NewHashFromString
* [DEV-330] Added test case for TestNewHashFromStr that is not the default daghash.Hash value
* [DEV-303] Implement block finality
* [DEV-303] Add finality tests
* [DEV-303] Make finality tests use maybeAcceptBlock
* [DEV-303] Only check finality rules if we are fastAdd mode
* [DEV-303] replace hasBiggerFinalityScoreThan checks with .finalityScore()
* [DEV-308] Enforce tx order in block to be by sub-network ID
* [DEV-308] Use mutable block in TestCheckBlockSanity
* [DEV-308] Fixed comment
* [DEV-308] Fixed TestCheckBlockSanity: use invalid block instead of modifying good block
* [DEV-308] Changed name of block in TestCheckBlockSanity
* [DEV-301] implement blockNode.chainHeight as the height of the selected parent chain
* [DEV-301] add description to TestChainHeight
* [DEV-301] Delete comment on TestChainHeight and expalain about the dag
* [DEV-257] Include transaction acceptance status in getBlock RPC call
* [DEV-257] change txRawResult-accepted to camel case
* [DEV-257] gofmt
* [DEV-257] change getBlock-acceptedtx help entry to getBlock-acceptedTx
* [DEV-257] delete TxRawResult.accepted and instead create TxRawResult.acceptedBy
* [DEV-81] Overwrite maxOpenFiles for testInterface to force tests to check the LRU-mechanism in openFile
* [DEV-81] Added database.UseLogger test
* [DEV-81] Completed coverage of reconcileDB()
* [DEV-81] Added some tests for dbcache
* [DEV-81] Moved init and UseLogger to separate file to make them more easily-testable + added tests
* [DEV-81] Added tests for deleteFile
* [DEV-81] Added tests to cursor.Delete + made sure it returns error when transaction is not writable
* [DEV-81] Moved database/error_test.go from database_test package to database package + added test for IsErrorCode
* [DEV-81] Added tests for handleRollback error-cases
* [DEV-81] Added tests for cursor.skipPendingUpdates
* [DEV-81] Added tests for various cursor edge-cases
* [DEV-81] tx.putKey no longer returns error, because there is no case when it does
* [DEV-81] Added tests to CreateBucket error cases
* [DEV-81] Added tests to bucket.Get and .Delete error cases + .Delete now returns error on empty key
* [DEV-81] Added test for ForEachBucket
* [DEV-81] Added tests to StoreBlock
* [DEV-81] Added test for deleting a double nested bucket
* [DEV-81] Removed log_test, as it is no longer necessary with the logging system re-design
* [DEV-81] Added test to some of writePendingAndCommit error-cases
* [DEV-81] Update references from btcutil to btcd/util
* [DEV-81] Add tests for dbCacheIterator{.Next(), .Prev(), .Key, .Value()} in cases when iterator is exhausted
* [DEV-81] Added tests for ldbIterator placeholder functions
* [DEV-81] Added test name to Error messsages in TestSkipPendingUpdates
* [DEV-81] Begin writing TestSkipPendingUpdatesCache
* [DEV-81] Added error-cases for DBCache.flush() and DBCache.commitTreaps()
* [DEV-81] Use monkey.patch from bou.ke and not from github
* [DEV-81] Rewrote IsErrorCode in both database and txscript packages to be more concise
* [DEV-81] Rename any database.Tx to dbTx instead of tx - to remove confusion with coin Tx
* [DEV-81] Fix typo
* [DEV-81] Use os.TempDir() instead of /tmp/ to be cross-platform
* [DEV-81] use SimNet for database tests + Error if testDB exists after deleting it
* [DEV-81] Removed useLogger - it's redundant
* [DEV-81] Added comment on how CRC32 checksums are calculated in reconcile_test.go
* [DEV-81] Added comment that explains what setWriteRow does
* [DEV-81] Use constant instead of hard-coded value
* [DEV-81] Fixed some typo's + better formatting
* [DEV-242] Modified some help functionality to convert to lowercase camel case instead of just lowercase.
* [DEV-242] Corrected help functionality for struct field names.
* [DEV-242] Corrected help functionality for struct names.
* [DEV-242] Cleaned up toLowercaseCamelCase.
* [DEV-242] Renamed toLowercaseCamelCase to toCamelCase.
* [DEV-242] Converted the rest of the stuff in rpcserverhelp.go to camelCase. Fixed a bug in the camelCase converter.
* [DEV-242] camelCase-ified the last few RPC parameter names.
* [DEV-242] Fixed an off-by-one bug in toCamelCase.
* [DEV-242] Changed back from "jsonRpc" to "jsonrpc".
* [DEV-242] Moved toCamelCase into utils, wrote unit tests for it, and fixed an off-by-one bug.
* [DEV-242] Re-exported DefaultHomeDir because it's required in windows_service.go.
* [DEV-242] Added a comment above DefaultHomeDir to satisfy golint.
* [DEV-242] Formatted config/config.go.
* [DEV-234] add TxAcceptedInBlock and TxBlocks
* [DEV-234] test TxAcceptedInBlock and TxBlocks
* [DEV-234] test TxAcceptedInBlock and TxFirstBlockRegion
* [DEV-234] rename selectedPathSet to selectedPathChain
* [DEV-234] set indexers db as part of index manager initialization
* [DEV-234] remove redudant dag instance in txindex
* [DEV-234] fix TestTxIndexConnectBlock and add DAGParams as part of config in DAGSetup
* [DEV-234] TestTxIndexConnectBlock make K=1 to make calculations easier
* [DEV-234] rename TxAcceptingBlock to BlockThatAcceptedTx
* [DEV-234] update block fields names in txindex_test.go
* [DEV-234] rename selectedPathChain -> selectedPathChainSet
* [DEV-236] add counter to blockdag that will increment for each valid block that is connected to the dag
* [DEV-236] increment dag.blockCount while building block index in initDagState
* [DEV-236] changed dag.BlockCount to return an unsigned value
* [DEV-255] create checkConnectToPastUTXO and move the required functionalities to it from checkConnectBlock
* [DEV-255] get rid of checkConnectBlock
* [DEV-255] rename pNode -> node
* [DEV-255] add comment to describe ErrWithDiff
* [DEV-134] Implement Continuous Integration
Squashed commit:
[5e41d830] Dev 223 fix txindex (#100)
* [DEV-201] In handleGetBlockDAGInfo calculate difficulty by the tip with the lowest bits
* [DEV-202] Move VirtualBlock.GetUTXOEntry to BlockDAG
* [DEV-203] Move VirtualBlock.SelectedTip() to BlockDAG
* [DEV-203] Move VirtualBlock.SelectedTip() to BlockDAG
* [DEV-204] Unexport VirtualBlock() and add CalcMedianTime method for DAG
* [DEV-204] add explanation about difficulty in CurrentBits() comment
* [DEV-204] unexport VirtualBlock type
* [DEV-223] make applyUTXOChanges return pastUTXOResults
* [DEV-223] add bluestxdata for current block as well
* [DEV-223] re-design tx index
* [DEV-223] edit txindex comments
* [DEV-223] rename BluesTxData -> AcceptedTxData, and return from applyUTXOChanges only transactions that got accepted
* [DEV-223] add unit test for txindex
* [DEV-223] fix comments and unite blueTransaction and AcceptedTxData to one type
* [DEV-223] use bucket cursor for dbFetchFirstTxRegion
* [DEV-223] use the same cursor instance for dbFetchFirstTxRegion
* [DEV-223] write in dbFetchFirstTxRegion's comment that it returns the first block region
* [DEV-223] rename type BlueBlockTransaction to TxWithBlockHash
* [DEV-223] add named returned value for applyUTXOChanges
[4c95e293] [DEV-134] Made golint ignore the vendor directory.
[21736dbc] [DEV-134] Renamed ExampleBlockChain_ProcessBlock to ExampleBlockDAG_ProcessBlock to satisfy go vet.
[beea6486] [DEV-134] Removed pushing the built docker to a remove repository. That's unnecessary at this stage.
[bee911ed] [DEV-134] Made all precompilation checks run on everything instead of only the root dir.
[585f92ae] [DEV-134] Added "github.com/pkg/errors" to dep.
[5f02f570] [DEV-134] -vendor-only is written with only one hyphen.
[3eee7f95] [DEV-134] go vet instead of go tool vet.
[0c2d4343] [DEV-134] Split all the pre-compile checks to separate lines to be able to tell which of them is failing.
[780519c8] [DEV-134] Ran gofmt on everything.
[8247146b] Dev 223 fix txindex (#100)
* [DEV-201] In handleGetBlockDAGInfo calculate difficulty by the tip with the lowest bits
* [DEV-202] Move VirtualBlock.GetUTXOEntry to BlockDAG
* [DEV-203] Move VirtualBlock.SelectedTip() to BlockDAG
* [DEV-203] Move VirtualBlock.SelectedTip() to BlockDAG
* [DEV-204] Unexport VirtualBlock() and add CalcMedianTime method for DAG
* [DEV-204] add explanation about difficulty in CurrentBits() comment
* [DEV-204] unexport VirtualBlock type
* [DEV-223] make applyUTXOChanges return pastUTXOResults
* [DEV-223] add bluestxdata for current block as well
* [DEV-223] re-design tx index
* [DEV-223] edit txindex comments
* [DEV-223] rename BluesTxData -> AcceptedTxData, and return from applyUTXOChanges only transactions that got accepted
* [DEV-223] add unit test for txindex
* [DEV-223] fix comments and unite blueTransaction and AcceptedTxData to one type
* [DEV-223] use bucket cursor for dbFetchFirstTxRegion
* [DEV-223] use the same cursor instance for dbFetchFirstTxRegion
* [DEV-223] write in dbFetchFirstTxRegion's comment that it returns the first block region
* [DEV-223] rename type BlueBlockTransaction to TxWithBlockHash
* [DEV-223] add named returned value for applyUTXOChanges
[bff68aa3] [DEV-134] Gave executable permission to deploy.sh
[638a99d9] [DEV-134] Added jenkinsfile and deploy script.
* [DEV-134] Added a robust testing script.
* [DEV-134] Fixed a bash-ism.
* [DEV-134] Disabled testing with coverage for now.
* [DEV-134] Disabled golint and removed removing debug symbols.
* [DEV-134] Disabled aligncheck.
* [DEV-134] Disabled structcheck and varcheck.
* [DEV-134] Added "don't inline functions" to compiler flags for testing.
* [DEV-134] Made build fail if gofmt prints out anything.
* [DEV-134] Fixed misleading comment.
* [DEV-134] Added comments to test.sh.
* [DEV-134] Renamed tm to measure_runtime and removed do_ prefixes from functions.
* [DEV-134] Fixed gofmt line in build script.
* [DEV-134] Fixed gofmt some more.
* [DEV-134] Fixed gofmt not actually failing due to logical or.
* [DEV-91] add TestAddrIndex
* [DEV-91] add TestFeeEstimator
* [DEV-91] rename TestFeeEstimator -> TestFeeEstimatorCfg
* [DEV-91] added TestCount
* [DEV-91] add TestExtractRejectCode in mempool_test.go
* [DEV-91] get rid of fakeErr type and defer unpatching monkey patch
* [DEV-91] use the result of monkey.patch to unpatch
* [DEV-222] Changed type of TxOut.Value and util.Amount to uin64
* [DEV-222] Replaced math.MaxUint64 with 0
* [DEV-222] Fixed comment to reflect uint64 instead of int64
* [DEV-222] Fixed overflow comment
* [DEV-201] In handleGetBlockDAGInfo calculate difficulty by the tip with the lowest bits
* [DEV-202] Move VirtualBlock.GetUTXOEntry to BlockDAG
* [DEV-203] Move VirtualBlock.SelectedTip() to BlockDAG
* [DEV-203] Move VirtualBlock.SelectedTip() to BlockDAG
* [DEV-204] Unexport VirtualBlock() and add CalcMedianTime method for DAG
* [DEV-204] add explanation about difficulty in CurrentBits() comment
* [DEV-204] unexport VirtualBlock type
* [DEV-223] make applyUTXOChanges return pastUTXOResults
* [DEV-223] add bluestxdata for current block as well
* [DEV-223] re-design tx index
* [DEV-223] edit txindex comments
* [DEV-223] rename BluesTxData -> AcceptedTxData, and return from applyUTXOChanges only transactions that got accepted
* [DEV-223] add unit test for txindex
* [DEV-223] fix comments and unite blueTransaction and AcceptedTxData to one type
* [DEV-223] use bucket cursor for dbFetchFirstTxRegion
* [DEV-223] use the same cursor instance for dbFetchFirstTxRegion
* [DEV-223] write in dbFetchFirstTxRegion's comment that it returns the first block region
* [DEV-223] rename type BlueBlockTransaction to TxWithBlockHash
* [DEV-223] add named returned value for applyUTXOChanges
* [DEV-167] Created Dockerfile, dockerignore, and docker-compose.
* [DEV-167] Updated docker-compose to use remote image.
* [DEV-167] Added --addrindex to docker-compose.
* [DEV-167] Switched to testnet and plugged in the correct address.
* [DEV-167] Removed the third and fourth nodes from docker-compose.
* [DEV-167] Exposed RPC port, added rpcuser and rpcpass.
* [DEV-167] Wrangled RPC stuff into shape. Moved docker stuff into btcd/docker.
* [DEV-167] Moved dockerignore back to root. Corrected path for rpc.cert and rpc.key.
* [DEV-167] Added Jenkins job stuff.
* [DEV-167] Added deploy.sh.
* [DEV-167] Removed .travis.yaml and a couple of files that shouldn't be in this branch.
* [DEV-224] Fix getblockcount RPC command to show real data
* [DEV-227] provisionalNode.commit() should not update the original parents if the provisionalNode was created with withRelatives=false
* [DEV-224] remove debug info from TestBlockCount and BlockDAG.BlockCount()
* [DEV-201] In handleGetBlockDAGInfo calculate difficulty by the tip with the lowest bits
* [DEV-202] Move VirtualBlock.GetUTXOEntry to BlockDAG
* [DEV-203] Move VirtualBlock.SelectedTip() to BlockDAG
* [DEV-203] Move VirtualBlock.SelectedTip() to BlockDAG
* [DEV-204] Unexport VirtualBlock() and add CalcMedianTime method for DAG
* [DEV-204] add explanation about difficulty in CurrentBits() comment
* [DEV-204] unexport VirtualBlock type
* [DEV-206] allow block timestamp to be equal to past median time
* [DEV-206] make more specific error messages
* [DEV-206] make independent nodes in TestPastMedianTime
* [DEV-149] Add test case for CVE-2018-17144 (Bitcoin DoS/Double Spend bug)
* [DEV-149] change t.Errorf + return to t.Fatalf
* [DEV-149] fix malformed blocks
* [DEV-149] change test blocks to use simnet genesis
* [DEV-92] Covered lookupPreviousNodes in tests.
* [DEV-92] Covered accept.go in tests.
* [DEV-92] Fixed a typo.
* [DEV-92] Covered blockindex.go in tests.
* [DEV-92] Replaced Errorf + return with Errorf. Added the test case to the error messages.
* [DEV-92] Fixed grammar in a comment.
* [DEV-92] Split casting error checking and ErrorCode checking into separate tests.
* [DEV-92] Improved errors.
* [DEV-92] Made errors a tiny bit more descriptive.
* [DEV-189] invert arguments order or daghash.Less
* [DEV-189] invert arguments order of daghash.Less in blockset.highest
* [DEV-189] change to equivalent condition in Hash.Less to make it prettier
* [DEV-194] change prevblocks order to be from low hash to high hash
* [DEV-189] invert arguments order or daghash.Less
* [DEV-189] invert arguments order of daghash.Less in blockset.highest
* [DEV-189] change to equivalent condition in Hash.Less to make it prettier
* [DEV-169] add to BlockDAG TipHashes() and HighestTipHash() and remove Tiphashes() and SelectedTipHash from VirtualBlock
* [DEV-169] move highest node logic to separate method of blockset
* [DEV-66] fix endless loop in validateParents
* [DEV-166] add TestValidateParents
* [DEV-166] use generateNode in TestValidateParents to avoid repetition
* [DEV-166] use blockDAG.genesis instead of blockDAG.virtual.SelectedTip()
* [DEV-132] change TargetTimePerBlock to 10 seconds
* [DEV-132] change pow limits to regtest limits in all networks, enable cpu mining in testnet, and get rid of dns seeds in testnet
* [DEV-91] add tests for RemoveOrphansByTag
* [DEV-91] Add tests for orphan tx expiration
* [DEV-91] add TestDoubleSpends
* [DEV-91] add TestFetchTransaction
* [DEV-91] added a test for MinHighPriority
* [DEV-91] test unparsable scripts
* [DEV-91] test MaxOrphanTxs=0
* [DEV-91] add TestRemoveTransaction
* [DEV-105] use utxodiff in mempool instead of utxoview
* [DEV-105] get rid of utxoview
* [DEV-105] fix tests to use utxoset
* [DEV-105] remove utxoview type
* [DEV-105] move DagSetup to test_utils.go
* [DEV-105] add comments and add blockdag/test_utils_test.go
* [DEV-105] change checkPoolDoubleSpend to check utxodiff
* [DEV-105] add restoreInputs arg to removeTransaction
* [DEV-91] add restoreInputs arg to removeTransaction in unit tests
* [DEV-105] add restoreInputs arg to removeTransaction
* [DEV-105] give more descriptive names to vars
* [DEV-115] close txChan outside of HandleNewBlock
* [DEV-105] rename DagSetup -> DAGSetup
* [DEV-91] remove IsSpentInDiff
* [DEV-91] fix comment
* [DEV-91] Make IsPushOnlyScript return an error if the script cannot be parsed
* [DEV-91] add more tests for IsPushOnlyScript
* [DEV-91] fix comments
* [DEV-91] fix NewAddressPubKeyHash
* [DEV-91] add tests for RemoveOrphansByTag
* [DEV-91] Add tests for orphan tx expiration
* [DEV-91] add TestDoubleSpends
* [DEV-91] add TestFetchTransaction
* [DEV-91] added a test for MinHighPriority
* [DEV-91] test unparsable scripts
* [DEV-91] test MaxOrphanTxs=0
* [DEV-91] add TestRemoveTransaction
* [DEV-105] use utxodiff in mempool instead of utxoview
* [DEV-105] get rid of utxoview
* [DEV-105] fix tests to use utxoset
* [DEV-105] remove utxoview type
* [DEV-105] move DagSetup to test_utils.go
* [DEV-105] add comments and add blockdag/test_utils_test.go
* [DEV-105] change checkPoolDoubleSpend to check utxodiff
* [DEV-105] add restoreInputs arg to removeTransaction
* [DEV-91] add restoreInputs arg to removeTransaction in unit tests
* [DEV-105] add restoreInputs arg to removeTransaction
* [DEV-105] give more descriptive names to vars
* [DEV-115] close txChan outside of HandleNewBlock
* [DEV-105] rename DagSetup -> DAGSetup
* [DEV-91] remove IsSpentInDiff
* [DEV-91] fix comment
* [DEV-91] Make IsPushOnlyScript return an error if the script cannot be parsed
* [DEV-91] add more tests for IsPushOnlyScript
* [DEV-91] fix comments
* [DEV-105] use utxodiff in mempool instead of utxoview
* [DEV-105] get rid of utxoview
* [DEV-105] fix tests to use utxoset
* [DEV-105] remove utxoview type
* [DEV-105] move DagSetup to test_utils.go
* [DEV-105] add comments and add blockdag/test_utils_test.go
* [DEV-105] add restoreInputs arg to removeTransaction
* [DEV-105] give more descriptive names to vars
* [DEV-115] close txChan outside of HandleNewBlock
* [DEV-105] rename DagSetup -> DAGSetup
* [DEV-75] Removed fetchEntryByHash, which was no longer used.
* [DEV-75] Removed a few functions in manager.go that weren't used by anything.
* [DEV-75] checkConnectBlock will soon not accept a utxoViewpoint, so removed the call to view.Tip() so that it could be deleted.
* [DEV-75] Got rid of tips in UtxoViewpoint.
* [DEV-75] Added the full UTXO set to the virtual block.
* [DEV-75] Implemented UTXO-wrangling when adding a new block.
* [DEV-75] Added isCoinbase to utxoEntry creation.
* [DEV-75] Added blockHeight to utxoEntry creation.
* Implemented fetching the fullUTXOSet from the database.
* [DEV-75] Simplified DAGState because almost all of the state in it was unnecessary. Also got rid of dbDAGState.
* [DEV-75] Made the process around adding a new block, UTXO-wise, much safer.
* [DEV-75] Implemented melding the virtual UTXO diff to the database.
* [DEV-75] Fixed utxoSet loading from the wrong bucket.
* [DEV-75] Began pruning utxoviewpoint.go. Replaced FetchUtxoEntry with a fullUTXOSet-based GetUTXOEntry.
* [DEV-75] Removed fetchUtxos.
* [DEV-75] Moved GetUTXOEntry into the virtual block.
* [DEV-75] Updated IndexManager to not use utxoViewpoints.
* [DEV-75] Fixed some bad login in restoreUTXO involving nodeDiffs.
* [DEV-75] Got rid of the UTXO spend journal, which wasn't used anywhere.
* [DEV-75] Moved some STXO-related validation logic out of connectToDAG and into checkConnectBlock.
* [DEV-75] Renamed UtxoXxx to UTXOXxx. Removed a bunch of methods that were no longer used by anything.
* [DEV-75] Another Utxo -> UTXO rename.
* [DEV-75] Removed IsModified from UTXOView, which was not used anywhere.
* [DEV-75] Renamed nodeDiff to provisionalNode. Added a bunch of comments.
* [DEV-75] Removed the test for genesis in pastUTXO, since it can never happen.
* [DEV-75] Wrote tests for errors in pastUTXO.
* [DEV-75] Wrote tests for errors in restoreUTXO.
* [DEV-75] Improved testErrorThroughPatching.
* [DEV-75] Wrote tests for errors in verifyAndBuildUTXO.
* [DEV-75] Used TipHashes instead of tips().hashes(), fixed comments in a few places, and brought back an error return that I erroneously removed.
* [DEV-75] Removed UTXO set logs.
* [DEV-75] Recreated add/remove/contains functions for utxoCollection.
* [DEV-75] Changed newUTXOEntry to use an object initializer.
* [DEV-75] Changed the UTXO bucket version to 1.
* Added a comment that clarifies that the index is not initialized before initDAGState is called.
* [DEV-75] Renamed GetVirtualBlock to VirtualBlock.
* [DEV-75] Removed superfluous variables.
* [DEV-75] Combined connectBlockToParents with updateParentsDiffs.
* [DEV-75] Removed another superfluous variable.
* [DEV-75] In pastUTXO, first fetch transactions from the database and only then add them.
* [DEV-75] Reworded the comment for commit().
* [DEV-75] Made all the connectUTXO subfunctions not BlockDAG methods.
* [DEV-75] Updated the comment for connectUTXO.
* [DEV-75] Updated the comment explaining why we're creating provisionalNodes.
* [DEV-75] Removed a couple of unnecessary calls to toProvisionalNode.
* [DEV-75] Renamed connectUTXO to applyUTXOChanges.
* [DEV-75] Replaced allProvisionalNodes with provisionalNodeSet, an object that holds provisionalNodes for this particular call to applyUTXOChanges.
* [DEV-75] Changed createProvisionalNode to accept a boolean "withParents" instead of relying on the caller to supply the node's parents or an empty set.
* [DEV-75] Made most applyUTXOChanges subfunctions be methods on provisionalNode.
* [DEV-75] Fixed a couple of bad comments.
* [DEV-75] Added descriptive error messages to callers of applyUTXOChanges.
* [DEV-75] Fixed weird English.
* [DEV-75] Replaced DAGState with a slice of DAG tip hashes.
* [DEV-75] In createProvisionalNode, changed withParents to withRelatives to avoid certain kinds of attacks.
* [DEV-75] Renamed createdProvisionalNode to newProvisionalNode.
* [DEV-75] Added precalculating the amount of transactions inside a new block's blue set.
* [DEV-75] Pruned unnecessary variable.
* [DEV-107] Disallow situation where a parent is also an ancestor of another parent
* [DEV-107] Add comment for validateParents
* [DEV-107] move validateParents to checkBlockContext
* [DEV-107] break validateParents error to 2 lines
* [DEV-107] remove TestProcessBlock
* [DEV-107] fix comment that explains block 3c test
* [DEV-107] remove blk_3C from TestCheckConnectBlockTemplate
* [DEV-76] handleGetBlockTemplateLongPoll to use tips instead of tip
* [DEV-76] move concatHashesToString to daghash package as JoinHashesStrings
* [DEV-76] use strings.Join for JoinHashesStrings
* [DEV-76] rename templateID -> longPollID
* [DEV-76] add function prefix to errors
* [DEV-98] Move script flags from relay rules to consensus
* [DEV-98] remove flags from script_tests.json
* [DEV-98] fix multisig and remove test that assume no minimal data rule
* [DEV-98] rename bip16 bool to isP2sh
* [DEV-98] add sighash type to overly long signature in script_tests.json
* [DEV-98] add test for NUMEQUAL for non equal numbers script_tests.json
* [DEV-98] remove debugging if
* [DEV-98] remove ErrCleanStack from EVAL_FALSE
* [DEV-98] change isP2sh to isP2SH to comply with Go style
* [DEV-98] add ScriptNoFlags to explictly indicate for empty ScriptFlags
* [DEV-98] rename ErrPubKeyType -> ErrPubKeyFormat
* [DEV-98] rename PUBKEYTYPE -> PUBKEYFORMAT
* [DEV-97] Moved github.com/daglabs/btcutil into github.com/daglabs/btcd/btcutil.
* [DEV-97] Updated Gopkg.toml to no longer refer to btcutil.
* [DEV-97] Renamed btcutil to util.
* [DEV-82] break down main to packages
* [DEV-82] separate rpcserver and server
* [DEV-82] Fixed Windows-related code that failed to compile.
* [DEV-82] remove params.go and use only dagconfig.Params
* [DEV-82] fix log.go license
* [DEV-74] Implemented and written tests for utxoIterator.
* [DEV-74] Improved utxoIterator tests.
* [DEV-74] Implemented utxoCollection tests.
* [DEV-74] Implemented utxoDiff and its tests.
* [DEV-74] Implemented utxoSet.
* [DEV -74] Added tests for fullUTXOSet.
* [DEV-74] Added some tests for diffUTXOSet.
* [DEV-74] Wrote tests for diffUTXOSet iterator.
* [DEV-74] Added a negative test for diffUTXOSet.withDiff.
* [DEV-74] Wrote tests for addTx.
* [DEV-74] Wrote a toRemove test for addTx.
* [DEV-74] Changed blockNode.utxoDiff to be of type utxoDiff.
* [DEV-74] Removed superfluous whitespace.
* [DEV-74] Renamed confusing "previousHash" to "hash".
* [DEV-74] Fixed bad test and corrected failing test.
* [DEV-74] Moved confusing "negatives" test to be part of the general utxoCollection test.
* [DEV-74] Removed utxoDiff.inverted.
* [DEV-74] Renamed blockNode.utxoDiff to blockNode.diff.
* [DEV-74] Renamed diff to diffFrom for clarity's sake.
* [DEV-74] Converted the type of utxoCollection from map[daghash.Hash]map[uint32]*wire.TxOut to map[wire.OutPoint]*UtxoEntry.
* [DEV-74] Corrected test names in utxoCollection_test.
* [DEV-74] Removed superfluous utxoCollection iterator and moved utxoIterator into utxoset.go.
* [DEV-74] Renamed variables in utxoset.go.
* [DEV-74] Renamed verifyTx to areInputsInUTXO and removed a superfulous test.
* [DEV-74] Fixed bad test logic in TestDiffUTXOSet_addTx.
* [DEV-74] Added a few comments. Added reference-equals checks to clone functions.
* [DEV-74] Moved utxoCollection and utxoDiff into utxoset.go.
* [DEV-74] Wrote explanations for utxoCollection and utxoDiff tests.
* [DEV-74] Wrote explanations for all utxoSet tests besides addTx.
* [DEV-74] Wrote explanations for TestDiffUTXOSet_addTx.
* [DEV-74] Moved the documentation for utxoDiff into utxoset.go.
* [DEV-74] Wrote an explanation on utxoSet.
* [DEV-75] Found a typo.
* [DEV-75] Renamed dag -> virtual, dagView -> virtualBlock.
* [DEV-75] Renamed newDAGView to newVirtualBlock.
* [DEV-75] Moved queries for the genesis block from virtualBlock to BlockDAG.
* [DEV-75] Got rid of chainView height and findFork.
* [DEV-75] Renamed receivers from c to v.
* [DEV-75] Updated initBlockNode to allow for virtual (headerless) nodes, updated dbDAGState to contain multiple tip hashes, implemented virtualBlock.setTips.
* [DEV-75] Got rid of virtualBlock.equals, which was not used anywhere.
* [DEV-75] Got rid of virtualBlock.tip().
* [DEV-75] Got rid of SetTip everywhere except for tests.
* [DEV-75] Got rid of Next().
* [DEV-75] Got rid of Contains().
* [DEV-75] Got rid of HeightRange(), as no one was using it.
* [DEV-75] Made verifyDAG in rpcserver.go not use block height for iteration.
* [DEV-75] Got rid of the part of Manager.Init() that handled "catching up" for side chains, which allowed me to get rid of BlockDAG.BlockByHeight().
* [DEV-75] Dropped support for the RPC command getblockhash since it was getting blocks by their height.
* [DEV-75] Dropped getnetworkhashps since it was reliant on height, fixed another couple of RPC commands to return nextHashes instead of a nextHash, and got rid of nodeByHeight in virtualBlock.
* [DEV-75] Got rid of setTip().
* [DEV-75] Moved blockLocator() out of virtualBlock and into BlockDAG. Also removed TestLocateInventory().
* [DEV-75] Implemented addTip().
* [DEV-75] Cleaned up virtualblock.go a bit.
* [DEV-75] Erased irrelevant tests in virtualblock_test.go. Moved dag-related tests into dag_test.go.
* [DEV-75] Removed unnecessary nil check.
* [DEV-75] Wrote tests for virtualBlock.
* [DEV-75] Fixed bad test, added explanations to tests.
* [DEV-89] Fixed a comment.
* [DEV-89] Fixed another comment.
* [DEV-89] Removed the section in Manager::Init that handled rolling back indexes to the main chain if their tip is an orphaned fork. This could only happen during reorg, which no longer exists. Also removed BlockDAG::MainChainHasBlock, which was no longer used by anyone.
* [DEV-89] Removed the nil check inside initBlockNode() and amended the one place that called it with nil.
* [DEV-89] Renamed the receiver param for BlockDAG from b to dag.
* [DEV-89] Moved fastLog2Floor from dag.go to btcutil/btcmath.go.
* [DEV-89] Renamed tstTip to testTip.
* [DEV-89] Renamed phanom_test.go to phantom_test.go.
* [DEV-89] Fixed comments, renamed mainChainHeight to dagHeight.
* [DEV-89] Rewrote virtualBlock.addTip().
* [DEV-89] Fixed a comment. (chain -> DAG)
* [DEV-89] Fixed another chain -> DAG comment.
* [DEV-74] Implemented and written tests for utxoIterator.
* [DEV-74] Improved utxoIterator tests.
* [DEV-74] Implemented utxoCollection tests.
* [DEV-74] Implemented utxoDiff and its tests.
* [DEV-74] Implemented utxoSet.
* [DEV -74] Added tests for fullUTXOSet.
* [DEV-74] Added some tests for diffUTXOSet.
* [DEV-74] Wrote tests for diffUTXOSet iterator.
* [DEV-74] Added a negative test for diffUTXOSet.withDiff.
* [DEV-74] Wrote tests for addTx.
* [DEV-74] Wrote a toRemove test for addTx.
* [DEV-74] Changed blockNode.utxoDiff to be of type utxoDiff.
* [DEV-74] Removed superfluous whitespace.
* [DEV-74] Renamed confusing "previousHash" to "hash".
* [DEV-74] Fixed bad test and corrected failing test.
* [DEV-74] Moved confusing "negatives" test to be part of the general utxoCollection test.
* [DEV-74] Removed utxoDiff.inverted.
* [DEV-74] Renamed blockNode.utxoDiff to blockNode.diff.
* [DEV-74] Renamed diff to diffFrom for clarity's sake.
* [DEV-74] Converted the type of utxoCollection from map[daghash.Hash]map[uint32]*wire.TxOut to map[wire.OutPoint]*UtxoEntry.
* [DEV-74] Corrected test names in utxoCollection_test.
* [DEV-74] Removed superfluous utxoCollection iterator and moved utxoIterator into utxoset.go.
* [DEV-74] Renamed variables in utxoset.go.
* [DEV-74] Renamed verifyTx to areInputsInUTXO and removed a superfulous test.
* [DEV-74] Fixed bad test logic in TestDiffUTXOSet_addTx.
* [DEV-74] Added a few comments. Added reference-equals checks to clone functions.
* [DEV-74] Moved utxoCollection and utxoDiff into utxoset.go.
* [DEV-74] Wrote explanations for utxoCollection and utxoDiff tests.
* [DEV-74] Wrote explanations for all utxoSet tests besides addTx.
* [DEV-74] Wrote explanations for TestDiffUTXOSet_addTx.
* [DEV-74] Moved the documentation for utxoDiff into utxoset.go.
* [DEV-74] Wrote an explanation on utxoSet.
* [DEV-74] Moved diffChild next to diff, improved their comments, moved the explanations for diffFrom and withDiff to the appropriate methods, got rid of utxoIterator, and renamed areInputsInUTXO to containsInputs.
* [DEV-74] Replaced boring old-fashioned reference equality with special, fancy reference equality for maps, slices, and channels.
* [DEV-74] Wrote additional explanations for test cases.
* [DEV-62] add phantom constructs to blocknode
* [DEV-62] add phantom constructs to blocknode
* [DEV-72] write blues()
* [DEV-72] write blues()
* [DEV-72] write blues()
* [DEV-62] add comments to new phantom constructs in blocknode
* Fixed dbIndexConnectBlock. (#33)
* Fixed dbIndexConnectBlock.
* Removed redundant check in storeFilter.
* Created a new method to BlockHeader: IsGenesis.
* [DEV-71] Implement BlockHeap (#35)
* [DEV-71] Implemented BlockHeap.
* [DEV-71] Removed irrelevant comment.
* [DEV-71] Renamed variables in Pop() and split Less() to multiple lines.
* [DEV-72] write blues()
* [DEV-72] write blues()
* [DEV-72] write blues()
* [DEV-72] write blues tests
* [DEV-72] write blues tests
* [DEV-72] remove relevant past
* [DEV-72] write blues tests
* [DEV-72] write blues tests
* [DEV-72] write blues tests
* [DEV-72] write functions to order blockSet by hash and write blue tests
* [DEV-72] add secret mining and censorship attack tests
* [DEV-72] remove prints
* [DEV-72] remove K from dagconfig.Params
* [DEV-72] remove K from dagconfig.Params
* [DEV-72] change blueScore to uint64
* [DEV-72] block V was missing, so renamed w -> v, x -> w etc
* [DEV-72] use node.String instead of %v
* [DEV-72] block V was missing, so renamed w -> v, x -> w etc
* [DEV-72] add K to dagconfig.Params, and add expected reds to all phantom tests
* [DEV-72] set K=10 and add comments to phantom and phantom tests
* [DEV-72] fix formatting and add comments to TestPhantom
* [DEV-72] fix grammar
* [DEV-77] Added tests for ScriptFreeList
* [DEV-77] Fixed formatting in BlockHeader .SelectedPrevBlock and IsGenesis
* [DEV-77] Restructured returnScriptBuffers to be more idiomatic and testable
* [DEV-77] Added tests for binaryFreeList
* [DEV-77] Added test-cases to TestVarIntWireErrors to check for errors when writing the varint length
* [DEV-77] Added MsgSendHeaders and MsgFeeFilter test-cases for TestMessage
* [DEV-77] Added test for BlockHeader.IsGenesis()
* [DEV-77] using binaryFreeListMaxItems instead of freeListMaxItems in TestBinaryFreeList
* [DEV-77] Fixed error message copy-paste typo
* [DEV-42] Fixed rpcserverhelp.go.
* [DEV-42] Fixed GetBlockHeaderVerboseResult, GetBlockVerboseResult, and GetBlockTemplateResult.
* [DEV-42] Fixed rpcserver.go.
* [DEV-42] Got rid of the rescan command. Apparently it was deprecated and succeeded by the rescanBlocks command.
* [DEV-42] Modified handleRescanBlocks to satisfy compilation.
* [DEV-42] Fixed failing tests in rpcserver.go and rpcserverhelp.go.
* [DEV-42] Removed tests for rescan.
* [DEV-42] Removed rescan from notify.go.
* [DEV-42] Fixed a couple of comments I've missed.
* [DEV-36] Fixed import.go.
* [DEV-65] Renamed BestState to State and GetSnapshot to GetCurrentState.
* [DEV-65] Implemented State and SelectedTip, updated dagState (the serializable version of State), and updated State creations.
* [DEV-65] Fixed references to GetCurrentState and fixed tests.
* [DEV-40] Created a new method AreEquals for comparing hash slices. Updated mining package to work with primitive DAG.
* [DEV-65] Renamed State to DAGState, corrected comments, created a new thread-safe method setDAGState, fixed broken references I missed.
* [DEV-65] Converted dagState (the object) to dbDAGState, and made it serialize/deserialize into/out of JSON.
* [DEV-65] Renamed Txns to Txs and fixed a couple of comments.
* [DEV-65] Renamed some more "Txns" and removed some dead code.
* [DEV-36] Fixed findcheckpoint.go.
* [DEV-36] Fixed indentation.
* [DEV-36] Fixed weird phrasing.
* [DEV-36] Fixed some more weird phrasing.
* [DEV-36] Removed haveBlocks variable from the function HaveBlocks.
* [DEV-65] Renamed BestState to State and GetSnapshot to GetCurrentState.
* [DEV-65] Implemented State and SelectedTip, updated dagState (the serializable version of State), and updated State creations.
* [DEV-65] Fixed references to GetCurrentState and fixed tests.
* [DEV-40] Created a new method AreEquals for comparing hash slices. Updated mining package to work with primitive DAG.
* [DEV-65] Renamed State to DAGState, corrected comments, created a new thread-safe method setDAGState, fixed broken references I missed.
* [DEV-65] Converted dagState (the object) to dbDAGState, and made it serialize/deserialize into/out of JSON.
* [DEV-65] Renamed Txns to Txs and fixed a couple of comments.
* [DEV-65] Renamed some more "Txns" and removed some dead code.
* [DEV-40] Fixed formatting errors...
* [DEV-40] Reworded confusing test failure messages.
* [DEV-40] Fixed merge error.
* [DEV-40] Fixed style for AreEqual and TestAreEqual.
* [DEV-65] Renamed BestState to State and GetSnapshot to GetCurrentState.
* [DEV-65] Implemented State and SelectedTip, updated dagState (the serializable version of State), and updated State creations.
* [DEV-65] Fixed references to GetCurrentState and fixed tests.
* [DEV-65] Renamed State to DAGState, corrected comments, created a new thread-safe method setDAGState, fixed broken references I missed.
* [DEV-65] Converted dagState (the object) to dbDAGState, and made it serialize/deserialize into/out of JSON.
* [DEV-65] Renamed Txns to Txs and fixed a couple of comments.
* [DEV-65] Renamed some more "Txns" and removed some dead code.
* [DEV-18] changed CSV/CLTV to be regular op codes, and returned nop2 and nop3 to be regular nops
* [DEV-18] remove csv/cltv flags - part 1
* [DEV-18] remove csv/cltv flags - part 2
* [DEV-18] remove csv/cltv activation rules
* [DEV-18] remove csv/cltv activation rules
* [DEV-18] csv_fork_test fixes
* [DEV-18] readd chain params
* [DEV-18] readd chain params and remove csv activation rules
* [DEV-18] returned build flags to integration test
* [DEV-18] make csv/cltv to pop the the first element of the stack instead of peeking it
* [DEV-18] fix comments related to CSV/CLTV to remove any reference to soft fork
* [DEV-18] fix comments related to CSV/CLTV to remove any reference to soft fork
* [DEV-18] rename csv_fork_test.go to csv_fork.go
* [DEV-18] change mTx location
* [DEV-18] remove BIP0065Height
* [DEV-18] add function isUpgradableNop for readability
* [DEV-45] Updated the BlockDB files to contain blocks that could be deserialized.
* [DEV-45] Fixed TestHaveBlock (finally)
* [DEV-45] Fixed TestLocateInventory and everything that's reliant on chainview_test::chainedNodes.
* [DEV-45] Fixed TestChainViewNil.
* [DEV-45] Fixed TestNotifications.
* [DEV-45] Fixed ExampleBlockChain_ProcessBlock.
* [DEV-45] Fixed TestCheckBlockScripts.
* [DEV-45] Fixed TestCheckConnectBlockTemplate.
* [DEV-45] Renamed the BlockDBs to their original names.
* [DEV-45] Skipping TestFullBlocks for until we have implemented Phantom.
* [DEV-45] Deleted a couple of methods that are no longer used. (They were previously used for reorganization)
* [DEV-56] Remove any occurance of OP_CODESEPARATOR
* [DEV-56] Modified sighash tests to exclude OP_CODESEPARATOR
* [DEV-56] Fixed sighash.json to not include any codeseparators at all
* [DEV-56] Reformatted sighash.json
* [DEV-56] Rename any instance of subScript to script
* [DEV-56] Extracted check for opUnknown to separate function
* [DEV-58] removed addition of multisig dummy in signMultiSig
* [DEV-58] Removed temporary code for test cleanup
* [DEV-56] Remove any occurance of OP_CODESEPARATOR
* [DEV-56] Modified sighash tests to exclude OP_CODESEPARATOR
* [DEV-56] Fixed sighash.json to not include any codeseparators at all
* [DEV-56] Reformatted sighash.json
* [DEV-56] Rename any instance of subScript to script
* [DEV-56] Extracted check for opUnknown to separate function
* [DEV-58] Removed temporary code for test cleanup
* [DEV-50] Remove the Multisig bug requiring a dummy push
* [DEV-53] SigHashSingle now errors when the index is wrong
* [DEV-53] Fixed tests for SIGHASH_SINGLE
* [DEV-55] Removed FindAndDelete functionality that deletes sognatures
* [DEV-55] Removed some more redundant tests
* [DEV-55] Fixed redundant comment
* [DEV-50] Remove the Multisig bug requiring a dummy push
* [DEV-53] SigHashSingle now errors when the index is wrong
* [DEV-53] Fixed tests for SIGHASH_SINGLE
* [DEV-53] Removed redundant part of comment
* [DEV-50] Remove the Multisig bug requiring a dummy push
* [DEV-50] Removed redundant comment, since dummy bug was fixed
* [DEV-50] Removed some more dummy-related comments and error message
* [DEV-27] Fixed dbtool to work with multiple previous blocks
* [DEV-27] Fix Example_blockStorageAndRetrieval expected bytes
* [DEV-27] Added generator for testdata, and generated new testdata to conform to DAG
* [DEV-27] Fixed FetchBlockHeader(s) and database interface test for DAG
This modifies the utxoset in the database and related UtxoViewpoint to
store and work with unspent transaction outputs on a per-output basis
instead of at a transaction level. This was inspired by similar recent
changes in Bitcoin Core.
The primary motivation is to simplify the code, pave the way for a
utxo cache, and generally focus on optimizing runtime performance.
The tradeoff is that this approach does somewhat increase the size of
the serialized utxoset since it means that the transaction hash is
duplicated for each output as a part of the key and some additional
details such as whether the containing transaction is a coinbase and the
block height it was a part of are duplicated in each output.
However, in practice, the size difference isn't all that large, disk
space is relatively cheap, certainly cheaper than memory, and it is much
more important to provide more efficient runtime operation since that is
the ultimate purpose of the daemon.
While performing this conversion, it also simplifies the code to remove
the transaction version information from the utxoset as well as the
spend journal. The logic for only serializing it under certain
circumstances is complicated and it isn't actually used anywhere aside
from the gettxout RPC where it also isn't used by anything important
either. Consequently, this also removes the version field of the
gettxout RPC result.
The utxos in the database are automatically migrated to the new format
with this commit and it is possible to interrupt and resume the
migration process.
Finally, it also updates the tests for the new format and adds a new
function to the tests to convert the old test data to the new format for
convenience. The data has already been converted and updated in the
commit.
An overview of the changes are as follows:
- Remove transaction version from both spent and unspent output entries
- Update utxo serialization format to exclude the version
- Modify the spend journal serialization format
- The old version field is now reserved and always stores zero and
ignores it when reading
- This allows old entries to be used by new code without having to
migrate the entire spend journal
- Remove version field from gettxout RPC result
- Convert UtxoEntry to represent a specific utxo instead of a
transaction with all remaining utxos
- Optimize for memory usage with an eye towards a utxo cache
- Combine details such as whether the txout was contained in a
coinbase, is spent, and is modified into a single packed field of
bit flags
- Align entry fields to eliminate extra padding since ultimately
there will be a lot of these in memory
- Introduce a free list for serializing an outpoint to the database
key format to significantly reduce pressure on the GC
- Update all related functions that previously dealt with transaction
hashes to accept outpoints instead
- Update all callers accordingly
- Only add individually requested outputs from the mempool when
constructing a mempool view
- Modify the spend journal to always store the block height and coinbase
information with every spent txout
- Introduce code to handle fetching the missing information from
another utxo from the same transaction in the event an old style
entry is encountered
- Make use of a database cursor with seek to do this much more
efficiently than testing every possible output
- Always decompress data loaded from the database now that a utxo entry
only consists of a specific output
- Introduce upgrade code to migrate the utxo set to the new format
- Store versions of the utxoset and spend journal buckets
- Allow migration process to be interrupted and resumed
- Update all tests to expect the correct encodings, remove tests that no
longer apply, and add new ones for the new expected behavior
- Convert old tests for the legacy utxo format deserialization code to
test the new function that is used during upgrade
- Update the utxostore test data and add function that was used to
convert it
- Introduce a few new functions on UtxoViewpoint
- AddTxOut for adding an individual txout versus all of them
- addTxOut to handle the common code between the new AddTxOut and
existing AddTxOuts
- RemoveEntry for removing an individual txout
- fetchEntryByHash for fetching any remaining utxo for a given
transaction hash
This commit adds a logic to the addSpentRequests that inspects the
mempool for any spends of the outputs. Before this commit, a spend would
only be checked when a transaction was first accepted into the mempool.
The index will hold three types of entries for each filter type, block
pair: filter, header, and hash. Since they all have similar methods
and implementations, refactor to reduce duplication.
In this commit, we modify the fee estimation to use vsize as a base
rather than size. A recent commit landed to track the fee rate using
vsize in the mempool, and also correct some incorrect unit math. This is
a follow up to that commit to ensure that fee estimation is uniform
throughout.
This commit changes the value of bytesPerKb to 1000 from 1024.
This is done to ensure consistency between the fee estimator
and the mempool, where the feeRate is set to
fee * 1000 / serializedSize
This commit extends the work started by roasbeef in the
previous commit to bring full cancellation of pending
connection requests. It also adds minor refactors to
channel send/receives to help cleanup potentially
lingering go routines.
This commit adds the ability for callers to remove pending connections
via a call to the Remove() method. With this change, upstream users of
this package can use the connmgr for more elaborate connectivity needs
as they can now cancel pending connections that are no longer needed.
new txs that it observes. The block manager alerts the fee estimator
of new and orphaned blocks.
Check for invalid state and recreate FeeEstimator if necessary.
Rollback takes a block hash rather than a BlockStamp.
Increase rounds in TestEstimateFeeRollback to test dropping txs that have been in the mempool too long.
Since the tx hash has moved to the basic filter, generating an extended filter
can result in `ErrNoData`. This is handled by writing a nil filter and giving
it a zero hash.
The cfilter BIP specifies that the filter type is a uint8. The
current code encodes it correctly on the wire, but everywhere else,
it's treated as a boolean (false for basic filter, true for
extended). This commit corrects that to account for possible
additional filter types in the future. All package changes are
done in one commit as they're all interdependent. The following
packages are updated:
* blockchain/indexers
* btcjson
* peer
* wire
* main (server.go and rpcserver.go)
The btclog package has been changed to defining its own logging
interface (rather than seelog's) and provides a default implementation
for callers to use.
There are two primary advantages to the new logger implementation.
First, all log messages are created before the call returns. Compared
to seelog, this prevents data races when mutable variables are logged.
Second, the new logger does not implement any kind of artifical rate
limiting (what seelog refers to as "adaptive logging"). Log messages
are outputted as soon as possible and the application will appear to
perform much better when watching standard output.
Because log rotation is not a feature of the btclog logging
implementation, it is handled by the main package by importing a file
rotation package that provides an io.Reader interface for creating
output to a rotating file output. The rotator has been configured
with the same defaults that btcd previously used in the seelog config
(10MB file limits with maximum of 3 rolls) but now compresses newly
created roll files. Due to the high compressibility of log text, the
compressed files typically reduce to around 15-30% of the original
10MB file.
In this commit, we update the glide.lock file to be pinned against the
latest btcutil commit hash. btcutil has recently been updated to pull in
all changes from roasbeef's fork. Notably, it now includes the code
necessary for creating GCS filters (BIP 158).
Go 1.10 made some changes such that json.Unmarshal can no longer
unmarshal into exported fields that are themselves embedded via an
uninitialized unexported pointer.
Since rpcclient previously relied on this behavior, this updates the
client to create the pointers before unmarshalling into the struct.
The block index now tracks the set of dirty block nodes with status
changes that haven't been persisted and flushes the changes to the DB
at the appropriate times.
Now that all headers are stored in the in-memory index, the database
bucket managed by blockchain, and in the flat files, it makes sense to
drop the redundant data from the block index bucket in ffldb.
To avoid modifying the database interface, this reimplements
FetchBlockHeader(s) to use header data stored in flat files. This can
be trivially implemented by delegating to FetchBlockRegion.
Currently only the blocks in the active chain are loaded into the
block index on initialization. This instead iterates over the entire
block index bucket in LevelDB and loads all nodes.
The bucket contains block headers keyed by the block height encoded as
big-endian concatenated with the block hash. This allows block headers
to be fetched from the DB in height order with a cursor.
These method allows safe concurrent access to reading and modifying
block node statuses. When block statuses get persisted in a later
change, the setter methods can be used to mark block nodes as dirty.
Each node in the block index records some flags about its validation
state. This is just stored in memory for now, but can save effort if
attempting to reconnect a block that failed validation or was
disconnected.
Previously, rpctest would start a btcd node using the btcd executable
in the environment PATH. This caused difficult-to-find issues where
the code would be tested against an older version of btcd, or another
fork entirely. Now it compiles btcd the first time it is needed and
uses that fresh version when launching nodes.
This was only used to test block proposals, which has been changed to
instead use CheckConnectBlockTemplate. The flag complicated the
implementation of some chain processing routines and would be
difficult to implement with headers-first syncing.
This renames CheckConnectBlock to CheckConnectBlockTemplate and
modifies it to be easily consumable by the getblocktemplate RPC
handler. Performs full block validation now instead of partial
validation.
This propagates the interrupt channel through to blockchain and the
indexers so that it is possible to interrupt long-running operations
such as catching up indexes.
The helper function parseListeners has been changed to return a
slice of net.Addrs with Network() returning tcp4 or tcp6 instead of
returning two slices of IPv4 and IPv6 addresses to simplify calling
code. Also improves how local addresses are added to the address
manager when listening on wildcard addresses.
Also splits some newServer logic into new method initListeners.
The modifies the encoding of witness stacks in JSON responses to use a
slice of strings instead of a single space-separated string for
compatibility with Core.
This updates the README to remove the mailing lists which no longer
exist and to link the GPG public key used to sign the release tags
directly from the repository instead of the no longer active website.
This update adds additional callsite logging options via btclog and
fixes an error with the rotator package that caused it to stop running
when creating any log messages larger than 4096 bytes.
While here, switch to the new Write method of the Rotator object as
this is more efficient than using the Reader interface with a pipe.
Changes from @jrick.
This refactors the code that locates blocks (inventory discovery) out of
server and into blockchain where it can make use of the new much more
efficient chain view and more easily be tested. As an aside, it really
belongs in blockchain anyways since it's purely dealing with the block
index and best chain.
Since the majority of the network has moved to header-based semantics,
this also provides an additional optimization to allow headers to be
located directly versus needing to first discover the hashes and then
fetch the headers.
The new functions are named LocateBlocks and LocateHeaders. The former
returns a slice of located hashes and the latter returns a slice of
located headers.
Finally, it also updates the RPC server getheaders call and related
plumbing to use the new LocateHeaders function.
A comprehensive suite of tests is provided to ensure both functions
behave correctly for both correct and incorrect block locators.
This modifies calcSignatureHash to use a shallow copy of the transaction
versus a deep copy since the actual scripts themselves are not modified
and therefore don't need to be copied.
This is being done because profiling the most overall allocated space
shows that the deep copy performed in calcSignatureHash accounts for
nearly 20% of all allocations on a synced running instance. Also,
copying all of the additional data makes it more time consuming as well.
With this change, that figure drops from ~20% to ~5% of all allocations.
The following benchmark shows the relative speedups and allocation
reduction as a result of the optimization on my system. In particular,
the changes result in approximately a 15% speedup and a whopping 99.89%
reduction in allocations when using a large transaction with thousands
of inputs which was the worst case scenario.
benchmark old allocs new allocs delta
--------------------------------------------------------------------
BenchmarkCalcSignatureHash 11151 12 -99.89%
benchmark old ns/op new ns/op delta
--------------------------------------------------------------------
BenchmarkCalcSignatureHash 3599845 3056359 -15.10%
- Remove inMainChain from block nodes since that can now be efficiently
determined by using the chain view
- Track the best chain via a chain view instead of a single block node
- Use the tip of the best chain view everywhere bestNode was used
- Update chain view tip instead of updating best node
- Change reorg logic to use more efficient chain view fork finding logic
- Change block locator code over to use more efficient chain view logic
- Remove now unused block-index-based block locator code
- Move BlockLocator definition to chain.go
- Move BlockLocatorFromHash and LatestBlockLocator to chain.go
- Update both to use more efficient chain view logic
- Rework IsCheckpointCandidate to use block index and chain view
- Optimize MainChainHasBlock to use chain view instead of hitting db
- Move to chain.go since it no longer involves database I/O
- Removed error return since it can no longer fail
- Optimize BlockHeightByHash to use chain view instead of hitting db
- Move to chain.go since it no longer involves database I/O
- Removed error return since it can no longer fail
- Optimize BlockHashByHeight to use chain view instead of hitting db
- Move to chain.go since it no longer involves database I/O
- Removed error return since it can no longer fail
- Optimize HeightRange to use chain view instead of hitting db
- Move to chain.go since it no longer involves database I/O
- Optimize BlockByHeight to use chain view for main chain check
- Optimize BlockByHash to use chain view for main chain check
The purpose is to remove the dependency of blockmanager on serverPeer,
which is defined in the main package. Instead, we split out some of
the fields from serverPeer into a separate struct called peerSyncState
in blockmanager.go. While they are in the same package now, this
change makes it easier to move blockManager into its own package along
with peerSyncState. The blockManager tracks a map of Peer pointers to
the peer state and keeps it updated as peers connect and disconnect.
This removes the DisableVerify function and related state since nothing
uses it anymore since the command line option was removed. It is a
remnant of initial development.
This exposes the ability to more efficiently create a block locator from
a chain view for a given block node by using their ability to do O(1)
lookups.
It also adds tests to ensure the behavior is correct.
This significantly optimizes and simplifies the generation of block
locators by making use of the fact that all block nodes are now in
memory and therefore it is no longer necessary to consult the database
for the hashes or worry about issues related to dynamic loading of nodes.
Also, it slightly modifies the algorithm so that the doubling doesn't
start for one additional iteration in order to mirror other prominent
clients on the network. Due to the way block locators are used, this
does not change any semantics in terms of requesting and locating
blocks.
Finally, the semantics of BlockLocatorFromHash have been changed to
return a locator for the current tip in the case the hash is unknown.
This is far preferable since only including the passed block hash, when
it isn't known, could end up leading to causing a redownload of the
entire chain under certain circumstances.
Putting the test code in the same package makes it easier for forks
since they don't have to change the import paths as much and it also
gets rid of the need for internal_test.go to bridge.
While here, remove the reorganization test since it is much better
handled by the full block tests and is no longer needed and do some
light cleanup on a few other tests.
The full block tests had to remain in the separate test package since it
is a circular dependency otherwise. This did require duplicating some
of the chain setup code, but given the other benefits this is
acceptable.
This introduces the concept of a synthetic block chain that can be used
in the tests to avoid needing setup a full blown chain instance with a
database and generate valid blocks and converts the sequence lock tests
in TestCalcSequenceLock to use it.
Not only does this speed up the test execution time, but it allows the
dependency on rpctest to be removed which will allow the sequence locks
tests to be consolidated into the main package without creating a
circular dependency.
This modifies the function to set the tip in the new chainview code to
bulk copy existing nodes when it needs to expand the cap rather than
simply creating a new empty slice and allowing the walk code below it to
repopulate it. This is a nice optimization since, in practice, most of
the time expanding the cap is only required when the active chain is
being extended after having run for a while which means the end result
is that it will be able to bulk copy all the nodes and just add the most
recent one versus having to walk them all and add them back.
Also, while here expand the tests for setting the tip to ensure the
nodes contained in the resulting view are correct after forcing the
resizes and correct a bug they exposed where changing between a
longer-shorter-longer chain where the longer chain is the same chain
could result in not populating the view correctly.
Finally, update the fake nodes generated by the tests to use a
nonce generated by a deterministic prng in order to ensure the hashes of
all fake nodes are unique, but reproducible.
This regenerates the precomputed secp256k1 byte points used to optimize
scalar multiplication. This should have been done as part of the
normalization correction.
This implements a new type in the blockchain package that takes
advantage of the fact that all block nodes are now in memory to provide
a flat view of a specific chain of blocks (a specific branch of the
overall block tree) from a given tip all the way back to the genesis
block along with several convenience functions such as efficiently
comparing two views, quickly finding the fork point (if any) between two
views, and O(1) lookup of the node at a specific height.
The view is not currently used, but the intent is that the code will be
refactored to make use of these views to simplify and optimize several
areas such as best chain selection and reorg logic and finding successor
nodes. They will also greatly simplify the process of disconnecting the
download logic from the connection logic.
A comprehensive suite of tests is provided to ensure the chain views
behave correctly.
This cleans up the test in TestCalcSequenceLock in the following ways:
- Use calculated values instead of magic value so it is easier to update
the tests as needed
- Make tests match the comments
- Change comments to be more consistent and fix some grammar errors
- Set mempool flag for unconfirmed tx tests since they are intended to
mimic transactions in the mempool
This commit contains the entire btcrpcclient repository along with
several changes needed to move all of the files into the rpcclient
directory in order to prepare it for merging. This does NOT update btcd
or any of the other packages to use the new location as that will be
done separately.
- All import paths in the old btcrpcclient files have been changed to
the new location
- All references to btcrpcclient as the package name have been changed to
rpcclient
This modifies the code that determines the most recently known
checkpoint to take advantage of recent changes which make the entire
block index available in memory by only storing a reference to the
specific node in the index that represents the latest known checkpoint.
Previously, the entire block was stored and new checkpoints required
loading it from the database.
This completely removes the threshold state database caching code since
it can very quickly be calculated at startup now that the entire block
index is loaded first.
This reworks the block index code such that it loads all of the headers
in the main chain at startup and constructs the full block index
accordingly.
Since the full index from the current best tip all the way back to the
genesis block is now guaranteed to be in memory, this also removes all
code related to dynamically loading the nodes and updates some of the
logic to take advantage of the fact traversing the block index can
longer potentially fail. There are also more optimizations and
simplifications that can be made in the future as a result of this.
Due to removing all of the extra overhead of tracking the dynamic state,
and ensuring the block node structs are aligned to eliminate extra
padding, the end result of a fully populated block index now takes quite
a bit less memory than the previous dynamically loaded version.
The main downside is that it now takes a while to start whereas it was
nearly instant before, however, it is much better to provide more
efficient runtime operation since that is its ultimate purpose and the
benefits far outweigh this downside.
Some benefits are:
- Since every block node is in memory, the recent code which
reconstructs headers from block nodes means that all headers can
always be served from memory which is important since the majority of
the network has moved to header-based semantics
- Several of the error paths can be removed since they are no longer
necessary
- It is no longer expensive to calculate CSV sequence locks or median
times of blocks way in the past
- It will be possible to create much more efficient iteration and
simplified views of the overall index
- The entire threshold state database cache can be removed since it is
cheap to construct it from the full block index as needed
An overview of the logic changes are as follows:
- Move AncestorNode from blockIndex to blockNode and greatly simplify
since it no longer has to deal with the possibility of dynamically
loading nodes and related failures
- Rename RelativeNode to RelativeAncestor, move to blockNode, and
redefine in terms of AncestorNode
- Move CalcPastMedianTime from blockIndex to blockNode and remove no
longer necessary test for nil
- Change calcSequenceLock to use Ancestor instead of RelativeAncestor
since it reads more clearly
Instead of having both server and blockManager be aware of the
txProcessed and blockProcessed channels, now the server passed them as
method arguments to blockProcessor.
This takes care of a few minor nits on the recently merged subscribe
code. In particular:
- Avoid extra unnecessary allocation on notifications slice
- Avoid defer overhead on notification mutex in sendNotifications
- Make test function comment start with the name of the function per
Effective Go guidelines
- Use constant for number of subscribers in test
- Don't exceed column 80 in test print
This refactors the RPC server to accept and take ownership of already
configured listeners and refactors the logic to setup those listeners to
the server. This mirrors the logic used by the connection manager and
is desirable since it is another step closer to being able to split the
RPC server code out into a separate package and will make it much easier
to internally test since it allows creating mock listeners.
This modifies all of the RPC code to use the chain parameters that are
associated with the RPC server instead of the global activeNetParams and
thus moves one step closer to being able to split the RPC server out
into a separate package.
This decouples the RPC server from the internal btcd server to move
closer to being able to split it out into a separate package.
In order to accomplish this, it introduces an rpcserverConfig type and
several new interfaces, named rpcserverPeer, rpcserverConnManager, and
rpcserverBlockManager, which are necessary to break the direct
dependencies on the main server and block manager instances.
It also adds concrete implementations of the new interfaces and uses
them to configure the RPC server.
Ultimately, the RPC server should ideally be decoupled even more such
that all of the types in the configuration struct use interfaces instead
of the concrete types. Doing this would make the RPC server much easier
to internally test since it would allow creating lightweight stubs for
the various pieces.
Instead of having the block manager notify the RPC server about
accepted, connected, and disconnected blocks, the RPC server will
directly listen for notifications from the blockchain.
The objective is to remove the reference from blockManager to
server. Since the blockManager is responsible for keeping the mempool
in sync, it should have a direct reference to it.
The BlockChain struct emits notifications for various events, but
it is only possible to register one listener. This changes the
interface and implementations to allow multiple listeners.
This removes the standardness check to reject transactions with a lock
time greater than a maxint32 because the old bitcoind nodes which it was
designed to protect against are no longer valid for other reasons and
thus there are no longer any of them on the network to worry about.
This makes the code for getheaders more consistent with the rest of the
code in terms of making use of existing error functions and using the
same RPC error codes as other handlers.
While here, it also performs the fetching of headers directly instead of
using a function from server which makes it more tightly coupled.
This replaces the ErrDoubleSpend and ErrMissingTx error codes with a
single error code named ErrMissingTxOut and updates the relevant errors
and expected test results accordingly.
Once upon a time, the code relied on a transaction index, so it was able
to definitively differentiate between a transaction output that
legitimately did not exist and one that had already been spent.
However, since the code now uses a pruned utxoset, it is no longer
possible to reliably differentiate since once all outputs of a
transaction are spent, it is removed from the utxoset completely.
Consequently, a missing transaction could be either because the
transaction never existed or because it is fully spent.
If no existing btcctl.conf file exists, btcctl creates a default one
using the RPC username and password from the btcd.conf. If the
--wallet flag is passed, however, it should read from btcwallet.conf
instead.
https://github.com/btcsuite/btcd/issues/875.
This commit updates the block template generation logic to only include
witness transactions once the soft-fork has activated and to also
include the OP_RETURN witness commitment (with additional block weight
accounting).
This commit updates the new segwit validation logic within block
validation to be guarded by an initial check to the version bits state
before conditionally enforcing the logic based off of the state.
This commit adds set of BIP0009 (Version Bits) deployment parameters
for all networks detailing the activation parameters for the segwit
soft-fork.
Additionally, the BIP0009 integration test has been updated to test for
the proper transitioning of version bits state for the segwit soft
fork. Finally, the `getblockchaininfo` test has also been updated to
properly display the state of segwit.
This commit modifies the existing block selection logic to limit
preferentially by weight instead of serialized block size, and also to
adhere to the new sig-op cost limits which are weighted according to
the witness discount.
This commit implements the new block validation rules as defined by
BIP0141. The new rules include the constraints that if a block has
transactions with witness data, then there MUST be a commitment within
the conies transaction to the root of a new merkle tree which commits
to the wtxid of all transactions. Additionally, rather than limiting
the size of a block by size in bytes, blocks are now limited by their
total weight unit. Similarly, a newly define “sig op cost” is now used
to limit the signature validation cost of transactions found within
blocks.
This commit implements the new “weight” metric introduced as part of
the segwit soft-fork. Post-fork activation, rather than limiting the
size of blocks and transactions based purely on serialized size, a new
metric “weight” will instead be used as a way to more accurately
reflect the costs of a tx/block on the system. With blocks constrained
by weight, the maximum block-size increases to ~4MB.
This commit adds verification of the post-segwit standardness
requirement that all pubkeys involved in checks operations MUST be
serialized as compressed public keys. A new ScriptFlag has been added
to guard this behavior when executing scripts.
This commit modifies the op-code execution for OP_IF and OP_NOTIF to
enforce the additional “minimal if” constraints which require the
top-stack item when the op codes are encountered to be either an empty
vector, or exactly [0x01].
This commit adds a new function to btcec: IsCompressedPubKey. This
function returns true iff the passed serialized public key is encoded
in compressed format.
This commit implements the flag activation portion of BIP 0147. The
verification behavior triggered by the NULLDUMMY script verification
flag has been present within btcd for some time, however it wasn’t
activated by default.
With this commit, once segwit has activated, the ScriptStrictMultiSig
will also be activated within the Script VM. Additionally, the
ScriptStrictMultiSig is now a standard script verification flag which
is used unconditionally within the mempool.
This commit implements full witness program validation for the
currently defined version 0 witness programs. This includes validation
logic for nested p2sh, p2wsh, and p2wkh. Additionally, when in witness
validation mode, an additional set of constrains are enforced such as
using the new sighash digest algorithm and enforcing clean stack
behavior within witness programs.
This commit fixes an off-by-one error which is only manifested by the
new behavior of OP_CODESEPARATOR within sig hashes triggered by the
segwit behavior. The current behavior within the Script VM
(txscript.Engine) is known to be fully correct to the extent that it has
been verified. However, once segwit activates a consensus divergence
would emerge due to *when* the program counter was incremented in the
previous code (pre-this-commit).
Currently (pre-segwit) when calculating the pre-image to a transaction
sighash for signature verification, *all* instances of OP_CODESEPARATOR
are removed from the subScript being signed before generating the final
sighash. SegWit has additional nerfed the behavior of OP_CODESEPARATOR
by no longer removing them (and starting after the last instance), but
instead simply starting the subScript to be directly *after* the last
instance of an OP_CODESEPARATOR within the pkScript.
Due to this new behavior, without this commit, an off-by-one error
(which only matters post-segwit), would cause txscript to generate an
incorrect subScript since the instance of OP_CODESEPARATOR would remain
as part of the subScript instead of being sliced off as the new behavior
dictates. The off-by-one error itself is manifested due to a slight
divergence in txscript.Engine’s logic compared to Bitcoin Core. In
Bitcoin Core script verification is as follows: first the next op-code
is fetched, then program counter is incremented, and finally the op-code
itself is executed. Before this commit, btcd flipped the order
of the last two steps, executing the op-code *before* the program
counter was incremented.
This commit fixes the post-segwit consensus divergence by incrementing
the program-counter *before* the next op-code is executed. It is
important to note that this divergence is only significant post-segwit,
meaning that txscript.Engine is still consensus compliant independent of
this commit.
This commit introduces a series of internal and external helper
functions which enable the txscript package to be aware of the new
standard script templates introduced as part of BIP0141. The two new
standard script templates recognized are pay-to-witness-key-hash
(P2WKH) and pay-to-witness-script-hash (P2WSH).
This commit implements most of BIP0143 by adding logic to implement the
new sighash calculation, signing, and additionally introduces the
HashCache optimization which eliminates the O(N^2) computational
complexity for the SIGHASH_ALL sighash type.
The HashCache struct is the equivalent to the existing SigCache struct,
but for caching the reusable midstate for transactions which are
spending segwitty outputs.
This commit modifies the logic within the block manager and service to
preferentially fetch transactions and blocks which include witness data
from fully upgraded peers.
Once the initial version handshake has completed, the server now tracks
which of the connected peers are witness enabled (they advertise
SFNodeWitness). From then on, if a peer is witness enabled, then btcd
will always request full witness data when fetching
transactions/blocks.
This commit modifies the base peer struct to ascertain when a peer is
able to understand the new witness encoding, and specify the peer’s
supported encoding explicitly before/after the version handshake.
This commit implements the new witness encoding/decoding for
transactions as specified by BIP0144. After segwit activation, a
special transaction encoding is used to signal to upgraded nodes that
the transaction being deserialized bares witness data. The prior
BtcEncode and BtcDecode methods have been extended to be aware of the
new signaling bytes and the encoding of witness data within
transactions.
Additionally, a new method has been added to calculate the “stripped
size” of a transaction/block which is defined as the size of a
transaction/block *excluding* any witness data.
This commit adds the new inventory types for segwit which are used by
the responder to explicitly request that transactions/blocks sent for a
particular inv hash should include all witness data.
This commit modifies the existing wire.Message interface to introduce a
new MessageEncoding variant which dictates the exact encoding to be
used when serializing and deserializing messages. Such an option is now
necessary due to the segwit soft-fork package, as btcd will need to be
able to optionally encode transactions/blocks without witness data to
un-upgraded peers.
Two new functions have been introduced: ReadMessageWithEncodingN and
WriteMessageWithEncodingN which wrap BtcDecode/BtcEncode with the
desired encoding format.
This commit introduces the new SFNodeWitness service bit which has been
added to the protocol as part of BIP0144. The new service bit allows
peers on the network to signal their acceptance and adherence to the
new rules defined as part of the segwit soft-fork package.
The IsBech32SegwitPrefix method takes a string prefix and
determines if it is a valid prefix for a Bech32 encoded segwit
address for any of the default or registered networks.
Version 0.15.0 of Bitcoin Core will include a new RPC command that will
allow us to obtain the amount of time (in seconds) that the server has
been running.
This modifies the goclean.sh script that is executed on Travis to
only run the tests without the race detector.
While it is nice to run the race detector on the tests, unfortunately
there is a limit to the number of goroutines that can be launched while
running it. Since Travis is now much slower than it once was, this
causes a ton of false positive failures.
This corrects the assertion in the decodeSpentTxOut function so it does
not improperly cause a panic when unwinding transactions during a reorg
under certain circumstances. In particular, the provided transaction
version that is passed when a stxo entry does not exist is now -1 in
order to properly distinguish it from the zero value.
It also updates the tests accordingly.
This was discovered by the reorg on testnet from block
00000000000018c58c2d2816f03dac327d975a18af6edf1a369df67ecddaf816 to
0000000000001c1161a367156465cc6226e9f862d9c585f94db5779fdf5455ff.
The btclog package has been changed to defining its own logging
interface (rather than seelog's) and provides a default implementation
for callers to use.
There are two primary advantages to the new logger implementation.
First, all log messages are created before the call returns. Compared
to seelog, this prevents data races when mutable variables are logged.
Second, the new logger does not implement any kind of artifical rate
limiting (what seelog refers to as "adaptive logging"). Log messages
are outputted as soon as possible and the application will appear to
perform much better when watching standard output.
Because log rotation is not a feature of the btclog logging
implementation, it is handled by the main package by importing a file
rotation package that provides an io.Reader interface for creating
output to a rotating file output. The rotator has been configured
with the same defaults that btcd previously used in the seelog config
(10MB file limits with maximum of 3 rolls) but now compresses newly
created roll files. Due to the high compressibility of log text, the
compressed files typically reduce to around 15-30% of the original
10MB file.
This slightly optimizes the NAF function by avoiding returning the
unused bit when there is not a carry.
It also adds a bunch of additional unit tests which I made while
debugging.
This modifies the normalize function of the internal field value to
both optimize it and address an issue where the reduction could
lead to an incorrect result with a small range of values. It also adds
tests to ensure the behavior is correct.
The following benchmark shows the relative speedups as a result of the
optimization on my system. In particular, the changes result in
approximately a 14% speedup in Normalize, which ultimately translates to
a 2% speedup in signature verifies.
benchmark old ns/op new ns/op delta
--------------------------------------------------------------------
BenchmarkAddJacobian 1364 1289 -5.50%
BenchmarkAddJacobianNotZOne 3150 3091 -1.87%
BenchmarkScalarBaseMult 134117 132816 -0.97%
BenchmarkScalarBaseMultLarge 135067 132966 -1.56%
BenchmarkScalarMult 411218 402217 -2.19%
BenchmarkSigVerify 671585 657833 -2.05%
BenchmarkFieldNormalize 36.0 31.0 -13.89%
The github markdown interpreter has been changed such that it no longer
allows spaces in between the brackets and parenthesis of links. This
updates the markdown files accordingly.
While here, it also corrects a couple of inconsistencies in regards to
other README.md files in the project.
The github markdown interpreter has been changed such that it no longer
allows spaces in between the brackets and parenthesis of links and now
requires a newline in between anchors and other formatting. This
updates all of the markdown files accordingly.
While here, it also corrects a couple of inconsistencies in some of the
README.md files.
This updates the GetNetworkInfoResult structure to include the latest
fields added to Core for compatibility purposes.
While here, also move the definitions of the subtypes for the result
before their use for consistency.
This changes the nonce generated to detect self connections over to use
pseudo randoms instead of a cryptographically random nonce.
There is really not a good reason for it to be cryptographically strong,
using the prng is much faster, and the prng also doesn't burn entropy.
This removes the field that tracks whether the version has been sent
since it is no longer used after the version negotiation was separated
from the main read and write code.
This commit modifies the existing block validation logic to examine the
current version bits state of the CSV soft-fork, enforcing the new
validation rules (BIPs 68, 112, and 113) accordingly based on the
current `ThesholdState`.
This commit publicly exports the CreateBlock function as it can be very
useful for generating blocks for tests. Additionally, the behavior of
the function has been modified slightly to build off of the genesis
block for the specified chain if the `prevBlock` paramter is nil.
This commit adds BIP-9 deployment parameters for all registered
networks for the CSV soft-fork package.
The mainnet and testnet parameters have been set in accordance to the
finalized BIPs.
For simnet, and the regression net, the activation date is back-dated
in order to allow signaling for the soft-fork at any time. Additionally
the expiration time for simnet and regrets has been set to
math.MaxInt64, meaning they’ll never expire.
Now that glide is used for version management and a specific commit of
the upstream repository can be locked it is no longer necessary to
maintain a fork of the package specifically to keep a stable dependency.
While here, update the glide dependency for btcutil as well since it was
switched to use the upstream path as well.
Invalid tokens: github had made some changes the past months, the old style is not rendering at all, so I fixed that for you.
Other contributors can do the same for all of the project's documents.
Thanks.
This modifies the goclean.sh script that is executed on Travis to
only run the tests once.
While it is nice to see coverage reports in the log, unfortunately it
appears that both the -race and -cover flags can't be used together, and
the tests have grown in complexity such that they are starting to get
close to TravisCI time limits.
This simplifies the code based on the recommendations of the gosimple
lint tool.
Also, it increases the deadline for the linters to run to 10 minutes and
reduces the number of threads that is uses. This is being done because
the Travis environment has become increasingly slower and it also seems
to be hampered by too many threads running concurrently.
This modifies the blockNode and BestState structs in the blockchain
package to store hashes directly instead of pointers to them and updates
callers to deal with the API change in the exported BestState struct.
In general, the preferred approach for hashes moving forward is to store
hash values in complex data structures, particularly those that will be
used for cache entries, and accept pointers to hashes in arguments to
functions.
Some of the reasoning behind making this change is:
- It is generally preferred to avoid storing pointers to data in cache
objects since doing so can easily lead to storing interior pointers
into other structs that then can't be GC'd
- Keeping the hash values directly in the block node provides better
cache locality
Since the code base is currently in the process of changing over to
decouple download and connection logic, but not all of the necessary
parts are updated yet, ensure blocks that are in the database, but do
not have an associated main chain block index entry, are treated as if
they do not exist for the purposes of chain connection and selection
logic.
This refactors the block index logic into a separate struct and
introduces an individual lock for it so it can be queried independent of
the chain lock.
This modifies the block node structure to include a couple of extra
fields needed to be able to reconstruct the block header from a node,
and exposes a new function from chain to fetch the block headers which
takes advantage of the new functionality to reconstruct the headers from
memory when possible. Finally, it updates both the p2p and RPC servers
to make use of the new function.
This is useful since many of the block header fields need to be kept in
order to form the block index anyways and storing the extra fields means
the database does not have to be consulted when headers are requested if
the associated node is still in memory.
The following timings show representative performance gains as measured
from one system:
new: Time to fetch 100000 headers: 59ms
old: Time to fetch 100000 headers: 4783ms
This removes the CalcPastMedianTime since it is now exposed much more
efficiently via the MedianTime field of the BestState snapshot returned
from the BestSnapshot function.
This modifies the block nodes used in the blockchain package for keeping
track of the block index to use int64 for the timestamps instead of
time.Time.
This is being done because a time.Time takes 24 bytes while an int64
only takes 8 and the plan is to eventually move the entire block index
into memory instead of the current dynamically-loaded version, so
cutting the number of bytes used for the timestamp by a third is highly
desirable.
Also, the consensus code requires working with unix-style timestamps
anyways, so switching over to them in the block node does not seem
unreasonable.
Finally, this does not go so far as to change all of the time.Time
references, particularly those that are in the public API, so it is
purely an internal change.
This modifies the rpctest harness and its associated memwallet to make
use of the new filter-based notifications since the old notifications
are now deprecated.
It also updates the glide.lock file to require the necessary
btcrpcclient version.
This modifies the blockchain code to store all blocks that have passed
proof-of-work and contextual validity tests in the database even if they
may ultimately fail to connect.
This eliminates the need to store those blocks in memory, allows them to
be available as orphans later even if they were never part of the main
chain, and helps pave the way toward being able to separate the download
logic from the connection logic.
This modifies the .travis.yml file to install a release version of glide
instead of using its latest master branch. This will prevent upstream
changes from inadvertently breaking the CI builds.
This contains a bit of cleanup and additional logic to improve the
recently-added ability to specify additional checkpoints via the
--addcheckpoint option.
In particular:
- Improve error messages in the checkpoint parsing
- Correct the mergeCheckpoints function to weed out duplicate height
checkpoints while using the most-recently provided one as described by
its comment
- Add an assertion to blockchain.New that the provided checkpoints are
sorted as required
- Keep comments to 80 columns and use two spaces after periods in them to
be consistent with the rest of the code base
- Make the entry in doc.go match the actual btcd -h output
The most recent version of glide creates a working directory named
.glide that includes .go files and thus must be excluded from the find
operation that generates merged test coverage of all packages.
A DNS lookup was being attempted on onion addresses causing
connections to fail. This has been fixed by introducing type
onionAddr (which implements a net.Addr interface) and passing
it to btcdDial.
Also, the following onion related fixes have been made:
* getaddednodeinfo - updated to handle onion addrs.
* TorLookupIP - fixed err being shadowed.
* newServer - rename tcpAddr to netAddr
* addrStringToNetAddr - skip if host is already an IP addr.
* addrStringToNetAddr - err if tor is disabled
* getaddednodeinfo - check if host is already an IP addr.
ScriptVerifyNullFail defines that signatures must be empty if a
CHECKSIG or CHECKMULTISIG operation fails.
This commit also enables ScriptVerifyNullFail at the mempool policy
level.
This updates the data driven transaction script tests to use the most
recent format and test data as implemented by Core so the test data can
more easily be updated and help prove cross-compatibility correctness.
In particular, the new format combines the previously separate valid and
invalid test data files into a single file and adds a field for the
expected result. This is a nice improvement since it means tests can
now ensure script failures are due to a specific expected reason as
opposed to only generically detecting failure as the previous format
required.
The btcd script engine typically returns more fine grained errors than
the test data expects, so the test adapter handles this by allowing
expected errors in the test data to be mapped to multiple txscript
errors.
It should also be noted that the tests related to segwit have been
stripped from the data since the segwit PR has not landed in master yet,
however the test adapter does recognize the new ability for optional
segwit data to be supplied, though it will need to properly construct
the transaction using that data when the time comes.
This converts the majority of script errors from generic errors created
via errors.New and fmt.Errorf to use a concrete type that implements the
error interface with an error code and description.
This allows callers to programmatically detect the type of error via
type assertions and an error code while still allowing the errors to
provide more context.
For example, instead of just having an error the reads "disabled opcode"
as would happen prior to these changes when a disabled opcode is
encountered, the error will now read "attempt to execute disabled opcode
OP_FOO".
While it was previously possible to programmatically detect many errors
due to them being exported, they provided no additional context and
there were also various instances that were just returning errors
created on the spot which callers could not reliably detect without
resorting to looking at the actual error message, which is nearly always
bad practice.
Also, while here, export the MaxStackSize and MaxScriptSize constants
since they can be useful for consumers of the package and perform some
minor cleanup of some of the tests.
If a host is down and doesn't send a TCP RST, the net.Dial function
blocks until the OS times out the connection. Convert to using
DialTimeout with a 30 second default timeout.
Fail fast if glide or gometalinter isn't available.
And also fail if running glide or gometalinter fails.
Before goclean.sh could pass even if glide or gometalinter wasn't
available at all. It seems that even though we use `set -ex`, their
failure was hidden either inside the $() or behind the pipe.
This adds tests which exercise the BIP0009 state transitions using the
newly available soft fork status information in the getblockchaininfo
RPC.
The following is an overview of the tests added:
- Assert the chain height is 0 and the state is ThresholdDefined
- Generate 1 fewer blocks than needed to reach the first state
transition
- Assert chain height is expected and state is still ThresholdDefined
- Generate 1 more block to reach the first state transition
- Assert chain height is expected and state moved to ThresholdStarted
- Generate enough blocks to reach the next state transition window, but
only signal support in 1 fewer than the required number to achieve
ThresholdLockedIn
- Assert chain height is expected and state is still ThresholdStarted
- Generate enough blocks to reach the next state transition window with
only the exact number of blocks required to achieve locked in status
signalling support
- Assert chain height is expected and state moved to ThresholdLockedIn
- Generate 1 fewer blocks than needed to reach the next state transition
- Assert chain height is expected and state is still ThresholdLockedIn
- Generate 1 more block to reach the next state transition
- Assert chain height is expected and state moved to ThresholdActive
In addition, it updates the existing BIP0009 mining tests to include
extra assertions that the chain height is at the expected height in
addition to checking that the bits are correctly set according to the
expected state.
This commit updates the fields of GetBlockChainInfoResult to reflect
the current state of the RPC as widely implemented by other full-node
implementations.
Replace assignments to individual fields of wire.NetAddress with
creating the entire object at once, as one would do if the type was
immutable.
In some places this replaces the creation of a NetAddress with a
high-precision timestamp with a call to a 'constructor' that converts
the timestamp to single second precision. For consistency, the tests
have also been changed to use single-precision timestamps.
Lastly, the number of allocations in readNetAddress have been reduced by
reading the services directly into the NetAddress instead of first into
a temporary variable.
The thresholdState and deploymentState functions expect the block node
for the block prior to which the threshold state is calculated, however
the startup code which checked the threshold states was using the
current best node instead of its parent.
While here, also update the comments and rename a couple of variables to
help make this fact more clear.
The CSV consensus rules dictate that the opcode fails when the
transaction version is not at least version 2, however that only applies
if the disable flag is not set in the sequence.
This is not an issue at the current time because we do not yet enforce
CSV at a consensus level, however, I noticed this discrepancy when doing
a thorough audit of the CSV paths due to the ongoing work to add full
consensus-enforced CSV support.
As a result, this must be merged prior to enabling consensus enforcement
for CSV or it would open up the potential for a hard fork.
This modifies the code to only enforce the fairly expensive BIP0030
(duplicate transcactions) checks when the chain has not yet reached the
BIP0034 activation height (and is not one of the 2 special historical
blocks that break the rule and prompted BIP0034 to being with) since
that BIP made it impossible to create duplicate coinbases and thus
removed the possibility of creating transactions that overwrite older
ones.
This is a rather large optimization because the check is expensive due
to involving a ton of cache misses in the utxoset. For example, the
following are times it took to perform the BIP0030 check on blocks
425490 - 425502 and a system with a relatively old Hitachi spinner HDD:
block 425490: 674.5857ms
block 425491: 726.5923ms
block 425492: 827.6051ms
block 425493: 680.0863ms
block 425494: 722.0917ms
block 425495: 700.0889ms
block 425496: 647.5823ms
block 425497: 445.0565ms
block 425498: 602.5765ms
block 425499: 375.0476ms
block 425500: 771.0979ms
block 425501: 461.5586ms
block 425502: 603.0766ms
As can be seen from these numbers, this reduces the block validation
time by an average of just over half a second for the given
representative data set and hardware.
Signed-off-by: Dave Collins <davec@conformal.com>
Now that all softforking is done via BIP0009 versionbits, replace the
old isMajorityVersion deployment mechanism with hard coded historical
block heights at which they became active.
Since the activation heights vary per network, this adds new parameters
to the chaincfg.Params struct for them and sets the correct heights at
which each softfork became active on each chain.
It should be noted that this is a technically hard fork since the
behavior of alternate chain history is different with these hard-coded
activation heights as opposed to the old isMajorityVersion code. In
particular, an alternate chain history could activate one of the soft
forks earlier than these hard-coded heights which means the old code
would reject blocks which violate the new soft fork rules whereas this
new code would not.
However, all of the soft forks this refers to were activated so far in
the chain history there is there is no way a reorg that long could
happen and checkpoints reject alternate chains before the most recent
checkpoint anyways. Furthermore, the same change was made in Bitcoin
Core so this needs to be changed to be consistent anyways.
This commit adds all of the infrastructure needed to support BIP0009
soft forks.
The following is an overview of the changes:
- Add new configuration options to the chaincfg package which allows the
rule deployments to be defined per chain
- Implement code to calculate the threshold state as required by BIP0009
- Use threshold state caches that are stored to the database in order
to accelerate startup time
- Remove caches that are invalid due to definition changes in the
params including additions, deletions, and changes to existing
entries
- Detect and warn when a new unknown rule is about to activate or has
been activated in the block connection code
- Detect and warn when 50% of the last 100 blocks have unexpected
versions.
- Remove the latest block version from wire since it no longer applies
- Add a version parameter to the wire.NewBlockHeader function since the
default is no longer available
- Update the miner block template generation code to use the calculated
block version based on the currently defined rule deployments and
their threshold states as of the previous block
- Add tests for new error type
- Add tests for threshold state cache
addrmgr.GetAddress() had a parameter `class string` originally intended
to support looking up addresses according to some type of filter such as
IPv4, IPv6, and only those which support specific wire.ServiceFlags
(full nodes, nodes that support bloom filters, nodes that support
segwit, etc). But currently the parameter is unused and also has an
inappropriate type `string`.
If it would ever be used, it's easy to add back and should then get an
appropriate type such as something that allows bitflags to be set so
that the caller could request combinations such as peers that support
IPv6, are full nodes, and support bloom filters.
This corrects an issue introduced by commit
e8f63bc295 where a failure to lookup a
hostname could lead to a panic in certain circumstances. An error is
now returned in that case as expected.
This modifies the error handling path in the peer read loop such that it
will no longer log an error when the error is io.ErrUnexpectedEOF. This
is being done because that error is almost always the result of a peer
being remotely disconnected and thus it isn't useful information to log.
However, since it might actually be due to a malformed message, a reject
message is still queued up to be sent back to the peer (which will
simply be discarded if the peer disconnected) before it is disconnected.
While it would be ideal to only log if it's not due to a disconnect, and
the code already attempts to handle that, it's not 100% possible to
detect upon the read returning an error without attempting to read again
which will not happen until the next loop iteration.
This commit modifies the `ConnManager` to use the `net.Add` interface
through the package instead of a plain string to represent and
manipulate addresses. This change makes the package much more general as
users of the package can possibly utilize custom implementations of the
`net.Addr` interface to establish connections.
More precisely, the `ConnReq` struct has been modified to use a net.Addr
instance explicitly, and the `DialFunc` type has also been modified to
take a `net.Addr` directly. This latter change gives functions that
adhere to the `DialFunc` type more flexibility as to exactly how the
connection is established.
Additionally, the `connmgr.Config.GetNewAddress` configuration option
now directly returns a `net.Addr. This change allows the `connmgr` to be
decoupled from all DNS queries which allows callers to preferentially
select more secure methods like performing DNS lookups over a Tor proxy.
This modifies the signatures of all serverPeer callbacks that are
provided as peer.Listeners to use _ for the first parameter name which
ensures the passed peer can't be used within the function and updates
all references to the server peer.
This helps ensure any overridden methods that might be defined on a
serverPeer will be invoked where directly calling methods on the passed
peer would not.
Also, while here, add a comment to the OnFeeFilter function.
This modifies the connection manager to provide support for accepting
inbound connections on a caller-provided set of listeners and notify the
caller via a callback.
This is only the minimum work necessary to get inbound support into the
connection manager. The intention for future commits is to move more
connection-related logic such as limiting the maximum number of overall
connections and banned peer tracking into the connection manager.
This removes the type definitions for the callback functions in favor of
declaring them directly in the Config struct. This is more consistent
with the rest of the code base and is preferred since it means callers
reviewing the documentation don't have to follow another level of
indirection to figure out the signature.
Rather than accepting a net.Addr interface and returning an error when
it's not specifically a *net.TCPConn, just accept a *net.TCPConn
directly so the compiler will assert it. Also, remove the error return
since it can no longer occur.
This function is a legacy function from way back during initial
development. Nothing actually uses it and it's not very useful anyways
since it requires the connection to be a specific type (net.TCPConn) and
therefore doesn't work right with things that typically provide their
own net.Conn implementation like proxies.
This modifies the goclean.sh script to include the unconvert lint tool
in the gometalinter configuration.
It also bumps the deadline to 45 seconds to give slower TravisCI
build servers more breathing room.
This makes two modifications based on the results of staticcheck.
The first is to use a simplified zero value for the time in the addrmgr
package tests.
The second is check any errors when opening a directory while attempting
to perform old version upgrades.
This switches `goclean` to use `gometalint` instead of running each checking tool one at a time. The `gometalint` tool runs them concurrently.
More importantly it will allow us to easily add additional linters as desired.
The current max orphan transaction size causes problems with dependent
transaction relay due to its artificially small size in relation to the
max standard transaction size.
Consequently, this modifies the orphan transaction policy by increasing
the max size of each orphan to the same value allowed for standard
non-orphan transactions and reducing the default max allowed number of
orphans to 100.
From a memory usage standpoint, the worst case max mem usage prior to
this change was 5MB plus structure and tracking overhead (1000 max
orphans * 5KB max each). With this, that is raised to 10MB (100 max
orphans * 100KB max each) in the worst case.
It is important to note that the values were originally implemented as a
naive means to control the size of the orphan pool before several of the
recent enhancements which more aggressively remove orphans from the
orphan pool were added, so they needed to be evaluated again.
For a very long time prior to recent changes, the orphan pool would
quickly reach the max allowed worst-case usage and effectively stay
there forever whereas with more recent changes, the actual run-time
orphan pool usage is usually much smaller.
Finally, as another point in favor of this change, as the network has
evolved, nodes have generally become better about orphan management and
as such missing ancestors will typically either be broadcast or mined
fairly quickly resulting in fewer overall orphans.
This removes any remaining orphan transactions that were sent by a peer
when it disconnects since it is extremely unlikely that the missing
parents will ever materialize from elsewhere.
This allows a caller-provided tag to be associated with orphan
transactions. This is useful since the caller can use the tag for
purposes such as keeping track of which peers orphans were first seen
from.
Also, since a parameter is required now anyways, it associates the peer
ID with processed transactions from remote peers.
This does the minimum work necessary to refactor the CPU miner code into
its own package. The idea is that separating this code into its own
package will improve its testability and ultimately be useful to other
parts of the codebase such as the various tests which currently
effectively have their own stripped-down versions of this code.
The API will certainly need some additional cleanup and changes to make
it more usable outside of the specific circumstances it was originally
designed to support (namely the generate RPC), however it is better to
do that in future commits in order to keep the changeset as small as
possible during this refactor.
Overview of the major changes:
- Create the new package
- Move cpuminer.go -> cpuminer/cpuminer.go
- Update mining logging to use the new cpuminer package logger
- Rename cpuminerConfig to Config (so it's now cpuminer.Config)
- Rename newCPUMiner to New (so it's now cpuminer.New)
- Update all references to the cpuminer to use the package
- Add a skeleton README.md
This moves the error check for an attempt to call the generate RPC when
on a network that there is virtually no chance of mining a block with
the CPU into the RPC server where it more naturally belongs. The caller
of the CPU should be the one to determine if it wants to allow mining or
not. While here, use a more accurate RPC error code of ErrDifficulty
instead of ErrInternal.
This change is a step towards being able to separate the CPU mining code
into its own subpackage.
The inability for a peer to negotiate is not something that should be a
warning which implies something is wrong. On the contrary, it is quite
expected that various peers will connect (or be connected to) that are
unable to properly negotiate for a variety of reasons. One example would
be a peer that is too old.
Also, while here take care of a few style nits.
This corrects a few issues introduced with the connection manager where
the server was not notifying the connection manager when a connection
request is available again.
The cases resolved are:
- Unable to initialize a server peer instance in response to the connection
- Failure to associate the connection with the server peer instance
- Disconnection of a non-persistent outbound peer
It also changes the log message to a debug in the former case because
it's not something that should be shown to the user as an error given
it's not due to anything the user has misconfigured nor is it even
unexpected if an invalid address is provided.
This adds a new field to the peer struct which stores the protocol
version advertised by the remote peer and updates the StatsSnapshot to
return the advertised version instead of the negotiated version.
Now that glide is used for version management the deps.txt file is no
longer needed. It will still be available in the git history if it is
ever needed.
This modifies the NewMsgTx function to accept the transaction version as
a parameter and updates all callers.
The reason for this change is so the transaction version can be bumped
in wire without breaking existing tests and to provide the caller with
the flexibility to create the specific transaction version they desire.
This does the minimum work necessary to refactor the block template
generation code into the mining package. The idea is that separating
this code into the mining package will greatly improve its testability,
allow independent benchmarking and profiling, and open up some
interesting opportunities for future development related to mining.
There are some areas related to policy and other configuration that
could be further refactored, however it is better to do that in future
commits in order to keep the changeset as small as possible during this
refactor.
Overview of the major changes:
- Move mining.go -> mining/mining.go
- Move mining_test.go -> mining/mining_test.go
- Add logger to mining package
- Update the MINR subsystem to use the new mining package logger
- Export CoinbaseFlags from the mining package
- BlkTmplGenerator is now mining.BlkTmplGenerator
- Update all references to the mining code to use the package
This move the export for MinHighPriority from the mempool package to the
mining package. This should have been done when the priority
calculation code was moved to the mining package.
This modifies the block template generate for the mining code such that
it takes chain instance and params instead of requiring a fully
initialized blockManager instance.
Also, in preparation for being able to more easily separate the code, it
exposes and makes use of two new functions:
- BestSnapshot which returns the state snapshot from the underlying
chain instance
- TxSource which returns the underlying transaction source
This is a step towards being able to separate the mining code into its
own package. No functional change.
This introduces a cpuminerConfig type which contains the necessary
information to break the direct dependency on the main server instance.
This change is a step towards being able to separate the cpu miner into
its own subpackage. No functional change.
This commit adds a new option to the mempool’s policy configuration
which determines which transaction versions should be accepted as
standard.
The default version set by the policy within the server is 2; this
allows accepting transactions which have version 2 enabled in order to
utilize the new sequence locks feature.
This commit adds a new function to the package: `LockTimeToSequence`.
The function is a simple utility function which aides the caller to
mapping a targeted time or block based relative lock-time to the
appropriate sequence number.
This commit introduces the concept of “sequence locks” borrowed from
Bitcoin Core for converting an input’s relative time-locks to an
absolute value based on a particular block for input maturity
evaluation.
A sequence lock is computed as the most distant maturity height/time
amongst all the referenced outputs within a particular transaction.
A transaction with sequence locks activated within any of its inputs
can *only* be included within a block if from the point-of-view of that
block either the time-based or height-based maturity for all referenced
inputs has been met.
A transaction with sequence locks can only be accepted to the mempool
iff from the point-of-view of the *next* (yet to be found block) all
referenced inputs within the transaction are mature.
This introduces a new type named BlkTmplGenerator which encapsulates the
various state needed to generate block templates.
This is useful since it means code that needs to generate block
templates can simply accept the generator rather than needing access to
all of the additional state which in turn will ultimately make it easier
to split the mining code into its own package.
This moves the priority-related code from the mempool package to the
mining package and also exports a new constant named UnminedHeight which
takes the place of the old unexported mempoolHeight.
Even though the mempool makes use of the priority code to make decisions
about what it will accept, priority really has to do with mining since
it influences which transactions will end up into a block. This change
also has the side effect of being a step towards enabling separation of
the mining code into its own package which, as previously mentioned,
needs access to the priority calculation code as well.
Finally, the mempoolHeight variable was poorly named since what it
really represents is a transaction that has not been mined into a block
yet. Renaming the variable to more accurately reflect its purpose makes
it clear that it belongs in the mining package which also needs the
definition now as well since the priority calculation code relies on it.
This will also benefit an outstanding PR which needs access to the same
value.
This fixes a bug where a transaction would lose reference to other
transactions dependant on it when being considered for inclusion in a
new block template. The issue only occurs when the transaction being
considered triggers a change of priority queue ordering to ordering by
fee. It results in none of the dependant transactions being considered
for inclusion in the new block template.
This implements orphan expiration in the mempool such that any orphans
that have not had their ancestors materialize within 15 minutes of their
initial arrival time will be evicted which in turn will remove any other
orphans that attempted to redeem it.
In order to perform the evictions with reasonable efficiency, an
opportunistic scan interval of 5 minutes is used. That is to say that
there is not a hard deadline on the scan interval and instead it runs
when a new orphan is added to the pool if enough time has passed.
The following is an example of running this code against the main
network for around 30 minutes:
23:05:34 2016-10-24 [DBG] TXMP: Expired 3 orphans (remaining: 254)
23:10:38 2016-10-24 [DBG] TXMP: Expired 112 orphans (remaining: 231)
23:15:43 2016-10-24 [DBG] TXMP: Expired 95 orphans (remaining: 206)
23:20:44 2016-10-24 [DBG] TXMP: Expired 90 orphans (remaining: 191)
23:25:51 2016-10-24 [DBG] TXMP: Expired 71 orphans (remaining: 191)
23:30:55 2016-10-24 [DBG] TXMP: Expired 70 orphans (remaining: 105)
23:36:19 2016-10-24 [DBG] TXMP: Expired 55 orphans (remaining: 107)
As can be seen from the above, without orphan expiration on this data
set, the orphan pool would have grown an additional 496 entries.
This introduces a new package to house tests which use the rpctest
package to programmatically drive end-to-end integration testing and
move the existing RPC integration tests into the new package.
This modifies the way orphan removal and processing is done to more
aggressively remove orphans that can no longer be valid due to other
transactions being added or removed from the primary transaction pool.
The net effect of these changes is that orphan pool will typically be
much smaller which greatly improves its effectiveness. Previously, it
would typically quickly reach the max allowed worst-case usage and
effectively stay there forever.
The following is a summary of the changes:
- Modify the map that tracks which orphans redeem a given transaction to
instead track by the specific outpoints that are redeemed
- Modify the various orphan removal and processing functions to accept
the full transaction rather than just its hash
- Introduce a new flag on removeOrphans which specifies whether or not
to remove the transactions that redeem the orphan being removed as
well which is necessary since only some paths require it
- Add a new function named removeOrphanDoubleSpends that is invoked
whenever a transaction is added to the main pool and thus the outputs
they spent become concrete spends
- Introduce a new flag on maybeAcceptTransaction which specifies whether
or not duplicate orphans should be rejected since only some paths
require it
- Modify processOrphans as follows:
- Make use of the modified map
- Use newly available flags and logic work more strictly work with tx
chains
- Recursively remove any orphans that also redeem any outputs redeemed
by the accepted transactions
- Several new tests to ensure proper functionality
- Removing an orphan that doesn't exist is removed both when there is
another orphan that redeems it and when there is not
- Removing orphans works properly with orphan chains per the new
remove redeemers flag
- Removal of multi-input orphans that double spend an output when a
concrete redeemer enters the transaction pool
This introduces a new pool membership test function to the mempool
testing infrastructure and refactors the tests to make use of it.
It is useful since it is common logic that is not only needed in the
existing tests, but will be needed by most mempool-related tests.
Avoid compatibility issues with software that relies on the behavior of
bitcoind's JSON-RPC implementation.
The JSON-RPC 1.0 spec defines that notifications must have their "id"
set to null and states that notifications do not have a response.
A JSON-RPC 2.0 notification is a request with "json-rpc":"2.0", and
without an "id" member. The specification states that notifications
must not be responded to. JSON-RPC 2.0 permits the null value as a
valid request id, therefore such requests are not notifications.
Bitcoin Core serves requests with "id":null or even an absent "id", and
responds to such requests with "id":null in the response.
Btcd does not respond to any request without and "id" or with "id":null,
regardless the indicated JSON-RPC protocol version.
In order to avoid compatibility issues with software relying on
Core's behavior, this commit implements "quirks mode" as follows:
- quirks mode can be enabled via configuration (disabled by default)
- If no JSON-RPC version is indicated in the request, accept and
respond to request with "id":null
- If no JSON-RPC version is indicated in the request, accept and
respond to requests without an "id" member
- In both cases above, use "id":null in the response
- Do not respond to request without an "id" or with "id":null when
JSON-RPC version is indicated in the request (process as notification)
This optimizes the way in which the mempool oprhan map is limited in the
same way the server block manager maps were previously optimized.
Previously the code would read a cryptographically random value large
enough to construct a hash, find the first entry larger than that value,
and evict it.
That approach is quite inefficient and could easily become a
bottleneck when processing transactions due to the need to read from a
source such as /dev/urandom and all of the subsequent hash comparisons.
Luckily, strong cryptographic randomness is not needed here. The primary
intent of limiting the maps is to control memory usage with a secondary
concern of making it difficult for adversaries to force eviction of
specific entries.
Consequently, this changes the code to make use of the pseudorandom
iteration order of Go's maps along with the preimage resistance of the
hashing function to provide the desired functionality. It has
previously been discussed that the specific pseudorandom iteration order
is not guaranteed by the Go spec even though in practice that is how it
is implemented. This is not a concern however because even if the
specific compiler doesn't implement that, the preimage resistance of the
hashing function alone is enough.
The following is a before and after comparison of the function for both
speed and memory allocations:
benchmark old ns/op new ns/op delta
----------------------------------------------------------------
BenchmarkLimitNumOrphans 3727 243 -93.48%
benchmark old allocs new allocs delta
-----------------------------------------------------------------
BenchmarkLimitNumOrphans 4 0 -100.00%
This renames the mempool.Config.RelayNonStd option to AcceptNonStd which
more accurately describes its behavior since the mempool was refactored
into a separate package.
The reasoning for this change is that the mempool is not responsible for
relaying transactions (nor should it be). Its job is to maintain a pool
of unmined transactions that are validated according to consensus and
policy configuration options which are then used to provide a source of
transactions that need to be mined.
Instead, it is the server that is responsible for relaying transactions.
While it is true that the current server code currently only relays txns
that were accepted to the mempool, this does not necessarily have to
be the case. It would be entirely possible (and perhaps even a good
idea as something do in the future), to separate the relay policy from
the mempool acceptance policy (and thus indirectly the mining policy).
Older nodes previously added the IP and port information to the address
manager which proved to be unreliable as an inbound connection from a
peer didn't necessarily mean the peer itself accepted inbound
connections.
This also fixes a bug where the peer package was incorrectly sending
the peer's services as its own.
This commit introduces package connmgr which contains connection
management related functionality.
The following is an overview of the features the package provides:
- Maintain fixed number of outbound connections
- Optional connect-only mode
- Retry persistent connections with increasing back-off
- Source peers from DNS seeds
- Use Tor to resolve DNS
- Dynamic ban scores
- Test coverage
In addition, btcd has been refactored to make use of the new package by
extending the connection manager to work with the server to source and
maintain peer connections. The following is a broad overview of the
changes to integrate the package:
- Simplify peer state by removing pending, retry peers
- Refactor to remove retries which are now handled by connmgr
- Use callback to add addresses sourced from the DNS seed
Finally the following connection-related things have been improved as a
part of this refactor:
- Fixes 100% cpu usage when network is down (#129)
- Fixes issues with max peers (#577)
- Simplify outbound peer connections management
This modifies the recently-added NullDataScript function in several
ways in an effort to make them more consistent with the tests in the
rest of the code base and improve/correct the logic:
- Use the hexToBytes and mustParseShortForm functions
- Consistently format the test errors
- Replace the valid bool flag with an expected error and test against it
- Ensure the returned script type is the expected type in all cases
This adds a new function named NullDataScript to the txscript package that returns a provably-pruneable OP_RETURN script with the provided data. The function will return an error if the provided data is larger than the maximum allowed length for a nulldata script to be be considered standard.
This modifies the rpctest framework to start btcd with the appropriate
network flags depending on the provided parameters.
Previously, it always started btcd with --simnet even if other
parameters, such as those for the regression test network, were
provided.
This modifies the ExtractCoinbaseHeight function to recognize small
canonically serialized block heights in coinbase scripts of blocks
higher than version 2.
This allows regression test chains in which blocks encode the serialized
height in the coinbase starting from block 1.
Putting the test code in the same package makes it easier for forks
since they don't have to change the import paths as much and it also
gets rid of the need for internal_test.go to bridge.
Also, do some light cleanup on a few tests while here.
This corrects the isNullData standard transaction type test to work
properly with canonically-encoded data pushes. In particular, single
byte data pushes that are small integers (0-16) are converted to the
equivalent numeric opcodes when canonically encoded and the code failed
to detect them properly.
It also adds several tests to ensure that both canonical and
non-canonical nulldata scripts are recognized properly and modifies the
test failure print to include the script that failed.
This does not affect consensus since it is just a standardness check.
This coincides with the mempool only, policy change which enforces
transaction finality according to the median-time-past rather than
blockheader timestamps. The behavior is pre-cursor to full blown BIP
113 consensus deployment, and subsequent activation.
As a result, the TimeSource field in the mempoolConfig is no longer
needed so it has been removed. Additionally, checkTransactionStandard has been
modified to instead take a time.Time as the mempool is no longer explicitly
dependant on a Chain instance.
This commit adds an additional closure function to the mempool’s config
which computes the median time past from the point of view of the best
node in the chain. The mempool test harness has also been updated to allow
setting a mock median time past for testing purposes.
In addition to increasing the testability of the mempool, this commit
should also speed up transaction and block validation for BIP 113 as
the MTP no longer needs to be re-calculated each time from scratch.
Putting the test code in the same package makes it easier for forks
since they don't have to change the import paths as much and it also
gets rid of the need for internal_test.go to bridge.
Also, remove the exception from the lint checks about returning the
unexported type since it is no longer required.
This adds new tests to the TestNormalize, TestMul, TestAdd2 functions
which trigger an issue with modular reduction that was fixed in the
prevous commit to prevent regressions.
As noted in issue #706, the existing code had an issue where the
normalized result was > P when both the first and second words of the
field representation being normalized were BOTH greater than or equal to
the first and second words of P. Although this condition is rare in
practice, it needs to be handled properly.
This resolves the issue by comparing the low words in the final
reduction step against the normalized low order prime bits to ensure the
final subtraction occurs correctly any time they're > P. This approach
retains the constant time property as well.
This adds a full-blown testing infrastructure in order to test consensus
validation rules. It is built around the idea of dynamically generating
full blocks that target specific rules linked together to form a block
chain. In order to properly test the rules, each test instance starts
with a valid block that is then modified in the specific way needed to
test a specific rule.
Blocks which exercise following rules have been added for this initial
version. These tests were largely ported from the original Java-based
'official' block acceptance tests as well as some additional tests
available in the Core python port. It is expected that further tests
can be added over time as consensus rules change.
* Enough valid blocks to have a stable base of mature coinbases to spend
for futher tests
* Basic forking and chain reorganization
* Double spends on forks
* Too much proof-of-work coinbase (extending main chain, in block that
forces a reorg, and in a valid fork)
* Max and too many signature operations via various combinations of
OP_CHECKSIG, OP_MULTISIG, OP_CHECKSIGVERIFY, and OP_MULTISIGVERIFY
* Too many and max signature operations with offending sigop after
invalid data push
* Max and too many signature operations via pay-to-script-hash redeem
scripts
* Attempt to spend tx created on a different fork
* Attempt to spend immature coinbase (on main chain and fork)
* Max size block and block that exceeds the max size
* Children of rejected blocks are either orphans or rejected
* Coinbase script too small and too large
* Max length coinbase script
* Attempt to spend tx in blocks that failed to connect
* Valid non-coinbase tx in place of coinbase
* Block with no transactions
* Invalid proof-of-work
* Block with a timestamp too far in the future
* Invalid merkle root
* Invalid proof-of-work limit (bits header field)
* Negative proof-of-work limit (bits header field)
* Two coinbase transactions
* Duplicate transactions
* Spend from transaction that does not exist
* Timestamp exactly at and one second after the median time
* Blocks with same hash via merkle root tricks
* Spend from transaction index that is out of range
* Transaction that spends more that its inputs provide
* Transaction with same hash as an existing tx that has not been
fully spent (BIP0030)
* Non-final coinbase and non-coinbase txns
* Max size block with canonical encoding which exceeds max size with
non-canonical encoding
* Spend from transaction earlier in same block
* Spend from transaction later in same block
* Double spend transaction from earlier in same block
* Coinbase that pays more than subsidy + fees
* Coinbase that includes subsidy + fees
* Invalid opcode in dead execution path
* Reorganization of txns with OP_RETURN outputs
* Spend of an OP_RETURN output
* Transaction with multiple OP_RETURN outputs
* Large max-sized block reorganization test (disabled by default since
it takes a long time and a lot of memory to run)
Finally, the README.md files in the main and docs directories have been
updated to reflect the use of the new testing framework.
This removes the exported CalcPastTimeMedian function from the
blockchain package as it is no longer needed since the information is
now available via the BestState snapshot.
Also, update the only known caller of this, which is the chain state in
block manager, to use the snapshot instead. In reality, now that
everything the block manager chain state provides is available via the
blockchain BestState snapshot, the entire thing can be removed, however
that will be done in a separate to commit to keep the changes targeted.
This makes the enforcement of the bloom filter service bit much more
strict. In particular, it does the following:
- Moves the enforcement of the bloom filter service bit out of the peer
package and into the server so the server can ban as necessary
- Disconnect peers that send filter commands when the server is
configured to disable them regardless of the protocol version
- Bans peers that are a high enough protocol version that they are
supposed to observe the service bit is disabled, but ignore it and
send filter commands regardless.
As an added bonus, this fixes the old logic which had a bug in that it
was examining the *remote* peer's supported services in order to choose
whether or not to disconnect instead of the *local* server's supported
services.
This modifies the blockchain.ProcessBlock function to return an
additional boolean as the first parameter which indicates whether or not
the block ended up on the main chain.
This is primarily useful for upcoming test code that needs to be able to
tell the difference between a block accepted to a side chain and a block
that either extends the main chain or causes a reorganize that causes it
to become the main chain. However, it is also useful for the addblock
utility since it allows a better error in the case a file with out of
order blocks is provided.
allAddr was being allocated with counters instead of the actual size
of the address map. This led to the possibility of including nils
in the returned slice, which resulted in a panic.
This adds a new build tag named rpctest which must be set in order for
rpctest-based tests to be executed. The new build tag is also added to
the goclean.sh script which is executed by Travis during continuous
integration builds.
This change is being made because the rpctest framework requires
additional careful user configuration to ensure the version of btcd
under test can be programmatically launched from the system path with
all of the necessary ports open whereas all of the other tests are
self-contained within the test binary itself.
Since said additional configuration is typically not done, it leads to a
lot of false positives. Putting the tests behind a build tag allows
them to remain to be available and run during continuous integration
without imposing the additional configuration requirements on users.
Use os.Getpid() to get the process ID, not os.Getppid(), which returns
the parent process ID. This resulted in multiple calls to
generateListeningAddresses() getting the same listening ports.
This modifies the ports that are selected for use for the p2p and rpc
ports to start with a port that is based on the process id instead of a
hard-coded value. The chosen ports are incremented for each running
instance similar to the previous code except the p2p and rpc ports and
now split into ranges instead of being 2 apart.
This is being done because the previous code only worked for a single
process which means it prevented the ability to run tests in parallel.
The new approach will work with multiple processes, however it must be
stated that there is still a very small probability that the stars could
align resulting in the same ports being selected.
Finally, this also reverts the recent change to run tests serially since
this fixes the underlying cause for that change.
This modifies the goclean.sh to execute all the tests amongst
the packages serially. The default behavior of the `go test` command is
to execute all tests in parallel amongst the listed packages. This
behavior can at times cause tests which use the `rpctest` package to
fail due to multiple `btcd` nodes attempting to bind to the same port
simultaneously. As only one node can successfully bind to the port, the
btcd processes for the other concurrent harness instances exit silently
causing the RPC clients to fail with connection timeouts as their
target process no longer exists. Executing all tests serially
eliminates such a race condition which can cause non-deterministic test
failures.
This modifies the code which handles failed server responses to attempt
to unmarshal the response bytes as a regular JSON-RPC response
regardless of the HTTP status code and, if that fails, return an error
that includes the status code as well as the raw response bytes.
This is being done because some JSON-RPC servers, such as Bitcoin Core,
don't follow the intended HTTP status code meanings. For example, if
you request a block that doesn't exist, it returns a status code of 500
which is supposed to mean internal server error instead of a standard
200 status code (since the HTTP request itself was successful) along
with the JSON-RPC error response.
The result is that errors from Core will now show up as an actual
RPCError instead of an error with the raw JSON bytes.
This also has the benefit of returning the HTTP status code in the
error for real HTTP failure cases such as 401 authentication failures,
which previously would just be an empty error when used against Core
since it doesn't return the actual response along with the status code
as it should.
This changes the GetBlockVersion API to not send a third parameter.
The third parameter is a boolean that expands the transaction data
structures as well. However, Bitcore Core does not recognize this
feature.
GetBlockVerbose now only takes a hash value.
Users of the GetBlockVerbose(hash, true) must now use
GetBlockVerboseTx(hash).
This modifies the return value of the GetBlock and GetBlockAsync
functions to a raw *wire.MsgBlock instead of *btcutil.Block.
This is being done because a btcutil.Block assumes there is a height
associated with the block and the block height is not part of a raw
serialized block as returned by the underlying non-verbose RPC API
getblock call and thus is not available to populate a btcutil.Block
correctly.
A raw wire.Block correctly matches what the underlying RPC call is
actually returning and thus avoids the confusion that could occur.
GetBlockHeader returns the blockheader of the specified blockhash.
GetBlockHeaderVerbose returns the data structure with information about
the blockheader of the specified blockhash.
Fixes#89
This modifies the rpctest harness to call the TearDown function in the
SetUp error path. This ensures any resources, such as temp directories,
that were created during setup before whatever the failure that caused
the error are properly cleaned up.
This corrects the case where any errors that might have happened when
creating the config file were not being displayed due to the logging
system not being initialized yet.
While here, consistently use fmt.Fprint{f,ln} throughout the loadConfig
function even after the logging system is initialized since it will
help prevent copy/paste errors such as the one that induced this change
to begin with.
This changes the transaction input standardness checks as follows:
- Allow any script in a pay-to-script-hash transaction to be relayed and
mined so long as it has no more than 15 signature operations
- Remove the obsolete checks which naively calculated the number of
expected inputs in favor of the more accurate ScriptVerifyCleanStack
and ScriptVerifySigPushOnly functionality of the script engine that was
added after these checks.
This commit modifies the current set of integration tests to ensure
that that the main harness always rejects non-standard transactions.
With this in place, even though we’re using simnet parameters, we
ensure that transaction acceptance/validation is identical to that of
main net.
This commit adds two new cli flags: one for accepting non-std
transactions, and the other for rejecting non-std transactions.
The two flag are rejected when using concurrently. Config parsing is
set up such that, the desired policy expressed via the config always
overrides the policy set by default for a particular chain.
The doc.go files and the sample-btcd.conf file have been updated to document
the new flags exposing further policy control.
This adds a basic test harness infrastructure for the mempool package
which aims to make writing tests for it much easier.
The harness provides functionality for creating and signing transactions
as well as a fake chain that provides utxos for use in generating valid
transactions and allows an arbitrary chain height to be set. In order
to simplify transaction creation, a single signing key and payment
address is used throughout and a convenience type for spendable outputs
is provided.
The harness is initialized with a spendable coinbase output by default
and the fake chain height set to the maturity height needed to ensure
the provided output is in fact spendable as well as a policy that is
suitable for testing.
Since tests are in the same package and each harness provides a unique
pool and fake chain instance, the tests can safely reach into the pool
policy, or any other state, and change it for a given harness without
affecting the others.
In order to be able to make use of the existing blockchain.Viewpoint
type, a Clone method has been to the UtxoEntry type which allows the
fake chain instance to keep a single view with the actual available
unspent utxos while the mempool ends up fetching a subset of the view
with the specifically requested entries cloned.
To demo the harness, this also contains a couple of tests which make use
of it:
- TestSimpleOrphanChain -- Ensures an entire chain of orphans is
properly accepted and connects up when the missing parent transaction
is added
- TestOrphanRejects -- Ensure orphans are actually rejected when the
flag on ProcessTransactions is set to reject them
This modifies the config for the new mempool package such that it takes
a callback function to obtain the best chain height instead of requiring
a fully initialized blockchain.BlockChain instance.
This will make it much easier to test the mempool since the tests will
be able to provide their own height function to test various
functionality without having create and manipulate full blocks and chain
instances.
This adds a new field to the best chain state snapshot for the
calculated past median time as returned by the calcPastMedianTime
function. This is useful since it provides fast access to it without
having to acquire the chain lock which is needed to recalculate it.
This will ultimately allow the associated exported function to be
removed since it only exists to be able to calculate this exact value,
however this commit only introduces the new field in order to keep the
changes minimal.
This commit adds a `defer` statement at the top of `TestRpcServer`
which will attempt a `recover` which tears down all active harnesses in
the event that one of the tests causes a panic in the main goroutine.
Before this commit, if a buggy test caused a panic while all integration
tests were being executed, then any active harnesses would fail to be
properly torn down. This would cause the running btcd processes to be
leaked, possibly interfering with future test runs until the process was
manually killed. This commit fixes such behavior.
In order to aide in debugging, when a test panics, the test number is
printed out along with a full stack-trace from the start of the test to
the panic point.
This corrects several issues with the new rpctest package.
The following is an overview of the issues that have been corrected:
- The JoinNodes code was incorrect for both mempool and blocks.
- For mempool it was only checking the first node against itself
- For blocks it was only testing the height instead of hash and height
which means the chains could be different and it would claim they're
synced
- The test for mempool joins was inaccurate in a few ways:
- It was generating separate chains such that the generated
transaction was invalid (an orphan) on one chain, but not
the other
- Mempools are not automatically synced when nodes are connected and
there is a large random delay before any transaction rebroadcast
happens, so it can't be relied on for the purposes of this test
- The test for block joins was generating two independent chains of the
same height with the same difficulty and was only passing due to the
aforementioned bug in JoinNodes
- All of the ConnectNode calls were connecting the main harness outbound
to the local test harness instances
- This is not correct because ConnectNode makes the outbound
connection persistent, which means once the local test harness is
gone, it would keep trying to connect for the remainder of the tests
to a node that is never coming back and instead ends up connecting to
an independent test harness.
This commit modifies the existing Travis configuration to also install
the btcd binary within the container’s PATH.
This change is necessary in order for tests written against the new
rpctest package to work properly within the Travis testing environment.
This commit introduces a new file: rpcserver_test.go dedicated for
including integration tests for btcd using the new rpctest package.
The tests are created using a TestMain instance first creates a single
main harness which is intended to be re-used across tests instances.
Afterwards all registered RPC tests are executed, with proper clean up
being executed regardless of the passing state of the tests.
The following RPC calls are excessed by the initial set of tests added:
* getbestblock
* getblockcount
* getblockhash
This commit adds a new package (rpctest) which provides functionality
for writing automated black box tests to exercise the RPC interface.
An instance of a rpctest consists of an active btcd process running in
(typically) --simnet mode, a btcrpcclient instance connected to said
node, and finally an embedded in-memory wallet instance (the memWallet)
which manages any created coinbase outputs created by the mining btcd
node.
As part of the SetUp process for an RPC test, a test author can
optionally opt to have a test blockchain created. The second argument
to SetUp dictates the number of mature coinbase outputs desired. The
btcd process will then be directed to generate a test chain of length:
100 + numMatureOutputs.
The embedded memWallet instance acts as a minimal, simple wallet for
each Harness instance. The memWallet itself is a BIP 32 HD wallet
capable of creating new addresses, creating fully signed transactions,
creating+broadcasting a transaction paying to an arbitrary set of
outputs, and querying the currently confirmed balance.
In order to test various scenarios of blocks containing arbitrary
transactions, one can use the Generate rpc call via the exposed
btcrpcclient connected to the active btcd node. Additionally, the
Harness also exposes a secondary block generation API allowing callers
to create blocks with a set of hand-selected transactions, and an
arbitrary BlockVersion or Timestamp.
After execution of test logic TearDown should be called, allowing the
test instance to clean up created temporary directories, and shut down
the running processes.
Running multiple concurrent rpctest.Harness instances is supported in
order to allow for test authors to exercise complex scenarios. As a
result, the primary interface to create, and initialize an
rpctest.Harness instance is concurrent safe, with shared package level
private global variables protected by a sync.Mutex.
Fixes#116.
The protocol was silently upgrade in Core some time ago to enforce a
limit of 256 for the user agent in version messages. This updates wire
to coincide with that change and consequently includes a wire version
bump to 0.4.1.
This does the minimum work necessary to refactor the mempool code into
its own package. The idea is that separating this code into its own
package will greatly improve its testability, allow independent
benchmarking and profiling, and open up some interesting opportunities
for future development related to the memory pool.
There are likely some areas related to policy that could be further
refactored, however it is better to do that in future commits in order
to keep the changeset as small as possible during this refactor.
Overview of the major changes:
- Create the new package
- Move several files into the new package:
- mempool.go -> mempool/mempool.go
- mempoolerror.go -> mempool/error.go
- policy.go -> mempool/policy.go
- policy_test.go -> mempool/policy_test.go
- Update mempool logging to use the new mempool package logger
- Rename mempoolPolicy to Policy (so it's now mempool.Policy)
- Rename mempoolConfig to Config (so it's now mempool.Config)
- Rename mempoolTxDesc to TxDesc (so it's now mempool.TxDesc)
- Rename txMemPool to TxPool (so it's now mempool.TxPool)
- Move defaultBlockPrioritySize to the new package and export it
- Export DefaultMinRelayTxFee from the mempool package
- Export the CalcPriority function from the mempool package
- Introduce a new RawMempoolVerbose function on the TxPool and update
the RPC server to use it
- Update all references to the mempool to use the package.
- Add a skeleton README.md
This reduces the mempool lock contention by removing an unnecessary
check when responding to a "mempool" request.
In particular, the code first gets a list of all transactions from the
mempool and then iterates them in order to construct the inventory
vectors and apply bloom filtering if it is enabled. Since it is
possible that the transaction was removed from the mempool by another
thread while that list is being iterated, the code was checking if each
transaction was still in the mempool. This is a pointless check because
the transaction might still be removed at any point after the check
anyways. For example, it might be removed after the mempool response
has been sent to the remote peer or even while the loop is still
iterating.
This refactors the notification state mutex out of the state itself to
the client. This is being done since the code makes a copy of the
notification state and accesses that copy immutably, and therefore there
is no need for it to have its own mutex.
This makes vet happy by manually copying the notification state fields
during the deep copy instead of copying the entire struct which contains
an embedded mutex.
This exposes a new function on the ScriptBuilder type named AddOps that
allows multiple opcodes to be added via a single call and adds tests to
exercise the new function.
Finally, it updates a couple of places in the signing code that were
abusing the interface by setting its private script directly to use the
new public function instead.
This rewrites the shutdown logic to simplify the shutdown signalling.
All cleanup is now run from deferred functions in the main function and
channels are used to signal shutdown either from OS signals or from
other subsystems such as the RPC server and windows service controller.
The RPC server has been modified to use a new channel for signalling
shutdown that is exposed via the RequestedProcessShutdown function
instead of directly calling Stop on the server as it previously did.
Finally, it adds a few checks for early termination during the main
start sequence so the process can be stopped without starting all the
subsystems if desired.
This is a backport of the equivalent logic from Decred with a few slight
modifications. Credits go to @jrick.
This moves several of the chain constants to the Params struct in the
chaincfg package which is intended for that purpose. This is mostly a
backport of the same modifications made in Decred along with a few
additional things cleaned up.
The following is an overview of the changes:
- Comment all fields in the Params struct definition
- Add locals to BlockChain instance for the calculated values based on
the provided chain params
- Rename the following param fields:
- SubsidyHalvingInterval -> SubsidyReductionInterval
- ResetMinDifficulty -> ReduceMinDifficulty
- Add new Param fields:
- CoinbaseMaturity
- TargetTimePerBlock
- TargetTimespan
- BlocksPerRetarget
- RetargetAdjustmentFactor
- MinDiffReductionTime
This updates all code to make use of the new chainhash package since the
old wire.ShaHash type and related functions have been removed in favor
of the abstracted package.
Also, while here, rename all variables that included sha in their name
to include hash instead.
This is mostly a backport of some of the same modifications made in
Decred along with a few additional things cleaned up. In particular,
this updates the code to make use of the new chainhash package.
Also, since this required API changes anyways and the hash algorithm is
no longer tied specifically to SHA, all other functions throughout the
code base which had "Sha" in their name have been changed to Hash so
they are not incorrectly implying the hash algorithm.
The following is an overview of the changes:
- Remove the wire.ShaHash type
- Update all references to wire.ShaHash to the new chainhash.Hash type
- Rename the following functions and update all references:
- wire.BlockHeader.BlockSha -> BlockHash
- wire.MsgBlock.BlockSha -> BlockHash
- wire.MsgBlock.TxShas -> TxHashes
- wire.MsgTx.TxSha -> TxHash
- blockchain.ShaHashToBig -> HashToBig
- peer.ShaFunc -> peer.HashFunc
- Rename all variables that included sha in their name to include hash
instead
- Update for function name changes in other dependent packages such as
btcutil
- Update copyright dates on all modified files
- Update glide.lock file to use the required version of btcutil
This is a backport of the chainhash package made in Decred along with a
few additional things cleaned up, finished test coverage, and rewording
of some documentation to make it more generic.
In particular, the new package provides the definition of the hash type
and associated hashing functions which will allow the rest of the code to be
agnostic to the specific hash algorithm.
This only implements the package and does not change any of the code
base over to use it.
Putting the test code in the same package makes it easier for forks
since they don't have to change the import paths as much and it also
gets rid of the need for internal_test.go to bridge.
This same thing should probably be done for the majority of the code
base.
Rather than making the caller to pass in the median time source on
ProcessBlock and IsCurrent, modify the Config struct to include the
median time source and associate it with the chain instance when it is
created.
This is being done because both the ProcessBlock and IsCurrent functions
require access to the blockchain state already, it is a little bit safer
to ensure the time source matches the chain instance state, it
simplifies the caller logic, and it also allows its use within the logic
of the blockchain package itself which will be required by upcoming
rule change warning logic that is part of BIP9.
This commit drastically reduces the number of allocations needed to
deserialize a transaction and its scripts by using the combination of a
free list for initially deserializing the individual scripts along with
copying them into a single contiguous byte slice after the final size is
known and modifying each script in the transaction to point to its
location within the contiguous blob.
The end result is only a single allocation that holds all of the scripts
for a transaction regardless of the total number of scripts it has.
The script free list allows a maximum of 12,500 items with each buffer
being 512 bytes. This implies it will have a peak usage of 6.1MB. The
values were chosen based on profiling data and a desire to allow at
least 100 scripts per transaction to be simultaneously deserialized by
125 peers.
Also, while optimizing, decode directly into the existing previous
outpoint structure of each transaction input in order to avoid the extra
allocation per input that is otherwise caused when the local escapes to
the heap.
The following is a before and after comparison of the allocations
with the benchmarks that did not change removed:
benchmark old allocs new allocs delta
-----------------------------------------------------------
ReadTxOut 1 0 -100.00%
ReadTxIn 2 0 -100.00%
DeserializeTxSmall 7 5 -28.57%
DeserializeTxLarge 11146 6 -99.95%
The current code involves a ton of small allocations which is harsh on
the garbage collector and in turn causes a lot of addition runtime
overhead both in terms of additional memory and processing time.
In order to improve the situation, this drasticially reduces the number
of allocations by creating contiguous slices of objects and
deserializing into them. Since the final data structures consist of
slices of pointers to the objects, they are constructed by pointing them
into the appropriate offset of the contiguous slice.
This could be improved upon even further by converting all of the data
structures provided the wire package to be slices of contiguous objects
directly, however that would be a major breaking API change and would
end up requiring updating a lot more code in every caller. I do think
that ultimately the API should be changed, but the changes in this
commit already makes a massive difference and it doesn't require
touching any of the callers, so it is a good place to begin.
The following is a before and after comparison of the allocations
with the benchmarks that did not change removed:
benchmark old allocs new allocs delta
-----------------------------------------------------------
DeserializeTxLarge 16715 11146 -33.32%
DecodeGetHeaders 501 2 -99.60%
DecodeHeaders 2001 2 -99.90%
DecodeGetBlocks 501 2 -99.60%
DecodeAddr 3001 2002 -33.29%
DecodeInv 50003 3 -99.99%
DecodeNotFound 50002 3 -99.99%
DecodeMerkleBlock 107 3 -97.20%
Since the protocol encodes timestamps differently depending on the
message, the code currently decodes into a local variable and then
converts it to a time.Time. However, this causes an allocation due to
the local having to escape to the heap in order for the readElement
function to write to it.
So, in order to avoid that, this introduces two new types for a
timestamp named uint32Time and int64Time that are encoded as the
respective type on the read. When calling the readElements function,
the time.Time field in the message is cast to a pointer of the
appropriate type which effectively allows the allocations to be avoided.
The following is a before and after comparison of the allocations
with the benchmarks that did not change removed:
benchmark old allocs new allocs delta
----------------------------------------------------------------------
ReadBlockHeader 1 0 -100.00%
DecodeHeaders 4001 2001 -49.99%
DecodeAddr 4001 3001 -24.99%
DecodeMerkleBlock 108 107 -0.93%
This introduces a new binary free list which provides a concurrent safe
list of unused buffers for the purpose of serializing and deserializing
primitive integers to their raw binary bytes.
For convenience, the type also provides functions for each of the
primitive unsigned integers that automatically obtain a buffer from the
free list, perform the necessary binary conversion, read from or write
to the given io.Reader or io.Writer, and return the buffer to the free
list.
A global instance of the type has been introduced with a maximum number
of 1024 items. Since each buffer is 8 bytes, it will consume a maximum
of 8KB. Theoretically, this value would only allow up to 1024 peers
simultaneously reading and writing without having to resort to burdening
the garbage collector with additional allocations. However, due to the
fact the code is designed in such a way that the buffers are quickly
used and returned to the free list, in practice it can support much more
than 1024 peers without involving the garbage collector since it is
highly unlikely every peer would need a buffer at the exact same time.
The following is a before and after comparison of the allocations
with the benchmarks that did not change removed:
benchmark old allocs new allocs delta
-------------------------------------------------------------
WriteVarInt1 1 0 -100.00%
WriteVarInt3 1 0 -100.00%
WriteVarInt5 1 0 -100.00%
WriteVarInt9 1 0 -100.00%
ReadVarInt1 1 0 -100.00%
ReadVarInt3 1 0 -100.00%
ReadVarInt5 1 0 -100.00%
ReadVarInt9 1 0 -100.00%
ReadVarStr4 3 2 -33.33%
ReadVarStr10 3 2 -33.33%
WriteVarStr4 2 1 -50.00%
WriteVarStr10 2 1 -50.00%
ReadOutPoint 1 0 -100.00%
WriteOutPoint 1 0 -100.00%
ReadTxOut 3 1 -66.67%
WriteTxOut 2 0 -100.00%
ReadTxIn 5 2 -60.00%
WriteTxIn 3 0 -100.00%
DeserializeTxSmall 15 7 -53.33%
DeserializeTxLarge 33428 16715 -50.00%
SerializeTx 8 0 -100.00%
ReadBlockHeader 7 1 -85.71%
WriteBlockHeader 10 4 -60.00%
DecodeGetHeaders 1004 501 -50.10%
DecodeHeaders 18002 4001 -77.77%
DecodeGetBlocks 1004 501 -50.10%
DecodeAddr 9002 4001 -55.55%
DecodeInv 150005 50003 -66.67%
DecodeNotFound 150004 50002 -66.67%
DecodeMerkleBlock 222 108 -51.35%
TxSha 10 2 -80.00%
This adds two new flags, --txindex and --addrindex, to the addblock
utility which mirror the flags on btcd. They serve to to specify that
the transaction index and/or address index, respectively, should be
built while importing from the bootstrap file.
This is technically not 100% required since btcd will build the indexes
on the first load (when enabled) if they aren't already built, however
it is much faster to build the indexes as the blocks are being validated
(particularly for the address index), so this makes the capability
available.
This converts the project to allow btcd to be used with the glide
package manager in order to provide stable and reproducible builds
without the user having to jump through all of the hoops as they do
today.
It consists of adding a glide.yaml file which identifies the project
dependencies and locations along with a glide.lock file which contains
the complete dependency tree pinned to specific versions. Glide uses
these files to download the packages (or updates) to a local vendor
directory and checkout the correct pinned versions. The go tool, in
turn, is used to build/install btcd and will use the pinned versions in
the vendor directory.
This also updates TravisCI to build using glide, removes some of the
exceptions in the lint checks which are no longer required, and updates
the README.md with the new instructions needed to build the project with
glide.
This reduces the target ratio of freshly allocated data to live data to
10% in order to limit excessive overallocations by the garbage collector
during data bursts such as processing complex blocks or rapidly
receiving a lot of large transactions.
When an OS reboots or shuts down, it sends all processes SIGTERM before
sending SIGKILL. This allows btcd to do a proper shutdown which most
importantly closes the database.
This adds support for serving headers instead of inventory messages in
accordance with BIP0130. btcd itself does not yet make use of the
feature when receiving data.
This adds decode benchmarks for several of the messages that profiling
has identified to cause a lot of allocations in addition to those that
already exist. By adding these benchmarks, it makes it easier to get
allocation and speed statistics which can in turn be used to compare
future improvements.
The following bencharmarks have been added:
DecodeGetHeaders, DecodeHeaders, DecodeGetBlocks, DecodeAddr, DecodeInv,
DecodeNotFound, and DecodeMerkleBlock
For reference, here is the benchmark data as of this commit.
DecodeGetHeaders 93261 ns/op 24120 B/op 1004 allocs/op
DecodeHeaders 2071263 ns/op 368399 B/op 18002 allocs/op
DecodeGetBlocks 92486 ns/op 24120 B/op 1004 allocs/op
DecodeAddr 850608 ns/op 136202 B/op 9002 allocs/op
DecodeInv 17107172 ns/op 3601447 B/op 150004 allocs/op
DecodeNotFound 17522225 ns/op 3601444 B/op 150004 allocs/op
DecodeMerkleBlock 21062 ns/op 5192 B/op 222 allocs/op
This modifies the benchmarks in the wire package to avoid creating a new
reader for each iteration. This is useful since it means that showing
the memory allocations will only show the function under test instead of
the allocation for the benchmark setup as well.
The following is a before and after comparison of the allocations
with the benchmarks that did not change removed:
benchmark old allocs new allocs delta
------------------------------------------------------------
ReadVarInt1 2 1 -50.00%
ReadVarInt3 2 1 -50.00%
ReadVarInt5 2 1 -50.00%
ReadVarInt9 2 1 -50.00%
ReadVarStr4 4 3 -25.00%
ReadVarStr10 4 3 -25.00%
ReadOutPoint 2 1 -50.00%
ReadTxOut 4 3 -25.00%
ReadTxIn 6 5 -16.67%
DeserializeTxSmall 16 15 -6.25%
DeserializeTxLarge 33430 33428 -0.01%
ReadBlockHeader 8 7 -12.50%
This adds a benchmark for deserializing a large transaction that is
often referred to as the megatransaction since it is the largest Bitcoin
transaction mined to date. It consists of 5569 inputs and 1 output and
its hash is:
bb41a757f405890fb0f5856228e23b715702d714d59bf2b1feb70d8b2b4e3e08.
This is being done so there is a benchmark that tests more of a
worst-case scenario which is a better candidate for identifying and
testing improvements.
The following benchmark results shows the how much more intensive this
transaction is over the existing mock transaction:
DeserializeTxSmall 1000000 1751 ns/op 376 B/op 16 allocs/op
DeserializeTxLarge 300 5093980 ns/op 1672829 B/op 33430 allocs/op
This removes the root field and all references to it from the BlockChain
since it is no longer required.
It was previously required because the chain state was not initialized
when the instance was created. However, that is no longer the case, so
there is no reason to keep it around any longer.
This changes the script template parsing function to use a pointer into
the constant global opcode array for parsed opcodes as opposed to making
a copy of the opcode entries which causes unnecessary allocations.
Profiling showed that after roughly 48 hours of operation, this
copy was the culprit of 207 million unnecessary allocations.
This removes the logging functions that are now implemented in the peer
package as they are no longer used by btcd itself and should have been
removed when they were copied into the peer package.
It is not the responsibility of mempool to relay transactions, so
return a slice of transactions accepted to the mempool due to the
passed transaction to the caller.
This improves the tests of the priority queue to include the secondary
sort ordering as well as adds some manual entries to ensure the edge
conditions are properly tested.
This also brings the priority queue test coverage up to 100%.
having 3 int32s above the uint64s in the struct
will cause misalignment for some 32-bit architectures.
see https://golang.org/pkg/sync/atomic/#pkg-note-BUG
This aligns bytesReceived and bytesSent.
Profiles discovered that lookups into the signature cache included an
expensive comparison to the stored `sigInfo` struct. This lookup had the
potential to be more expensive than directly verifying the signature
itself!
In addition, evictions were rather expensive because they involved
reading from /dev/urandom, or equivalent, for each eviction once the
signature cache was full as well as potentially iterating over every
item in the cache in the worst-case.
To remedy this poor performance several changes have been made:
* Change the lookup key to the fixed sized 32-byte signature hash
* Perform a full equality check only if there is a cache hit which
results in a significant speed up for both insertions and existence
checks
* Override entries in the case of a colliding hash on insert Add an
* .IsEqual() method to the Signature and PublicKey types in the
btcec package to facilitate easy equivalence testing
* Allocate the signature cache map with the max number of entries in
order to avoid unnecessary map re-sizes/allocations
* Optimize evictions from the signature cache Delete the first entry
* seen which is safe from manipulation due to
the pre image resistance of the hash function
* Double the default maximum number of entries within the signature
cache due to the reduction in the size of a cache entry
* With this eviction scheme, removals are effectively O(1)
Fixes#575.
The current code is needlessly checking the number of bytes needed to
serialize the unspentness bitmap in the utxo against a maximum value
that could never be returned because the function takes a uint32 output
index which is treated as a bit offset, and converts it bytes, which
will necessarily be less than a max uint32.
This check also causes a compile error on arm where native integers are
32 bits.
This simply removes the unneeded check.
This adds a test to ensure the priority queue works properly both for
sorting by fee per KB and priorities.
Thanks to @ceejep for the original test code and idea which was
subsequently modified and cleaned up a bit to the code here.
This introduces a new indexing infrastructure for supporting optional
indexes using the new database and blockchain infrastructure along with
two concrete indexer implementations which provide both a
transaction-by-hash and a transaction-by-address index.
The new infrastructure is mostly separated into a package named indexers
which is housed under the blockchain package. In order to support this,
a new interface named IndexManager has been introduced in the blockchain
package which provides methods to be notified when the chain has been
initialized and when blocks are connected and disconnected from the main
chain. A concrete implementation of an index manager is provided by the
new indexers package.
The new indexers package also provides a new interface named Indexer
which allows the index manager to manage concrete index implementations
which conform to the interface.
The following is high level overview of the main index infrastructure
changes:
- Define a new IndexManager interface in the blockchain package and
modify the package to make use of the interface when specified
- Create a new indexers package
- Provides an Index interface which allows concrete indexes to plugin
to an index manager
- Provides a concrete IndexManager implementation
- Handles the lifecycle of all indexes it manages
- Tracks the index tips
- Handles catching up disabled indexes that have been reenabled
- Handles reorgs while the index was disabled
- Invokes the appropriate methods for all managed indexes to allow
them to index and deindex the blocks and transactions
- Implement a transaction-by-hash index
- Makes use of internal block IDs to save a significant amount of
space and indexing costs over the old transaction index format
- Implement a transaction-by-address index
- Makes use of a leveling scheme in order to provide a good tradeoff
between space required and indexing costs
- Supports enabling and disabling indexes at will
- Support the ability to drop indexes if they are no longer desired
The following is an overview of the btcd changes:
- Add a new index logging subsystem
- Add new options --txindex and --addrindex in order to enable the
optional indexes
- NOTE: The transaction index will automatically be enabled when the
address index is enabled because it depends on it
- Add new options --droptxindex and --dropaddrindex to allow the indexes
to be removed
- NOTE: The address index will also be removed when the transaction
index is dropped because it depends on it
- Update getrawtransactions RPC to make use of the transaction index
- Reimplement the searchrawtransaction RPC that makes use of the address
index
- Update sample-btcd.conf to include sample usage for the new optional
index flags
This commit is the first stage of several that are planned to convert
the blockchain package into a concurrent safe package that will
ultimately allow support for multi-peer download and concurrent chain
processing. The goal is to update btcd proper after each step so it can
take advantage of the enhancements as they are developed.
In addition to the aforementioned benefit, this staged approach has been
chosen since it is absolutely critical to maintain consensus.
Separating the changes into several stages makes it easier for reviewers
to logically follow what is happening and therefore helps prevent
consensus bugs. Naturally there are significant automated tests to help
prevent consensus issues as well.
The main focus of this stage is to convert the blockchain package to use
the new database interface and implement the chain-related functionality
which it no longer handles. It also aims to improve efficiency in
various areas by making use of the new database and chain capabilities.
The following is an overview of the chain changes:
- Update to use the new database interface
- Add chain-related functionality that the old database used to handle
- Main chain structure and state
- Transaction spend tracking
- Implement a new pruned unspent transaction output (utxo) set
- Provides efficient direct access to the unspent transaction outputs
- Uses a domain specific compression algorithm that understands the
standard transaction scripts in order to significantly compress them
- Removes reliance on the transaction index and paves the way toward
eventually enabling block pruning
- Modify the New function to accept a Config struct instead of
inidividual parameters
- Replace the old TxStore type with a new UtxoViewpoint type that makes
use of the new pruned utxo set
- Convert code to treat the new UtxoViewpoint as a rolling view that is
used between connects and disconnects to improve efficiency
- Make best chain state always set when the chain instance is created
- Remove now unnecessary logic for dealing with unset best state
- Make all exported functions concurrent safe
- Currently using a single chain state lock as it provides a straight
forward and easy to review path forward however this can be improved
with more fine grained locking
- Optimize various cases where full blocks were being loaded when only
the header is needed to help reduce the I/O load
- Add the ability for callers to get a snapshot of the current best
chain stats in a concurrent safe fashion
- Does not block callers while new blocks are being processed
- Make error messages that reference transaction outputs consistently
use <transaction hash>:<output index>
- Introduce a new AssertError type an convert internal consistency
checks to use it
- Update tests and examples to reflect the changes
- Add a full suite of tests to ensure correct functionality of the new
code
The following is an overview of the btcd changes:
- Update to use the new database and chain interfaces
- Temporarily remove all code related to the transaction index
- Temporarily remove all code related to the address index
- Convert all code that uses transaction stores to use the new utxo
view
- Rework several calls that required the block manager for safe
concurrency to use the chain package directly now that it is
concurrent safe
- Change all calls to obtain the best hash to use the new best state
snapshot capability from the chain package
- Remove workaround for limits on fetching height ranges since the new
database interface no longer imposes them
- Correct the gettxout RPC handler to return the best chain hash as
opposed the hash the txout was found in
- Optimize various RPC handlers:
- Change several of the RPC handlers to use the new chain snapshot
capability to avoid needlessly loading data
- Update several handlers to use new functionality to avoid accessing
the block manager so they are able to return the data without
blocking when the server is busy processing blocks
- Update non-verbose getblock to avoid deserialization and
serialization overhead
- Update getblockheader to request the block height directly from
chain and only load the header
- Update getdifficulty to use the new cached data from chain
- Update getmininginfo to use the new cached data from chain
- Update non-verbose getrawtransaction to avoid deserialization and
serialization overhead
- Update gettxout to use the new utxo store versus loading
full transactions using the transaction index
The following is an overview of the utility changes:
- Update addblock to use the new database and chain interfaces
- Update findcheckpoint to use the new database and chain interfaces
- Remove the dropafter utility which is no longer supported
NOTE: The transaction index and address index will be reimplemented in
another commit.
mempoolPolicy contains the values that configure the mempool policy.
This decouples the values from the internals of btcd to move closer
to a mempool package.
Putting the test code in the same package makes it easier for forks
since they don't have to change the import paths as much and it also
gets rid of the need for internal_test.go to bridge.
This same thing should probably be done for the majority of the code
base.
This modifies the chaincfg package to register the default network
params via the init function instead of manually hard coding their data
into the maps. This is less error prone when adding new default
networks.
A new function named mustRegister has been introduced that panics if
there are any errors when registering the network that the new code
makes use of and appropriate tests have been added.
This optimizes the way in which the maps are limited by the block
manager.
Previously the code would read a cryptographically random value large
enough to construct a hash, find the first entry larger than that value,
and evict it.
That approach is quite inefficient and could easily become a bottleneck
when processing transactions due to the need to read from a source such
as /dev/urandom and all of the subsequent hash comparisons.
Luckily, strong cryptographic randomness is not needed here. The primary
intent of limiting the maps is to control memory usage with a secondary
concern of making it difficult for adversaries to force eviction of
specific entries.
Consequently, this changes the code to make use of the pseudorandom
iteration order of Go's maps along with the preimage resistance of the
hashing function to provide the desired functionality. It has
previously been discussed that the specific pseudorandom iteration order
is not guaranteed by the Go spec even though in practice that is how it
is implemented. This is not a concern however because even if the
specific compiler doesn't implement that, the preimage resistance of the
hashing function alone is enough.
Thanks to @Roasbeef for pointing out the efficiency concerns and the
fact that strong cryptographic randomness is not necessary.
This simply exports and adds some comments to the fields of the
BlockTemplate struct.
This is primarily being done as a step toward being able to separate the
mining code into its own package, but also it makes sense on its own
because code that requests new block template necessarily examines the
returned fields which implies they should be exported.
This prevents the node from repeatedly requesting and rejecting the
same transaction as different peers inv the same transaction.
Idea from Bitcoin Core commit 0847d9cb5fcd2fdd5a21bde699944d966cf5add9
Also, limit the number of both requested blocks and transactions.
The --blocksonly configuration option disables accepting transactions
from remote peers. It will still accept, relay, and rebroadcast
valid transactions sent via RPC or websockets.
This updates the Go versions using when running the TravisCI integration
tests to reflect the latest supported Go versions.
Also, the vet tool moved into the Go source tree as of Go 1.5. Its previous
location in the x/tools repo was deprecated at that time and has now
been removed.
Finally, old tool path is no longer needed, so it has been removed.
The vet tool moved into the Go source tree as of Go 1.5. Its previous
location in the x/tools repo was deprecated at that time and has now
been removed.
This commit updates the .travis.yml configuration to avoid fetching vet
from the old location and to simply use the version now available as
part of the standard Go install.
Also, while here, remove the check for changing the tool path since it
is no longer needed.
This modifies the peer package to add support for the sendheaders
protocol message introduced by BIP0030.
NOTE: This does not add support to btcd itself. That requires the server
and sync code to make use of the new functionality exposed by these
changes. As a result, btcd will still be using protocol version 70011.
This ensures the channel passed to QueueMessage is writable and that
QueueMessage will not read from the channel (write-only).
This change is merely a safety change. If a user of the API passes
a read-only channel to QueueMessage, it will now be caught at compile
time instead of panicking during runtime.
Also update internal functions.
When the RPC server is not running a buffered transaction notification
channel fills and eventually blocks. This commit ensures that the
channel continues to be drained irrespective of the RPC server status.
Since the latest golint no longer works with on Go 1.4 and Go 1.4 is no
longer officially supported by btcsuite, remove it from the
configurations tested by TravisCI.
This commit updates the main README.md and docs/README.md files to
replace the references to the now dead btcgui project with the
Windows-only Paymetheus project.
While here, it also updates some information to make it more current and
accurately describe the current status.
The existing check to only log errors from websocket connections when
the error is not because the connection is being shutdown only detects
the single case where the remote client was shutdown.
This commit modifies that logic to include when the connection is being
forcibly shutdown via closing the quit channel and when it has been
shutdown from a client side close.
Now that Go 1.6 has been released, update the required Go version in the
README to 1.5 and add Go 1.6 to the configurations tested by TravisCI.
Also, while here, update the Go 1.4 and 1.5 versions tested by TravisCI
to the latest point releases.
This updates the findcheckpoint utility to work when there are not
already any checkpoints. This doesn't really matter for Bitcoin at the
current time, but if a new testnet is created it will not have any
checkpoints to start with and this change also means the utility can
work for alts.
While here, switch a couple of error prints to ensure they contain a
final newline.
This removes the intermediate transaction log that was introduced as a
part of the database cache as a workaround for leveldb batches causing
massive memory usage spikes in favor of the recently introduced leveldb
transaction interface which no longer has the memory usage issues.
This approach is preferred because it can avoid the extra memory needed
for the transaction log and therefore all of the intermediate states as
well. As a result, the default cache size has been doubled since it
equals roughly the same amount of overall memory usage and the flush
interval has been raised as well.
Dynamic ban scores consist of a persistent and a decaying component. The
persistent score can be used to create simple additive banning policies
simlar to those found in other bitcoin node implementations. The
decaying score enables the creation of evasive logic which handles
misbehaving peers (especially application layer DoS attacks) gracefully
by disconnecting and banning peers attempting various kinds of flooding.
Dynamic ban scores allow these two approaches to be used in tandem.
This pull request includes the following:
- Dynamic ban score type & functions, with tests for core functionality
- Ban score of connected peers can be queried via rpc (getpeerinfo)
- Example policy with decaying score increments on mempool and getdata
- Logging of misbehavior once half of the ban threshold is reached
- Banning logic can be disabled via configuration (enabled by default)
- User defined ban threshold can be set via configuration
This updates a couple of logging statements to use the serverPeer
instance instead of the embedded peer.Peer so they are consistent with
all of the other log statements.
This commit does not change functionality. It makes the creation of inbound and outbound peers more homogeneous. As a result the Start method of peer was removed as it was found not to be necessary. This is the first of several pull requests/commits designed to make the peer public API and internals less complex.
This implements the wire protocol encoding portion of a new
sendheaders message as described by BIP0130. It purpose is to request
that a peer sends header commands instead of inv commands when
announcing new blocks. This includes a protocol version bump to 70012
and a wire version bump to 0.4.0.
Note that this does not implement logic to handle the command in btcd,
rather it only makes the command available at the wire protocol level.
A future commit which honors the command and therefore provides full
BIP0130 support is still required.
This optimizes the filter address handling code in the RPC server
handleSearchRawTransasctions path in a few ways:
- Only normalize the filter addresses provided in the command once
rather than for every transaction being returned
- Reset the passes filter flag just before it's used
- Use a local for the encoded address to avoid the bounds checking
overhead of accessing it from the slice
- Avoiding subsequent map lookups once the filter has already passed
Also, while here, modify a few of the related comments to match code
style and consistency.
This modifies the conversion of the output index from the JSON-based
test data for valid and invalid transactions as well as the signature
hash type for signature hash tests to first convert to a signed int and
then to an unsigned int. This is necessary because the result of a
direct conversion of a float to an unsigned int is implementation
dependent and doesn't result in the expected value on all platforms.
Also, while here, change the function names in the error prints to match
the actual names.
Fixes#600.
This modifies the test for createrawtransaction to specify the constant
size passed as an int64. This is necessary because the NewCmd function
accepts the parameters as interfaces in order to support varargs and a
raw numeric constant is treated as an integer. Since the constant value
is larger than an int32, this causes certain platforms like ARM which
treat a raw integer as a 32-bit integer to fail to compile.
This is work towards #600.
This commit adds a database cache layer to the ffldb database backend so
that callers can commit multiple transactions without having to incur
the overhead of a disk sync on every new block.
This commit contains a complete redesign and rewrite of the database
package that approaches things in a vastly different manner than the
previous version. This is the first part of several stages that will be
needed to ultimately make use of this new package.
Some of the reason for this were discussed in #255, however a quick
summary is as follows:
- The previous database could only contain blocks on the main chain and
reorgs required deleting the blocks from the database. This made it
impossible to store orphans and could make external RPC calls for
information about blocks during the middle of a reorg fail.
- The previous database interface forced a high level of bitcoin-specific
intelligence such as spend tracking into each backend driver.
- The aforementioned point led to making it difficult to implement new
backend drivers due to the need to repeat a lot of non-trivial logic
which is better handled at a higher layer, such as the blockchain
package.
- The old database stored all blocks in leveldb. This made it extremely
inefficient to do things such as lookup headers and individual
transactions since the entire block had to be loaded from leveldb (which
entails it doing data copies) to get access.
In order to address all of these concerns, and others not mentioned, the
database interface has been redesigned as follows:
- Two main categories of functionality are provided: block storage and
metadata storage
- All block storage and metadata storage are done via read-only and
read-write MVCC transactions with both manual and managed modes
- Support for multiple concurrent readers and a single writer
- Readers use a snapshot and therefore are not blocked by the writer
- Some key properties of the block storage and retrieval API:
- It is generic and does NOT contain additional bitcoin logic such spend
tracking and block linking
- Provides access to the raw serialized bytes so deserialization is not
forced for callers that don't need it
- Support for fetching headers via independent functions which allows
implementations to provide significant optimizations
- Ability to efficiently retrieve arbitrary regions of blocks
(transactions, scripts, etc)
- A rich metadata storage API is provided:
- Key/value with arbitrary data
- Support for buckets and nested buckets
- Bucket iteration through a couple of different mechanisms
- Cursors for efficient and direct key seeking
- Supports registration of backend database implementations
- Comprehensive test coverage
- Provides strong documentation with example usage
This commit also contains an implementation of the previously discussed
interface named ffldb (flat file plus leveldb metadata backend). Here
is a quick overview:
- Highly optimized for read performance with consistent write performance
regardless of database size
- All blocks are stored in flat files on the file system
- Bulk block region fetching is optimized to perform linear reads which
improves performance on spindle disks
- Anti-corruption mechanisms:
- Flat files contain full block checksums to quickly an easily detect
database corruption without needing to do expensive merkle root
calculations
- Metadata checksums
- Open reconciliation
- Extensive test coverage:
- Comprehensive blackbox interface testing
- Whitebox testing which uses intimate knowledge to exercise uncommon
failure paths such as deleting files out from under the database
- Corruption tests (replacing random data in the files)
In addition, this commit also contains a new tool under the new database
directory named dbtool which provides a few basic commands for testing the
database. It is designed around commands, so it could be useful to expand
on in the future.
Finally, this commit addresses the following issues:
- Adds support for and therefore closes#255
- Fixes#199
- Fixes#201
- Implements and closes#256
- Obsoletes and closes#257
- Closes#247 once the required chain and btcd modifications are in place
to make use of this new code
This implementation ensures that all addresses have an equal chance of
being included in the addr message. It also moves the pseudorandom number
generator seeding to package level so that it can be overridden for
testing if required.
When a persistent peer is disconnected (for example due to a
network timeout), a connection retry is issued. The logic for
doing so failed to remove the peer from the peerState, causing
dead peer connections to fill the peerState. Since connections
in the peerState are counted towards the maxPeers limit, this
would cause btcd to eventually stop retrying connection.
This commit fixes the issue by properly removing the peer from
the peerState.
The getaddr msg is usually replied to with an addr msg, but if
the other peer does not have any addresses to share it will not
reply at all (instead of replying with an addr msg with 0 addresses).
Therefore, the getaddr msg is not guaranteed a reply, so this commit
removes it from stall detection to avoid incorrectly kicking
such peers.
In case of an error during protocol negotiation in handleVersionMsg,
immediately break out and prevent further processing of OnVersion
listener which generally depends upon peer attributes like NA to be set
during the negotiation. Fixes#579.
The CPU miner relies on the mempool, so the mempool has to be created
before calling the function to create the CPU miner. When PR #568
introduced the mempool config struct, it moved the mempool creation
after the miner creation, which leads to the CPU miner crashing due to
trying to access a nil mempool.
This move the CPU miner creation after the mempool creation
appropriately.
This creates a skeleton mining package that simply contains a few of the
definitions used by the mining and mempool code.
This is a step towards decoupling the mining code from the internals of
btcd and ultimately will house all of the code related to creating block
templates and CPU mining.
The main reason a skeleton package is being created before the full
blown package is ready is to avoid blocking mempool separation which
relies on these type definitions.
This introduces the concept of a new interface named TxSource which aims
to generically provide a concurrent safe source of transactions to be
considered for inclusion in a new block. This is a step towards
decoupling the mining code from the internals of btcd. Ultimately the
intent is to create a separate mining package.
The new TxSource interface relies on a new struct named miningTxDesc,
which describes each entry in the transaction source. Once this code is
refactored into a separate mining package, the mining prefix can simply
be dropped leaving the type exported as mining.TxDesc.
To go along with this, the existing TxDesc type in the mempool has been
renamed to mempoolTxDesc and changed to embed the new miningTxDesc type.
This allows the mempool to efficiently implement the MiningTxDescs
method needed to satisfy the TxSource interface.
This approach effectively separates the direct reliance of the mining
code on the mempool and its data structures. Even though the memory
pool will still be the default concrete implementation of the interface,
making it an interface offers much more flexibility in terms of testing
and even provides the potential to allow more than one source (perhaps
multiple independent relay networks, for example).
Finally, the memory pool and all of the mining code has been updated to
implement and use the new interface.
This does three things:
- Splits the priority calculation logic from the TxDesc type
- Modifies the calcPriority function to perform the value age
calculation instead of accepting it as a parameter
- Changes the starting priority to be calculated when the transaction is
added to the pool
The first is useful as it is a step towards decoupling the mining code
from the internals of the memory pool. Also, the concept of priority is
related to mining policy, so it makes more sense to have the
calculations separate than being defined on the memory pool tx
descriptor.
The second change has been made because everywhere that uses the
calcPriority function first has to calculate the value age anyways and
by making it part of the function it can be avoided altogether in
certain circumstances thereby provided a bit of optimization.
The third change ensure the starting priority is safe for reentrancy
which will be important once the mempool is split into a separate
package.
This fixes an issue introduced during the peer refactor where persistent
peers that failed the initial connection are not retried as intended.
It also improves the retry logic as follows:
- Make the retry handler goroutine simply use a for loop instead of
launching a new goroutine for each backoff attempt. Even though
goroutines are fairly cheap to create, it is much more efficient to
simply loop
- Change the retry handler to accept a flag if it is the initial attempt
- Rather than dividing the const interval by 2 everywhere and passing
the retry duration in, just half the constant and set the initial
duration to it in the retry handler
Finally, include the address of the peer in the error message when a new
outbound peer can't be created.
This introduces the concept of a mining policy struct which is used to
control block template generation instead of directly accessing the
config struct. This is a step toward decoupling the mining code from
the internals of btcd. Ultimately the intent is to create a separate
mining package.
Now that the memory pool minimum fee calculation code is also
calculating a more precise value instead of rounding up to the nearest
kilobyte boundary, the comment in NewBlockTemplate regarding this
behavior is no longer accurate. Thus, this removes the comment.
Also, while here, change the calculation to use an int64 instead of
float since it matches the precision of the new calculation code used by
the memory pool and can also avoid the need for the slower floating
point math.
When the peer code was refactored, the lists of peers were converted to
maps however the code which runs when a peer disconnects still iterates
them like a slice. This is no longer necessary since they are maps
which means the peer can simply be looked up by its ID.
Also, the old code was comparing the map entry and the peer being
removed by their pointers which could lead to potentially not properly
finding the peer. This issue is also resolved by this change since it
looks up the peer by its ID.
This defines the peer stat fields directly in the Peer struct instead of
defining them in a separate struct and embedding them. It is slightly
more efficient this way and there is really no reason to have them
defined separately anyways since they are not passed around or otherwise
used independently.
Don't log "Established connection" message on new when
DisableConnectOnNew is set and no connection was actually established.
Do log that same message when Connect() successfully connects.
Fixes a rare hang during peer tests due to the same connection being
used for different outbound peers. Also fixed passing a chan to
QueueMessage(..) which was not being waited on, so was not being used.
500 tests with various transactions and scripts, verifying that
calcSignatureHash generates the expected hash in each case.
This requires changing SigHashType to uint32; that won't affect the
standard use-cases, but will make calcSignatureHash behave more like the
Core counterpart for non-standard SigHashType settings, like those in
some of these tests.
This is more efficient and prevents duplicate entries which can
lead to no-longer-orphans being attempted be added to the mempool
multiple times.
Bug found by me, debugged with @davecgh, and patch from @davecgh
This refactors the checkTransactionStandard function, along with related
constants, from the mempool to the policy.go file since it is strictly
related to policy.
In addition, it adds tests for the function which cover all code paths.
ProcessTransaction could have accepted a new transaction into mempool
but could have returned a reject message if a no-longer-orphan
transaction failed to be accepted. This would also skip any
additional no-longer-orphans, keeping them in the orphan pool.
Instead of returning an error incorrectly, log the error and skip
the no-longer-orphan transaction. This allows the rest of the
no-longer-orphans to be processed as well.
This modifies the IP parsing code to work with IPv6 zone ids. This is
needed since the net.ParseIP function does not allow zone ids even
though net.Listen does.
rpcserver:
If the locktime is given, the transaction inputs will be set to a
non-max value, activating the locktime. The locktime for the
new transaction will be set to the given value.
This mimics Bitcoin Core commit 212bcca92089f406d9313dbe6d0e1d25143d61ff
This commit modifies the peer logging so that it will not log an error
when it's due to the remote peer disconnecting or the peer being
forcibly disconnected or shutdown.
First, it removes the documentation section from all the README.md files
and instead puts a web-based godoc badge and link at the top with the
other badges. This is being done since the local godoc tool no longer
ships with Go by default, so the instructions no longer work without
first installing godoc. Due to this, pretty much everyone uses the
web-based godoc these days anyways. Anyone who has manually installed
godoc won't need instructions.
Second, it makes sure the ISC license badge is at the top with the other
badges and removes the textual reference in the overview section.
Finally, it's modifies the Installation section to Installation and
Updating and adds a '-u' to the 'go get' command since it works for both
and thus is simpler.
This commit implements stall detection logic at the peer level to detect
and disconnect peers that are either not following the protocol in
regards to expected response messages or have otherwise stalled. This
is accomplished by setting deadlines for each message type which expects
a response and periodically checking them while properly taking into
account processing time.
There are an increasing number of nodes on the network which claim to be
full nodes, but don't actually properly implement the entire p2p
protocol even though they implement it enough to cause properly
implemented nodes to make data requests to which they never respond.
Since btcd currently only syncs new blocks via single sync peer and,
prior to this commit only had very basic stall detection, this could
lead to a situation where the block download became stalled indefinitely
due to one of these misbehaving peers. This commit fixes that issue
since the stalled peer will now be detected and disconnected which leads
to a new sync peer being selected.
This logic will also fit nicely with the future multi-peer sync model
which is on the roadmap and for which infrastructure work is underway.
Fixes#486 and fixes#229.
This commit modifies the ping logic in the peer to ping on an interval
regardless of what other messages are being sent versus the previous
method of delaying the ping each time a message that is expected to
receive data is sent.
This helps improve the ping statistics and simplifies its logic.
This fleshes out the doc.go documentation which is shown on godoc, the
README.md shown on github, and improves a couple of comments for the
fields in the Config struct.
This commit introduces package peer which contains peer related features
refactored from peer.go.
The following is an overview of the features the package provides:
- Provides a basic concurrent safe bitcoin peer for handling bitcoin
communications via the peer-to-peer protocol
- Full duplex reading and writing of bitcoin protocol messages
- Automatic handling of the initial handshake process including protocol
version negotiation
- Automatic periodic keep-alive pinging and pong responses
- Asynchronous message queueing of outbound messages with optional
channel for notification when the message is actually sent
- Inventory message batching and send trickling with known inventory
detection and avoidance
- Ability to wait for shutdown/disconnect
- Flexible peer configuration
- Caller is responsible for creating outgoing connections and listening
for incoming connections so they have flexibility to establish
connections as they see fit (proxies, etc.)
- User agent name and version
- Bitcoin network
- Service support signalling (full nodes, bloom filters, etc.)
- Maximum supported protocol version
- Ability to register callbacks for handling bitcoin protocol messages
- Proper handling of bloom filter related commands when the caller does
not specify the related flag to signal support
- Disconnects the peer when the protocol version is high enough
- Does not invoke the related callbacks for older protocol versions
- Snapshottable peer statistics such as the total number of bytes read
and written, the remote address, user agent, and negotiated protocol
version
- Helper functions for pushing addresses, getblocks, getheaders, and
reject messages
- These could all be sent manually via the standard message output
function, but the helpers provide additional nice functionality such
as duplicate filtering and address randomization
- Full documentation with example usage
- Test coverage
In addition to the addition of the new package, btcd has been refactored
to make use of the new package by extending the basic peer it provides to
work with the blockmanager and server to act as a full node. The
following is a broad overview of the changes to integrate the package:
- The server is responsible for all connection management including
persistent peers and banning
- Callbacks for all messages that are required to implement a full node
are registered
- Logic necessary to serve data and behave as a full node is now in the
callback registered with the peer
Finally, the following peer-related things have been improved as a part
of this refactor:
- Don't log or send reject message due to peer disconnects
- Remove trace logs that aren't particularly helpful
- Finish an old TODO to switch the queue WaitGroup over to a channel
- Improve various comments and fix some code consistency cases
- Improve a few logging bits
- Implement a most-recently-used nonce tracking for detecting self
connections and generate a unique nonce for each peer
isMultiSig was not verifying the number of pubkeys specified matched
the number of pubkeys provided. This caused certain non-standard
scripts to be considered multisig scripts.
However, the script still would have failed during execution.
NOTE: This only affects whether or not the script is considered
standard and does NOT affect consensus.
Also, add a test for this check.
Also, update TravisCI goclean script to remove the special casing which
ignored 'Id' from the lint output since that exception is no longer
needed. It was previously required due to the old version of btcjson,
but that is no longer in the repo.
This commit exports the ReadVarString and WriteVarString functions so
they are available for callers to use.
A variable length string is encoded as a variable length integer
containing the length of the string followed by the bytes that represent
the string itself.
See https://github.com/bitcoin/bips/blob/master/bip-0065.mediawiki for
more information.
This commit mimics Bitcoin Core commit bc60b2b4b401f0adff5b8b9678903ff8feb5867b
and includes additional tests from Bitcoin Core commit
cb54d17355864fa08826d6511a0d7692b21ef2c9
We've already been generating lowS sigs for quite a while. This removes
the malleability vector.
This mimics Bitcoin Core commit 49dd5c629df0a08cf3b1ea8085c03312d1a81696
Introduce an ECDSA signature verification into btcd in order to
mitigate a certain DoS attack and as a performance optimization.
The benefits of SigCache are two fold. Firstly, usage of SigCache
mitigates a DoS attack wherein an attacker causes a victim's client to
hang due to worst-case behavior triggered while processing attacker
crafted invalid transactions. A detailed description of the mitigated
DoS attack can be found here: https://bitslog.wordpress.com/2013/01/23/fixed-bitcoin-vulnerability-explanation-why-the-signature-cache-is-a-dos-protection/
Secondly, usage of the SigCache introduces a signature verification
optimization which speeds up the validation of transactions within a
block, if they've already been seen and verified within the mempool.
The server itself manages the sigCache instance. The blockManager and
txMempool respectively now receive pointers to the created sigCache
instance. All read (sig triplet existence) operations on the sigCache
will not block unless a separate goroutine is adding an entry (writing)
to the sigCache. GetBlockTemplate generation now also utilizes the
sigCache in order to avoid unnecessarily double checking signatures
when generating a template after previously accepting a txn to the
mempool. Consequently, the CPU miner now also employs the same
optimization.
The maximum number of entries for the sigCache has been introduced as a
config parameter in order to allow users to configure the amount of
memory consumed by this new additional caching.
This commit modifies the SearchRawTransactions and
SearchRawTransactionsVerbose functions to work properly with the latest
searchrawtransactions API in btcd.
In particular, this involves adding a new parameter for the reverse
option which specifies whether or not to include the results in reverse
order.
While current existing numeric opcodes are limited to 4 bytes, new
opcodes may need different limits.
This mimics Bitcoin Core commit 99088d60d8a7747c6d1a7fd5d8cd388be1b3e138
This commit optimizes the createVinList function which is used to
generate the JSON list of transaction inputs. It also makes it more
consistent with the createVinListPrevOut function.
In particular, it entails the following changes:
- Only do a single coinbase check and return right away instead of
checking multiple times inside the loop over the inputs
- Use a pointer for populating the details of each entry to avoid
multiple unnecessary array lookups and bounds checks
- Group all fields that populate the entry for better readability
SFNodeBloom is a new service flag that a node is required to use to
indicate that it supports bloom filtering. This includes a protocol
version bump to 70011 and a wire version bump to 0.3.0.
btcd:
The SFNodeBloom flag is set by default. A new configuration option
--nopeerbloomfilters has been added to to disable bloom filtering.
Any node advertising a version greater than or equal to 70011 that
attempts to use bloom filtering will be disconnected if bloom
filtering is disabled.
This mimics Bitcoin Core commit afb0ccaf9c9e4e8fac7db3564c4e19c9218c6b03
This commit modifies the DisasmString function to use a bytes buffer for
constructing the disassembled string instead of naive string
concatenation. This makes a huge difference when disassembling scripts
with large numbers of opcodes.
The Bitcoin wire protocol includes several fields with their lengths
encoded according to a variable length integer encoding scheme that does
not enforce a unique encoding for all numbers.
This can lead to a situation where deserializing and re-serializing the
same data can result in different bytes. There are no currently known
issues due to this, but it is safer to reject such subtle differences as
they could potentially lead to exploits.
Consequently, this commit modifies the varint decoding function to error
when the value is not canonically encoded which effectively means that
all messages with varints that are not canonically encoded will now be
rejected. This will not cause issues with old client versions in
regards to blocks and transactions since the data is deserialized into
memory and then reserialized before being relayed thereby effectively
erasing any non-canonical encodings.
Also, new tests have been added to ensure non-canonical encodings are
properly rejected and exercise the new code, and the default user agent
version for wire has been bumped to version 0.2.1 to differentiate the
new behavior.
The equivalent logic was implemented in Bitcoin Core by PR 2884.
This commit improves the most-recently used inventory map human readable
string to only show the inventory vectors. and adds tests for the entire
structure to bring its coverage to 100%.
In addition, removes the type assertion check in the Add function since
the internal inventory list is only managed by the type itself and the
tests would now catch any mistakes in the type of entries in the list.
This commit modifies the SearchRawTransactions and
SearchRawTransactionsVerbose functions to work properly with the latest
searchrawtransactions API in btcd.
In particular, this involves changing the return value from
[]*btcjson.TxRawResult to []*btcjson.SearchRawTransactionResult and
adding the additional optional parameter which specifies whether or not
to include information about the previous outputs.
This commit converts all block height references to int32 instead of
int64. The current target block production rate is 10 mins per block
which means it will take roughly 40,800 years to reach the maximum
height an int32 affords. Even if the target rate were lowered to one
block per minute, it would still take roughly another 4,080 years to
reach the maximum.
In the mean time, there is no reason to use a larger type which results
in higher memory and disk space usage. However, for now, in order to
avoid having to reserialize a bunch of database information, the heights
are still serialized to the database as 8-byte uint64s.
This is being mainly being done in preparation for further upcoming
infrastructure changes which will use the smaller and more efficient
4-byte serialization in the database as well.
IsUnspendable takes a public key script and returns whether it is
spendable.
Additionally, hook this into the mempool isDust function, since
unspendable outputs can't be spent.
This mimics Bitcoin Core commit 0aad1f13b2430165062bf9436036c1222a8724da
The comment says "only allow recent nodes (10mins) after we failed 30 times",
but the server actually did the opposite and allowed only recent nodes before
30 failed connection attempts. This corrects the server's behavior.
This commit modifies the createTxRawResult code path along with callers
to work with block headers as opposed to btcutil.Blocks. This in turn
allows the code in handleGetRawTransaction and
handleSearchRawTransactions to perform a much cheaper block header load
as opposed to a full block load.
While here, also very slightly optimize the createVinList function to
avoid creating a util.Tx wrapper and to take advantage of the
btcutil.Amount type added after the function was originally written
This commit updates the merkle block handling to for the latest changes
to the btcutil API and optimizes it along the way.
Previously, the code was inefficiently reloading the transactions for
the matched hashes from the database instead of simply pulling them from
the full block that was used to create the merkle block.
At the current time, there is no difference between the wire encoding
at protocol version 0 and the stable long-term storage format. These
methods are simply for consistency with the other types.
This commit updates the wire tests for transactions which force
serialization and deserialization errors to force an error in the the
transaction lock time path.
This brings the wire test coverage back up to 100%.
This commit correctly replaces persistent peers that are being retried in
the list of persistent peers so it will continue to be retried as
intended.
Also, limit the maximum retry interval for persistent peers to 5 minutes.
Fixes#463.
This change moves IsFinalizedTransaction to txscript and also changes
the first argument to take a wire.MsgTx instead of btcutil.Tx. This
is needed for an upcoming diff in which txscript will require
IsFinalizedTransaction and we do not want to import the btcd/blockchain.
This allows clients watching for both to know when all mined
transaction notifications for the block have been received.
Otherwise, clients would be aware that there is a new block, see the
exact same block hash/height in the next tx notifications, but never
know when all txs from the block have been received and processed.
This fixes a synchronization issue in btcwallet where the wallet would
mark itself synced to some block height before any of those blocks'
transactions were processed. If the RPC connection is lost between
sending the blockconnected notification and receiving the last
transaction notification, the wallet would not know of this and
continue with missing transactions.
The configured onion proxy was not being used due to checking if the
passed string had suffix '.onion', which never matched because the
port number is part of the string.
This commit adds an additional step to the README.md install section to
run the go version command and check the version so people that are
installing it for the first time and ensure they are running a high
enough version and have GOROOT and GOPATH set correctly.
This commit corrects an issue where the reject message sent in response
to rejected transactions had the Cmd field set to "block" instead of
"tx".
Fixes#436.
Tor stream isolation randomizes proxy user credentials resulting in
Tor creating a new circuit for each connection. This makes it more
difficult to correlate connections.
Idea from Wladimir J. van der Laan via Bitcoin Core.
The option -maxorphantx allows the user to specify the number of
orphan transactions to keep in memory.
Also, lower the default max orphan count from 10000 to 1000.
This commit refactors the consensus rule checks for block headers and
blocks in the blockchain package into separate functions. These changes
contain no modifications to consensus rules and the code still passes all
block consensus tests. It is only a refactoring.
This is being done to help pave the way toward supporting concurrent
downloads. While the package already supports headers-first mode up
through the latest checkpoint through the use of the BFFastAdd flag and
hard-coded checkpoints, it currently only works when downloading from a
single peer. In order to support concurrent downloads from multiple
peers, the ability for the caller to do things such as independently
checking a block header (both context-free and full-context checks) will
be needed.
There are several more changes that will be necessary to support
concurrent downloads as well, such as making the package concurrent safe,
modifying it to make use of the new database API, etc. Those changes are
planned for future commits.
The following changes were made to ListTransactionsResult (which
models the long result format used by listtransactions,
listsinceblock, etc.):
- Fee made optional (float64 -> *float64 + omitempty)
- BlockIndex made optional (int64 + omitempty -> *int64 + omitempty)
- InvolvesWatchOnly added (bool + omitempty)
- Vout added (uint32)
The following changes were made to GetTransactionDetailsResult (which
models the short result format of listtransactions):
- InvolvesWatchOnly added (bool + omitempty)
- Fee added (*float64 + omitempty)
- Vout added (uint32)
The combination of pointer types and the omitempty struct tag allow
excluding the field from the JSON object, or including it with the
zero value. This is useful in particular for the fee fields, which
should be included whenever the category is "send" even if the fee is
zero. Other optional fields which are only added to the result object
with non-zero values (such as includeswatchonly) can be reduced to
simply an omitempty tag without the pointer type.
- Create FutureGenerateResult type with Receive() method
- Create GenerateAsync method for Client which returns a
FutureGenerateResult
- Create Generate method for Client which calls GenerateAsync
and then calls Receive() on the returned FutureGenerateResult
This commit contains several changes needed to update the client to use
the latest version of btcjson. In addition, it contains a couple of other
minor changes along the way.
While the underlying changes are quite large, the public API of this
package is still the same, so caller should generally not have to update
their code due to that. However, the underlying btcjson package API has
changed significantly. Since this package hides the vast majority of that
from callers, it should not afffect them very much. However, one area in
particular to watch out for is that the old btcjson.Error is now
btcjson.RPCError, so any callers doing any type assertions there will need
to update.
The following is a summary of the changes:
- The underlying btcjson significantly changed how commands work, so the
internals of this package have been reworked to be based off of requests
instead of the now non-existant btcjson.Cmd interface
- Update all call sites of btcjson.New<Foo>Cmd since they can no longer
error or take varargs
- The ids for each request are part of the request now instead of the
command to match the new btcjson model and are strict uint64s so type
assertions are no longer needed (slightly improved efficiency)
- Remove the old temporary workaround for the getbalance command with an
account of "*" since btcwallet has since been fixed
- Change all instances of JSONToAmount to btcutil.NewAmount since that
function was removed in favor of btcutil.Amount
- Change all btcws invocations to btcjson since they have been combined
Also update to use the new container-based builds
For now, golint has been disabled since it fails due to wanting
HTTPPostMode. This change will be made as a part of the upgrade to use
btcjson2 since it involves an API change.
This commit adds the Move, MoveAsync, MoveMinConf, MoveMinConfAsync,
MoveComment, and MoveCommentAsync functions for issuing the "move" RPC
with various optional parameters.
Closes#16.
Previously the exported Disconnect and Shutdown functions called each other
and therefore needed to release and reacquire the locks which could
potentionally allow a request to sneak in between the lock and unlock.
This commit changes the exported Shutdown function so that everything is
done under the request lock to prevent this possibility. In order to
support this, the shared code between the Disconnect and Shutdown
functions has been refactored into unexported functions and the locking
has been altered accordingly.
Also, replace the sendMessage function with a select over the send and
disconnect channels to prevent any issues with queued up messages.
joint debugging effort between myself and @jrick
Previously, requests could still be sent to a shutdown client and
added to the client's internal data structures, without ever
responding to the future with an error for a shutdown client (causing
hangs when blocking on the future receive). This change fixes this by
performing a non-blocking read of the client's shutdown channel before
adding a request, and responding with the shutdown error if the client
has begun or completed its shutdown.
ok @davecgh
This commit modifies the error handling for websocket connections to fall
back to returning the status text returned from the server if the
handshake fails and it's not due to an authentication or invalid endpoint.
This change set equips the RPC client with handling of non successful
HTTP responses. An HTTP response is considered non successful when its
status code is not in the range 200..299
The string needs to be unmarshaled rather than returned directly otherwise
it will have an extra set of quotes plus it's safter to properly unmarshal
it.
Pointed out by @jrick.
The submitblock RPC returns a string containing an error if it failed for
any reason. This comment detects a non-null return from submitblock and
converts that string to an error which is returned.
This commit modifies the TLS setup to only override the RootCAs for the
TLS connection if certificates are specified. This allows the
Certificates parameter to be ommitted from the connection config to use
the system CAs.
If connecting to a bitcoin RPC server as an HTTP POST client, full
response objects rather than just the raw result bytes were being
passed to the specific result unmarshalers, causing unmarshal errors
for the incorrect JSON types. To fix this, first unmarshal the
response body into a rawResponse, and pass only the raw result bytes
(or an error) to the specific handlers.
This was reported by nskelsey on IRC.
ok @davecgh
There are certain RPCs where an address should be sent as the raw string
instead of the encoded bitcoin address since the RPC handler on the other
end expects "keys" instead of Bitcoin addresses.
For example, the multisignature RPCs addmultisigaddress and createmultisig
can work with pubkeys (compressed, uncompressed, or hybrid) as well as
Bitcoin addresses.
The original issue which prompted these changes was report by Paul Snow on
IRC.
Rather than using bytes.NewBuffer, which is a read/write entity
(io.ReadWriter), use bytes.NewReader which is only a read entitiy
(io.Reader). Benchmarking shows it's slightly faster and it's also
technically more accurate since it ensures the data is read-only.
The btcutil package recently exposed a WIF type to provide more
functionality and simplify working with WIF strings. This commit changes
the DumpPrivKey and ImportPrivKey RPCs to use the new WIF type.
There are several RPCs which accept a pointer to a hash, transaction,
block, etc. Previously not all RPCs handled being passed a nil pointer
consistently.
Closes#4.
This commit adds logic to track all registered notifications that have
been registered by the client in a notification state when the default
automatic reconnect is enabled.
The notification state is then used to reregister for all previously
registered notifications on reconnect. This allows the caller to
continue receiving notifications across reconnect cycles.
Also, since the new package exposes more connection related error
information, add a new ErrInvalidEndpoint which is returned if the
specified enpoint does not appear to be a valid websocket provider and
only return the ErrInvalidAuth error when HTTP authorization failure
status codes are detected.
Closes#1.
Since the example illustrates callbacks for the OnBlockConnected and
OnBlockDisconnected callbacks, also register for the notifications with
NotifyBlocks.
While here, document the fact that most of the callbacks require
registration.
t.Errorf("Didn't expect to get an address because there shouldn't be any address from subnetwork ID %s or nil",localSubnetworkID)
}
// Checks that the total number of addresses incremented although the new address is not full node or a partial node of the same subnetwork as the local node.
numAddrs:=n.TotalNumAddresses()
ifnumAddrs!=1{
t.Errorf("Wrong number of addresses: got %d, want %d",numAddrs,1)
}
// Now we repeat the same process, but now the address has the expected subnetwork ID.
t.Fatalf("As long as the bluest parent's blue score is less then the difficulty adjustment window size, the difficulty should be the same as genesis'")
t.Fatalf("The difficulty should only change when nodeInThePast is in the past of a block bluest parent")
}
tip=nodeInThePast
tip=addNode(setFromSlice(tip),zeroTime)
iftip.bits!=nodeInThePast.bits{
t.Fatalf("The difficulty should only change when nodeInThePast is in the past of a block bluest parent")
}
tip=addNode(setFromSlice(tip),zeroTime)
ifcompareBits(tip.bits,nodeInThePast.bits)>=0{
t.Fatalf("tip.bits should be smaller than nodeInThePast.bits because nodeInThePast increased the block rate, so the difficulty should increase as well")
}
expectedBits:=uint32(0x207ff395)
iftip.bits!=expectedBits{
t.Errorf("tip.bits was expected to be %x but got %x",expectedBits,tip.bits)
t.Logf("Testing tip for block %s (hash %s, height %d)",
item.Name,block.Hash(),blockHeight)
// Ensure hash and height match.
ifdag.SelectedTipHash()!=item.Block.BlockHash()||
dag.ChainHeight()!=blockHeight{//TODO: (Ori) the use of dag.ChainHeight() and virtualBlock.HighestTipHash() is wrong, and was done only for compilation
t.Fatalf("block %q (hash %s, height %d) should be "+
log.Debugf("Adding orphan block %s. This is normal part of netsync process",blockHash)
}else{
log.Infof("Adding orphan block %s",blockHash)
}
dag.addOrphanBlock(block)
returntrue,0,nil
}
// The block has passed all context independent checks and appears sane
// enough to potentially accept it into the block DAG.
err=dag.maybeAcceptBlock(block,flags)
iferr!=nil{
returnfalse,0,err
}
// Accept any orphan blocks that depend on this block (they are
// no longer orphans) and repeat for those accepted blocks until
// there are no more.
err=dag.processOrphans(blockHash,flags)
iferr!=nil{
returnfalse,0,err
}
log.Debugf("Accepted block %s",blockHash)
returnfalse,0,nil
}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
