[NOD-747] Change FinalityInterval to be 24 hours, isCurrent to be true if the DAG's time is less than 12 hours than the present, and change MaxInvPerMsg to be 1 << 17 (#625)

* [NOD-719] Added defers to unlocks

* [NOD-719] Added another defer to another Unlock

* [NOD-719] Added yet another defer to yet another Unlock

.dockerignore

@@ -0,0 +1,2 @@
+vendor/
+**/docker-compose.yml

.gitignore

@@ -2,7 +2,7 @@
 *~
 
 # Databases
-btcd.db
+kaspad.db
 *-shm
 *-wal
 
@@ -17,6 +17,7 @@ btcd.db
 # Folders
 _obj
 _test
+vendor
 
 # Architecture specific extensions/prefixes
 *.[568vq]
@@ -31,3 +32,17 @@ _cgo_export.*
 _testmain.go
 
 *.exe
+ 
+# IDE
+.idea
+.vscode
+debug
+debug.test
+__debug_bin
+
+# CI
+version.txt
+coverage.txt
+
+testdbs/
+coverage.tmp

.travis.yml

@@ -1,4 +0,0 @@
-language: go
-go:
-  - release
-  - tip

CHANGES

@@ -1,301 +0,0 @@
-============================================================================
-User visible changes for btcd
-  A full-node bitcoin implementation written in Go
-============================================================================
-
-Changes in 0.6.0 (Tue Feb 04 2014)
-  - Fix an issue when parsing scripts which contain invalid signatures that
-    caused a chain fork on block
-    0000000000000001e4241fd0b3469a713f41c5682605451c05d3033288fb2244
-  - Correct an issue which could lead to an error in removeBlockNode
-    (https://github.com/conformal/btcchain/issues/4)
-  - Improve addblock utility as follows:
-    - Check imported blocks against all chain rules and checkpoints
-    - Skip blocks which are already known so you can stop and restart the
-      import or start the import after you have already downloaded a portion
-      of the chain
-    - Correct an issue where the utility did not shutdown cleanly after
-      processing all blocks
-    - Add error on attempt to import orphan blocks
-    - Improve error handling and reporting
-    - Display statistics after input file has been fully processed
-  - Rework, optimize, and improve headers-first mode:
-    - Resuming the chain sync from any point before the final checkpoint
-      will now use headers-first mode
-      (https://github.com/conformal/btcd/issues/69)
-    - Verify all checkpoints as opposed to only the final one
-    - Reduce and bound memory usage
-    - Rollback to the last known good point when a header does not match a
-      checkpoint
-    - Log information about what is happening with headers
-  - Improve btcctl utility in the following ways:
-    - Add getaddednodeinfo command
-    - Add getnettotals command
-    - Add getblocktemplate command (wallet-specific)
-    - Add getwork command (wallet-specific)
-    - Add getnewaddress command (wallet-specific)
-    - Add walletpassphrasechange command (wallet-specific)
-    - Add walletlock command (wallet-specific)
-    - Add sendfrom command (wallet-specific)
-    - Add sendmany command (wallet-specific)
-    - Add settxfee command (wallet-specific)
-    - Add listsinceblock command (wallet-specific)
-    - Add listaccounts command (wallet-specific)
-    - Add keypoolrefill command (wallet-specific)
-    - Add getreceivedbyaccount command (wallet-specific)
-    - Add getrawchangeaddress command (wallet-specific)
-    - Add gettxoutsetinfo command (wallet-specific)
-    - Add listaddressgroupings command (wallet-specific)
-    - Add listlockunspent command (wallet-specific)
-    - Add listlock command (wallet-specific)
-    - Add listreceivedbyaccount command (wallet-specific)
-    - Add validateaddress command (wallet-specific)
-    - Add verifymessage command (wallet-specific)
-    - Add sendtoaddress command (wallet-specific)
-  - Continue cleanup and work on implementing the RPC API:
-    - Implement submitblock command
-      (https://github.com/conformal/btcd/issues/61)
-    - Implement help command
-    - Implement ping command
-    - Implement getaddednodeinfo command
-      (https://github.com/conformal/btcd/issues/78)
-    - Implement getinfo command
-    - Update getpeerinfo to support bytesrecv and bytessent
-      (https://github.com/conformal/btcd/issues/83)
-  - Improve and correct several RPC server and websocket areas:
-    - Change the connection endpoint for websockets from /wallet to /ws
-      (https://github.com/conformal/btcd/issues/80)
-    - Implement an alternative authentication for websockets so clients
-      such as javascript from browsers that don't support setting HTTP
-      headers can authenticate (https://github.com/conformal/btcd/issues/77)
-    - Add an authentication deadline for RPC connections
-      (https://github.com/conformal/btcd/issues/68)
-    - Use standard authentication failure responses for RPC connections
-    - Make automatically generated certificate more standard so it works
-      from client such as node.js and Firefox
-    - Correct some minor issues which could prevent the RPC server from
-      shutting down in an orderly fashion
-    - Make all websocket notifications require registration
-    - Change the data sent over websockets to text since it is JSON-RPC
-    - Allow connections that do not have an Origin header set
-  - Expose and track the number of bytes read and written per peer
-    (https://github.com/conformal/btcwire/issues/6)
-  - Correct an issue with sendrawtransaction when invoked via websockets
-    which prevented a minedtx notification from being added
-  - Rescan operations issued from remote wallets are no stopped when
-    the wallet disconnects mid-operation
-    (https://github.com/conformal/btcd/issues/66)
-  - Several optimizations related to fetching block information from the
-    database
-  - General code cleanup
-
-Changes in 0.5.0 (Mon Jan 13 2014)
-  - Optimize initial block download by introducing a new mode which
-    downloads the block headers first (up to the final checkpoint)
-  - Improve peer handling to remove the potential for slow peers to cause
-    sluggishness amongst all peers
-    (https://github.com/conformal/btcd/issues/63)
-  - Fix an issue where the initial block sync could stall when the sync peer
-    disconnects (https://github.com/conformal/btcd/issues/62)
-  - Correct an issue where --externalip was doing a DNS lookup on the full
-    host:port instead of just the host portion
-    (https://github.com/conformal/btcd/issues/38)
-  - Fix an issue which could lead to a panic on chain switches
-    (https://github.com/conformal/btcd/issues/70)
-  - Improve btcctl utility in the following ways:
-    - Show getdifficulty output as floating point to 6 digits of precision
-    - Show all JSON object replies formatted as standard JSON
-    - Allow btcctl getblock to accept optional params
-    - Add getaccount command (wallet-specific)
-    - Add getaccountaddress command (wallet-specific)
-    - Add sendrawtransaction command
-  - Continue cleanup and work on implementing RPC API calls
-    - Update getrawmempool to support new optional verbose flag
-    - Update getrawtransaction to match the reference client
-    - Update getblock to support new optional verbose flag
-    - Update raw transactions to fully match the reference client including
-      support for all transaction types and address types
-    - Correct getrawmempool fee field to return BTC instead of Satoshi
-    - Correct getpeerinfo service flag to return 8 digit string so it
-      matches the reference client
-    - Correct verifychain to return a boolean
-    - Implement decoderawtransaction command
-    - Implement createrawtransaction command
-    - Implement decodescript command
-    - Implement gethashespersec command
-    - Allow RPC handler overrides when invoked via a websocket versus
-      legacy connection
-  - Add new DNS seed for peer discovery
-  - Display user agent on new valid peer log message
-    (https://github.com/conformal/btcd/issues/64)
-  - Notify wallet when new transactions that pay to registered addresses
-    show up in the mempool before being mined into a block
-  - Support a tor-specific proxy in addition to a normal proxy
-    (https://github.com/conformal/btcd/issues/47)
-  - Remove deprecated sqlite3 imports from utilities
-  - Remove leftover profile write from addblock utility
-  - Quite a bit of code cleanup and refactoring to improve maintainability
-
-Changes in 0.4.0 (Thu Dec 12 2013)
-  - Allow listen interfaces to be specified via --listen instead of only the
-    port (https://github.com/conformal/btcd/issues/33)
-  - Allow listen interfaces for the RPC server to be specified via
-    --rpclisten instead of only the port
-    (https://github.com/conformal/btcd/issues/34)
-  - Only disable listening when --connect or --proxy are used when no
-    --listen interface are specified
-    (https://github.com/conformal/btcd/issues/10)
-  - Add several new standard transaction checks to transaction memory pool:
-    - Support nulldata scripts as standard
-    - Only allow a max of one nulldata output per transaction
-    - Enforce a maximum of 3 public keys in multi-signature transactions
-    - The number of signatures in multi-signature transactions must not
-      exceed the number of public keys
-    - The number of inputs to a signature script must match the expected
-      number of inputs for the script type
-    - The number of inputs pushed onto the stack by a redeeming signature
-      script must match the number of inputs consumed by the referenced
-      public key script
-  - When a block is connected, remove any transactions from the memory pool
-    which are now double spends as a result of the newly connected
-    transactions
-  - Don't relay transactions resurrected during a chain switch since
-    other peers will also be switching chains and therefore already know
-    about them
-  - Cleanup a few cases where rejected transactions showed as an error
-    rather than as a rejected transaction
-  - Ignore the default configuration file when --regtest (regression test
-    mode) is specified
-  - Implement TLS support for RPC including automatic certificate generation
-  - Support HTTP authentication headers for web sockets
-  - Update address manager to recognize and properly work with Tor
-    addresses (https://github.com/conformal/btcd/issues/36) and
-    (https://github.com/conformal/btcd/issues/37)
-  - Improve btcctl utility in the following ways:
-    - Add the ability to specify a configuration file
-    - Add a default entry for the RPC cert to point to the location
-      it will likely be in the btcd home directory
-    - Implement --version flag
-    - Provide a --notls option to support non-TLS configurations
-  - Fix a couple of minor races found by the Go race detector
-  - Improve logging
-    - Allow logging level to be specified on a per subsystem basis
-      (https://github.com/conformal/btcd/issues/48)
-    - Allow logging levels to be dynamically changed via RPC
-      (https://github.com/conformal/btcd/issues/15)
-    - Implement a rolling log file with a max of 10MB per file and a
-      rotation size of 3 which results in a max logging size of 30 MB
-  - Correct a minor issue with the rescanning websocket call
-    (https://github.com/conformal/btcd/issues/54)
-  - Fix a race with pushing address messages that could lead to a panic
-    (https://github.com/conformal/btcd/issues/58)
-  - Improve which external IP address is reported to peers based on which
-    interface they are connected through
-    (https://github.com/conformal/btcd/issues/35)
-  - Add --externalip option to allow an external IP address to be specified
-    for cases such as tor hidden services or advanced network configurations
-    (https://github.com/conformal/btcd/issues/38)
-  - Add --upnp option to support automatic port mapping via UPnP
-    (https://github.com/conformal/btcd/issues/51)
-  - Update Ctrl+C interrupt handler to properly sync address manager and
-    remove the UPnP port mapping (if needed)
-  - Continue cleanup and work on implementing RPC API calls
-    - Add importprivkey (import private key) command to btcctl
-    - Update getrawtransaction to provide addresses properly, support
-      new verbose param, and match the reference implementation with the
-      exception of MULTISIG (thanks @flammit)
-    - Update getblock with new verbose flag (thanks @flammit)
-    - Add listtransactions command to btcctl
-    - Add getbalance command to btcctl
-  - Add basic support for btcd to run as a native Windows service
-    (https://github.com/conformal/btcd/issues/42)
-  - Package addblock utility with Windows MSIs
-  - Add support for TravisCI (continuous build integration)
-  - Cleanup some documentation and usage
-  - Several other minor bug fixes and general code cleanup
-
-Changes in 0.3.3 (Wed Nov 13 2013)
-  - Significantly improve initial block chain download speed
-    (https://github.com/conformal/btcd/issues/20)
-  - Add a new checkpoint at block height 267300
-  - Optimize most recently used inventory handling
-    (https://github.com/conformal/btcd/issues/21)
-  - Optimize duplicate transaction input check
-    (https://github.com/conformal/btcchain/issues/2)
-  - Optimize transaction hashing
-    (https://github.com/conformal/btcd/issues/25)
-  - Rework and optimize wallet listener notifications
-    (https://github.com/conformal/btcd/issues/22)
-  - Optimize serialization and deserialization
-    (https://github.com/conformal/btcd/issues/27)
-  - Add support for minimum transaction fee to memory pool acceptance
-    (https://github.com/conformal/btcd/issues/29)
-  - Improve leveldb database performance by removing explicit GC call
-  - Fix an issue where Ctrl+C was not always finishing orderly database
-    shutdown
-  - Fix an issue in the script handling for OP_CHECKSIG
-  - Impose max limits on all variable length protocol entries to prevent
-    abuse from malicious peers
-  - Enforce DER signatures for transactions allowed into the memory pool
-  - Separate the debug profile http server from the RPC server
-  - Rework of the RPC code to improve performance and make the code cleaner
-  - The getrawtransaction RPC call now properly checks the memory pool
-    before consulting the db (https://github.com/conformal/btcd/issues/26)
-  - Add support for the following RPC calls: getpeerinfo, getconnectedcount,
-    addnode, verifychain
-    (https://github.com/conformal/btcd/issues/13)
-    (https://github.com/conformal/btcd/issues/17)
-  - Implement rescan websocket extension to allow wallet rescans
-  - Use correct paths for application data storage for all supported
-    operating systems (https://github.com/conformal/btcd/issues/30)
-  - Add a default redirect to the http profiling page when accessing the
-    http profile server
-  - Add a new --cpuprofile option which can be used to generate CPU
-    profiling data on platforms that support it
-  - Several other minor performance optimizations
-  - Other minor bug fixes and general code cleanup
-
-Changes in 0.3.2 (Tue Oct 22 2013)
-  - Fix an issue that could cause the download of the block chain to stall
-    (https://github.com/conformal/btcd/issues/12)
-  - Remove deprecated sqlite as an available database backend
-  - Close sqlite compile issue as sqlite has now been removed
-    (https://github.com/conformal/btcd/issues/11)
-  - Change default RPC ports to 8334 (mainnet) and 18334 (testnet)
-  - Continue cleanup and work on implementing RPC API calls
-  - Add support for the following RPC calls: getrawmempool,
-    getbestblockhash, decoderawtransaction, getdifficulty,
-    getconnectioncount, getpeerinfo, and addnode
-  - Improve the btcctl utility that is used to issue JSON-RPC commands
-  - Fix an issue preventing btcd from cleanly shutting down with the RPC
-    stop command
-  - Add a number of database interface tests to ensure backends implement
-    the expected interface
-  - Expose some additional information from btcscript to be used for
-    identifying "standard"" transactions
-  - Add support for plan9 - thanks @mischief
-    (https://github.com/conformal/btcd/pull/19)
-  - Other minor bug fixes and general code cleanup
-
-Changes in 0.3.1-alpha (Tue Oct 15 2013)
-  - Change default database to leveldb
-    NOTE: This does mean you will have to redownload the block chain.  Since we
-    are still in alpha, we didn't feel writing a converter was worth the time as
-    it would take away from more important issues at this stage
-  - Add a warning if there are multiple block chain databases of different types
-  - Fix issue with unexpected EOF in leveldb -- https://github.com/conformal/btcd/issues/18
-  - Fix issue preventing block 21066 on testnet -- https://github.com/conformal/btcchain/issues/1
-  - Fix issue preventing block 96464 on testnet -- https://github.com/conformal/btcscript/issues/1
-  - Optimize transaction lookups
-  - Correct a few cases of list removal that could result in improper cleanup
-    of no longer needed orphans
-  - Add functionality to increase ulimits on non-Windows platforms
-  - Add support for mempool command which allows remote peers to query the
-    transaction memory pool via the bitcoin protocol
-  - Clean up logging a bit
-  - Add a flag to disable checkpoints for developers
-  - Add a lot of useful debug logging such as message summaries
-  - Other minor bug fixes and general code cleanup
-
-Initial Release 0.3.0-alpha (Sat Oct 05 2013):
-  - Initial release

LICENSE

@@ -1,3 +1,8 @@
+ISC License
+
+Copyright (c) 2018-2019 The kaspanet developers
+Copyright (c) 2013-2018 The btcsuite developers
+Copyright (c) 2015-2016 The Decred developers
 Copyright (c) 2013-2014 Conformal Systems LLC.
 
 Permission to use, copy, modify, and distribute this software for any
@@ -10,4 +15,4 @@ MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
 ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
 WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
 ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
-OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

README.md

@@ -1,118 +1,79 @@
-btcd
+
+Kaspad
+====
+Warning: This is pre-alpha software. There's no guarantee anything works.
 ====
 
-[![Build Status](https://travis-ci.org/conformal/btcd.png?branch=master)]
-(https://travis-ci.org/conformal/btcd)
+[![ISC License](http://img.shields.io/badge/license-ISC-blue.svg)](http://copyfree.org)
+[![GoDoc](https://img.shields.io/badge/godoc-reference-blue.svg)](http://godoc.org/github.com/kaspanet/kaspad)
 
-btcd is an alternative full node bitcoin implementation written in Go (golang).
+Kaspad is the reference full node Kaspa implementation written in Go (golang).
 
-This project is currently under active development and is in an Alpha state.
+This project is currently under active development and is in a pre-Alpha state. 
+Some things still don't work and APIs are far from finalized. The code is provided for reference only.
 
-It currently properly downloads, validates, and serves the block chain using the
-exact rules (including bugs) for block acceptance as the reference
-implementation (bitcoind).  We have taken great care to avoid btcd causing a
-fork to the block chain.  It passes all of the 'official' block acceptance tests
-(https://github.com/TheBlueMatt/test-scripts).
+## Requirements
 
-It also properly relays newly mined blocks, maintains a transaction pool,
-and relays individual transactions that have not yet made it into a block.  It
-ensures all individual transactions admitted to the pool follow the rules
-required into the block chain and also includes the vast majority of the more
-strict checks which filter transactions based on miner requirements ("standard"
-transactions).
-
-One key difference between btcd and bitcoind is that btcd does *NOT* include
-wallet functionality and this was a very intentional design decision. See the
-blog entry [here](https://blog.conformal.com/btcd-not-your-moms-bitcoin-daemon)
-for more details.  This means you can't actually make or receive payments
-directly with btcd.  That functionality is provided by the
-[btcwallet](https://github.com/conformal/btcwallet) and
-[btcgui](https://github.com/conformal/btcgui) projects which are both under
-active development.
+Latest version of [Go](http://golang.org) (currently 1.13).
 
 ## Installation
 
-#### Windows - MSI Available
-
-https://github.com/conformal/btcd/releases
-
-#### Linux/BSD/POSIX - Build from Source
+#### Build from Source
 
 - Install Go according to the installation instructions here:
   http://golang.org/doc/install
-  **NOTE: btcd requires features only available in Go version 1.2 or later.**
 
-- Run the following command to obtain btcd, all dependencies, and install it:
-  ```$ go get github.com/conformal/btcd```
+- Ensure Go was installed properly and is a supported version:
 
-- btcd will now be installed in either ```$GOROOT/bin``` or ```$GOPATH/bin```
-  depending on your configuration.  If you did not already add to your system
-  path during the installation, we recommend you do so now.
+```bash
+$ go version
+$ go env GOROOT GOPATH
+```
 
-## Updating
+NOTE: The `GOROOT` and `GOPATH` above must not be the same path. It is
+recommended that `GOPATH` is set to a directory in your home directory such as
+`~/dev/go` to avoid write permission issues. It is also recommended to add
+`$GOPATH/bin` to your `PATH` at this point.
 
-#### Windows
+- Run the following commands to obtain and install kaspad including all dependencies:
 
-Install a newer MSI
+```bash
+$ git clone https://github.com/kaspanet/kaspad $GOPATH/src/github.com/kaspanet/kaspad
+$ cd $GOPATH/src/github.com/kaspanet/kaspad
+$ ./test.sh
+$ go install . ./cmd/...
+```
+`./test.sh` tests can be skipped, but some things might not run correctly on your system if tests fail.
 
-#### Linux/BSD/POSIX - Build from Source
+- Kaspad (and utilities) should now be installed in `$GOPATH/bin`. If you did
+  not already add the bin directory to your system path during Go installation,
+  you are encouraged to do so now.
 
-**NOTE: btcd requires features only available in Go version 1.2 or later.**
-
-- Run the following command to update btcd, all dependencies, and install it:
-  ```$ go get -u -v github.com/conformal/btcd/...```
 
 ## Getting Started
 
-btcd has several configuration options avilable to tweak how it runs, but all
-of the basic operations described in the intro section work with zero
-configuration.
-
-#### Windows (Installed from MSI)
-
-Launch btcd from your Start menu.
+Kaspad has several configuration options available to tweak how it runs, but all
+of the basic operations work with zero configuration.
 
 #### Linux/BSD/POSIX/Source
 
 ```bash
-$ ./btcd
-````
-## Mailing lists
+$ ./kaspad
+```
 
-- btcd: discussion of btcd and its packages.
-- btcd-commits: readonly mail-out of source code changes.
-
-To subscribe to a given list, send email to list+subscribe@opensource.conformal.com
+## Discord
+Join our discord server using the following link: https://discord.gg/WmGhhzk
 
 ## Issue Tracker
 
-The [integrated github issue tracker](https://github.com/conformal/btcd/issues)
+The [integrated github issue tracker](https://github.com/kaspanet/kaspad/issues)
 is used for this project.
 
 ## Documentation
 
-The documentation is a work-in-progress.  It uses the [github wiki](https://github.com/conformal/btcd/wiki) facility.
-
-## GPG Verification Key
-
-All official release tags are signed by Conformal so users can ensure the code
-has not been tampered with and is coming from Conformal.  To verify the
-signature perform the following:
-
-- Download the public key from the Conformal website at
-  https://opensource.conformal.com/GIT-GPG-KEY-conformal.txt
-
-- Import the public key into your GPG keyring:
-  ```bash
-  gpg --import GIT-GPG-KEY-conformal.txt
-  ```
-
-- Verify the release tag with the following command where `TAG_NAME` is a
-  placeholder for the specific tag:
-  ```bash
-  git tag -v TAG_NAME
-  ```
+The documentation is a work-in-progress. It is located in the [docs](https://github.com/kaspanet/kaspad/tree/master/docs) folder.
 
 ## License
 
-btcd is licensed under the [copyfree](http://copyfree.org) ISC License.
+Kaspad is licensed under the [copyfree](http://copyfree.org) ISC License.
+

addrmanager_test.go

@@ -1,281 +0,0 @@
-// Copyright (c) 2013-2014 Conformal Systems LLC.
-// Use of this source code is governed by an ISC
-// license that can be found in the LICENSE file.
-
-package main
-
-import (
-	"github.com/conformal/btcwire"
-	"net"
-	"testing"
-	"time"
-)
-
-// naTest is used to describe a test to be perfomed against the NetAddressKey
-// method.
-type naTest struct {
-	in   btcwire.NetAddress
-	want string
-}
-
-type ipTest struct {
-	in       btcwire.NetAddress
-	rfc1918  bool
-	rfc3849  bool
-	rfc3927  bool
-	rfc3964  bool
-	rfc4193  bool
-	rfc4380  bool
-	rfc4843  bool
-	rfc4862  bool
-	rfc6052  bool
-	rfc6145  bool
-	local    bool
-	valid    bool
-	routable bool
-}
-
-// naTests houses all of the tests to be performed against the NetAddressKey
-// method.
-var naTests = make([]naTest, 0)
-var ipTests = make([]ipTest, 0)
-
-func addIpTest(ip string, rfc1918, rfc3849, rfc3927, rfc3964, rfc4193, rfc4380,
-	rfc4843, rfc4862, rfc6052, rfc6145, local, valid, routable bool) {
-	nip := net.ParseIP(ip)
-	na := btcwire.NetAddress{
-		Timestamp: time.Now(),
-		Services:  btcwire.SFNodeNetwork,
-		IP:        nip,
-		Port:      8333,
-	}
-	test := ipTest{na, rfc1918, rfc3849, rfc3927, rfc3964, rfc4193, rfc4380,
-		rfc4843, rfc4862, rfc6052, rfc6145, local, valid, routable}
-	ipTests = append(ipTests, test)
-}
-
-func addIpTests() {
-	addIpTest("10.255.255.255", true, false, false, false, false, false,
-		false, false, false, false, false, true, false)
-	addIpTest("192.168.0.1", true, false, false, false, false, false,
-		false, false, false, false, false, true, false)
-	addIpTest("172.31.255.1", true, false, false, false, false, false,
-		false, false, false, false, false, true, false)
-	addIpTest("172.32.1.1", false, false, false, false, false, false,
-		false, false, false, false, false, true, true)
-	addIpTest("169.254.250.120", false, false, true, false, false, false,
-		false, false, false, false, false, true, false)
-	addIpTest("0.0.0.0", false, false, false, false, false, false,
-		false, false, false, false, true, false, false)
-	addIpTest("255.255.255.255", false, false, false, false, false, false,
-		false, false, false, false, false, false, false)
-	addIpTest("127.0.0.1", false, false, false, false, false, false,
-		false, false, false, false, true, true, false)
-	addIpTest("fd00:dead::1", false, false, false, false, true, false,
-		false, false, false, false, false, true, false)
-	addIpTest("2001::1", false, false, false, false, false, true,
-		false, false, false, false, false, true, true)
-	addIpTest("2001:10:abcd::1:1", false, false, false, false, false, false,
-		true, false, false, false, false, true, false)
-	addIpTest("fe80::1", false, false, false, false, false, false,
-		false, true, false, false, false, true, false)
-	addIpTest("fe80:1::1", false, false, false, false, false, false,
-		false, false, false, false, false, true, true)
-	addIpTest("64:ff9b::1", false, false, false, false, false, false,
-		false, false, true, false, false, true, true)
-	addIpTest("::ffff:abcd:ef12:1", false, false, false, false, false, false,
-		false, false, false, false, false, true, true)
-	addIpTest("::1", false, false, false, false, false, false,
-		false, false, false, false, true, true, false)
-}
-
-func addNaTest(ip string, port uint16, want string) {
-	nip := net.ParseIP(ip)
-	na := btcwire.NetAddress{
-		Timestamp: time.Now(),
-		Services:  btcwire.SFNodeNetwork,
-		IP:        nip,
-		Port:      port,
-	}
-	test := naTest{na, want}
-	naTests = append(naTests, test)
-}
-
-// addNaTests
-func addNaTests() {
-	// IPv4
-	// Localhost
-	addNaTest("127.0.0.1", 8333, "127.0.0.1:8333")
-	addNaTest("127.0.0.1", 8334, "127.0.0.1:8334")
-
-	// Class A
-	addNaTest("1.0.0.1", 8333, "1.0.0.1:8333")
-	addNaTest("2.2.2.2", 8334, "2.2.2.2:8334")
-	addNaTest("27.253.252.251", 8335, "27.253.252.251:8335")
-	addNaTest("123.3.2.1", 8336, "123.3.2.1:8336")
-
-	// Private Class A
-	addNaTest("10.0.0.1", 8333, "10.0.0.1:8333")
-	addNaTest("10.1.1.1", 8334, "10.1.1.1:8334")
-	addNaTest("10.2.2.2", 8335, "10.2.2.2:8335")
-	addNaTest("10.10.10.10", 8336, "10.10.10.10:8336")
-
-	// Class B
-	addNaTest("128.0.0.1", 8333, "128.0.0.1:8333")
-	addNaTest("129.1.1.1", 8334, "129.1.1.1:8334")
-	addNaTest("180.2.2.2", 8335, "180.2.2.2:8335")
-	addNaTest("191.10.10.10", 8336, "191.10.10.10:8336")
-
-	// Private Class B
-	addNaTest("172.16.0.1", 8333, "172.16.0.1:8333")
-	addNaTest("172.16.1.1", 8334, "172.16.1.1:8334")
-	addNaTest("172.16.2.2", 8335, "172.16.2.2:8335")
-	addNaTest("172.16.172.172", 8336, "172.16.172.172:8336")
-
-	// Class C
-	addNaTest("193.0.0.1", 8333, "193.0.0.1:8333")
-	addNaTest("200.1.1.1", 8334, "200.1.1.1:8334")
-	addNaTest("205.2.2.2", 8335, "205.2.2.2:8335")
-	addNaTest("223.10.10.10", 8336, "223.10.10.10:8336")
-
-	// Private Class C
-	addNaTest("192.168.0.1", 8333, "192.168.0.1:8333")
-	addNaTest("192.168.1.1", 8334, "192.168.1.1:8334")
-	addNaTest("192.168.2.2", 8335, "192.168.2.2:8335")
-	addNaTest("192.168.192.192", 8336, "192.168.192.192:8336")
-
-	// IPv6
-	// Localhost
-	addNaTest("::1", 8333, "[::1]:8333")
-	addNaTest("fe80::1", 8334, "[fe80::1]:8334")
-
-	// Link-local
-	addNaTest("fe80::1:1", 8333, "[fe80::1:1]:8333")
-	addNaTest("fe91::2:2", 8334, "[fe91::2:2]:8334")
-	addNaTest("fea2::3:3", 8335, "[fea2::3:3]:8335")
-	addNaTest("feb3::4:4", 8336, "[feb3::4:4]:8336")
-
-	// Site-local
-	addNaTest("fec0::1:1", 8333, "[fec0::1:1]:8333")
-	addNaTest("fed1::2:2", 8334, "[fed1::2:2]:8334")
-	addNaTest("fee2::3:3", 8335, "[fee2::3:3]:8335")
-	addNaTest("fef3::4:4", 8336, "[fef3::4:4]:8336")
-}
-
-func TestGetAddress(t *testing.T) {
-	n := NewAddrManager()
-	if rv := n.GetAddress("any", 10); rv != nil {
-		t.Errorf("GetAddress failed: got: %v want: %v\n", rv, nil)
-	}
-}
-
-func TestIpTypes(t *testing.T) {
-	addIpTests()
-
-	t.Logf("Running %d tests", len(ipTests))
-	for _, test := range ipTests {
-		rv := RFC1918(&test.in)
-		if rv != test.rfc1918 {
-			t.Errorf("RFC1918 %s\n got: %v want: %v", test.in.IP, rv, test.rfc1918)
-			continue
-		}
-	}
-	for _, test := range ipTests {
-		rv := RFC3849(&test.in)
-		if rv != test.rfc3849 {
-			t.Errorf("RFC3849 %s\n got: %v want: %v", test.in.IP, rv, test.rfc3849)
-			continue
-		}
-	}
-	for _, test := range ipTests {
-		rv := RFC3927(&test.in)
-		if rv != test.rfc3927 {
-			t.Errorf("RFC3927 %s\n got: %v want: %v", test.in.IP, rv, test.rfc3927)
-			continue
-		}
-	}
-	for _, test := range ipTests {
-		rv := RFC3964(&test.in)
-		if rv != test.rfc3964 {
-			t.Errorf("RFC3964 %s\n got: %v want: %v", test.in.IP, rv, test.rfc3964)
-			continue
-		}
-	}
-	for _, test := range ipTests {
-		rv := RFC4193(&test.in)
-		if rv != test.rfc4193 {
-			t.Errorf("RFC4193 %s\n got: %v want: %v", test.in.IP, rv, test.rfc4193)
-			continue
-		}
-	}
-	for _, test := range ipTests {
-		rv := RFC4380(&test.in)
-		if rv != test.rfc4380 {
-			t.Errorf("RFC4380 %s\n got: %v want: %v", test.in.IP, rv, test.rfc4380)
-			continue
-		}
-	}
-	for _, test := range ipTests {
-		rv := RFC4843(&test.in)
-		if rv != test.rfc4843 {
-			t.Errorf("RFC4843 %s\n got: %v want: %v", test.in.IP, rv, test.rfc4843)
-			continue
-		}
-	}
-	for _, test := range ipTests {
-		rv := RFC4862(&test.in)
-		if rv != test.rfc4862 {
-			t.Errorf("RFC4862 %s\n got: %v want: %v", test.in.IP, rv, test.rfc4862)
-			continue
-		}
-	}
-	for _, test := range ipTests {
-		rv := RFC6052(&test.in)
-		if rv != test.rfc6052 {
-			t.Errorf("RFC6052 %s\n got: %v want: %v", test.in.IP, rv, test.rfc6052)
-			continue
-		}
-	}
-	for _, test := range ipTests {
-		rv := RFC6145(&test.in)
-		if rv != test.rfc6145 {
-			t.Errorf("RFC1918 %s\n got: %v want: %v", test.in.IP, rv, test.rfc6145)
-			continue
-		}
-	}
-	for _, test := range ipTests {
-		rv := Local(&test.in)
-		if rv != test.local {
-			t.Errorf("Local %s\n got: %v want: %v", test.in.IP, rv, test.local)
-			continue
-		}
-	}
-	for _, test := range ipTests {
-		rv := Valid(&test.in)
-		if rv != test.valid {
-			t.Errorf("Valid %s\n got: %v want: %v", test.in.IP, rv, test.valid)
-			continue
-		}
-	}
-	for _, test := range ipTests {
-		rv := Routable(&test.in)
-		if rv != test.routable {
-			t.Errorf("Routable %s\n got: %v want: %v", test.in.IP, rv, test.routable)
-			continue
-		}
-	}
-}
-
-func TestNetAddressKey(t *testing.T) {
-	addNaTests()
-
-	t.Logf("Running %d tests", len(naTests))
-	for i, test := range naTests {
-		key := NetAddressKey(&test.in)
-		if key != test.want {
-			t.Errorf("NetAddressKey #%d\n got: %s want: %s", i, key, test.want)
-			continue
-		}
-	}
-
-}

addrmgr/addrmanager_test.go

@@ -0,0 +1,123 @@
+// Copyright (c) 2013-2014 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package addrmgr
+
+import (
+	"net"
+	"testing"
+
+	"github.com/pkg/errors"
+
+	"github.com/kaspanet/kaspad/wire"
+)
+
+// naTest is used to describe a test to be performed against the NetAddressKey
+// method.
+type naTest struct {
+	in   wire.NetAddress
+	want string
+}
+
+// naTests houses all of the tests to be performed against the NetAddressKey
+// method.
+var naTests = make([]naTest, 0)
+
+// Put some IP in here for convenience. Points to google.
+var someIP = "173.194.115.66"
+
+// addNaTests
+func addNaTests() {
+	// IPv4
+	// Localhost
+	addNaTest("127.0.0.1", 16111, "127.0.0.1:16111")
+	addNaTest("127.0.0.1", 16110, "127.0.0.1:16110")
+
+	// Class A
+	addNaTest("1.0.0.1", 16111, "1.0.0.1:16111")
+	addNaTest("2.2.2.2", 16110, "2.2.2.2:16110")
+	addNaTest("27.253.252.251", 8335, "27.253.252.251:8335")
+	addNaTest("123.3.2.1", 8336, "123.3.2.1:8336")
+
+	// Private Class A
+	addNaTest("10.0.0.1", 16111, "10.0.0.1:16111")
+	addNaTest("10.1.1.1", 16110, "10.1.1.1:16110")
+	addNaTest("10.2.2.2", 8335, "10.2.2.2:8335")
+	addNaTest("10.10.10.10", 8336, "10.10.10.10:8336")
+
+	// Class B
+	addNaTest("128.0.0.1", 16111, "128.0.0.1:16111")
+	addNaTest("129.1.1.1", 16110, "129.1.1.1:16110")
+	addNaTest("180.2.2.2", 8335, "180.2.2.2:8335")
+	addNaTest("191.10.10.10", 8336, "191.10.10.10:8336")
+
+	// Private Class B
+	addNaTest("172.16.0.1", 16111, "172.16.0.1:16111")
+	addNaTest("172.16.1.1", 16110, "172.16.1.1:16110")
+	addNaTest("172.16.2.2", 8335, "172.16.2.2:8335")
+	addNaTest("172.16.172.172", 8336, "172.16.172.172:8336")
+
+	// Class C
+	addNaTest("193.0.0.1", 16111, "193.0.0.1:16111")
+	addNaTest("200.1.1.1", 16110, "200.1.1.1:16110")
+	addNaTest("205.2.2.2", 8335, "205.2.2.2:8335")
+	addNaTest("223.10.10.10", 8336, "223.10.10.10:8336")
+
+	// Private Class C
+	addNaTest("192.168.0.1", 16111, "192.168.0.1:16111")
+	addNaTest("192.168.1.1", 16110, "192.168.1.1:16110")
+	addNaTest("192.168.2.2", 8335, "192.168.2.2:8335")
+	addNaTest("192.168.192.192", 8336, "192.168.192.192:8336")
+
+	// IPv6
+	// Localhost
+	addNaTest("::1", 16111, "[::1]:16111")
+	addNaTest("fe80::1", 16110, "[fe80::1]:16110")
+
+	// Link-local
+	addNaTest("fe80::1:1", 16111, "[fe80::1:1]:16111")
+	addNaTest("fe91::2:2", 16110, "[fe91::2:2]:16110")
+	addNaTest("fea2::3:3", 8335, "[fea2::3:3]:8335")
+	addNaTest("feb3::4:4", 8336, "[feb3::4:4]:8336")
+
+	// Site-local
+	addNaTest("fec0::1:1", 16111, "[fec0::1:1]:16111")
+	addNaTest("fed1::2:2", 16110, "[fed1::2:2]:16110")
+	addNaTest("fee2::3:3", 8335, "[fee2::3:3]:8335")
+	addNaTest("fef3::4:4", 8336, "[fef3::4:4]:8336")
+}
+
+func addNaTest(ip string, port uint16, want string) {
+	nip := net.ParseIP(ip)
+	na := *wire.NewNetAddressIPPort(nip, port, wire.SFNodeNetwork)
+	test := naTest{na, want}
+	naTests = append(naTests, test)
+}
+
+func lookupFunc(host string) ([]net.IP, error) {
+	return nil, errors.New("not implemented")
+}
+
+func TestStartStop(t *testing.T) {
+	n := New("teststartstop", lookupFunc, nil)
+	n.Start()
+	err := n.Stop()
+	if err != nil {
+		t.Fatalf("Address Manager failed to stop: %v", err)
+	}
+}
+
+func TestNetAddressKey(t *testing.T) {
+	addNaTests()
+
+	t.Logf("Running %d tests", len(naTests))
+	for i, test := range naTests {
+		key := NetAddressKey(&test.in)
+		if key != test.want {
+			t.Errorf("NetAddressKey #%d\n got: %s want: %s", i, key, test.want)
+			continue
+		}
+	}
+
+}

addrmgr/doc.go

@@ -0,0 +1,34 @@
+/*
+Package addrmgr implements concurrency safe Kaspa address manager.
+
+Address Manager Overview
+
+In order maintain the peer-to-peer Kaspa network, there needs to be a source
+of addresses to connect to as nodes come and go. The Kaspa protocol provides
+the getaddr and addr messages to allow peers to communicate known addresses with
+each other. However, there needs to a mechanism to store those results and
+select peers from them. It is also important to note that remote peers can't
+be trusted to send valid peers nor attempt to provide you with only peers they
+control with malicious intent.
+
+With that in mind, this package provides a concurrency safe address manager for
+caching and selecting peers in a non-deterministic manner. The general idea is
+the caller adds addresses to the address manager and notifies it when addresses
+are connected, known good, and attempted. The caller also requests addresses as
+it needs them.
+
+The address manager internally segregates the addresses into groups and
+non-deterministically selects groups in a cryptographically random manner. This
+reduce the chances multiple addresses from the same nets are selected which
+generally helps provide greater peer diversity, and perhaps more importantly,
+drastically reduces the chances an attacker is able to coerce your peer into
+only connecting to nodes they control.
+
+The address manager also understands routability and tries hard to only return
+routable addresses. In addition, it uses the information provided by the caller
+about connected, known good, and attempted addresses to periodically purge
+peers which no longer appear to be good peers as well as bias the selection
+toward known good peers. The general idea is to make a best effort at only
+providing usable addresses.
+*/
+package addrmgr

addrmgr/internal_test.go

@@ -0,0 +1,25 @@
+// Copyright (c) 2013-2015 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package addrmgr
+
+import (
+	"time"
+
+	"github.com/kaspanet/kaspad/wire"
+)
+
+func TstKnownAddressIsBad(ka *KnownAddress) bool {
+	return ka.isBad()
+}
+
+func TstKnownAddressChance(ka *KnownAddress) float64 {
+	return ka.chance()
+}
+
+func TstNewKnownAddress(na *wire.NetAddress, attempts int,
+	lastattempt, lastsuccess time.Time, tried bool, refs int) *KnownAddress {
+	return &KnownAddress{na: na, attempts: attempts, lastattempt: lastattempt,
+		lastsuccess: lastsuccess, tried: tried, refs: refs}
+}

addrmgr/knownaddress.go

@@ -0,0 +1,105 @@
+// Copyright (c) 2013-2014 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package addrmgr
+
+import (
+	"time"
+
+	"github.com/kaspanet/kaspad/util/subnetworkid"
+
+	"github.com/kaspanet/kaspad/wire"
+)
+
+// KnownAddress tracks information about a known network address that is used
+// to determine how viable an address is.
+type KnownAddress struct {
+	na           *wire.NetAddress
+	srcAddr      *wire.NetAddress
+	attempts     int
+	lastattempt  time.Time
+	lastsuccess  time.Time
+	tried        bool
+	refs         int // reference count of new buckets
+	subnetworkID *subnetworkid.SubnetworkID
+}
+
+// NetAddress returns the underlying wire.NetAddress associated with the
+// known address.
+func (ka *KnownAddress) NetAddress() *wire.NetAddress {
+	return ka.na
+}
+
+// SubnetworkID returns the subnetwork ID of the known address.
+func (ka *KnownAddress) SubnetworkID() *subnetworkid.SubnetworkID {
+	return ka.subnetworkID
+}
+
+// LastAttempt returns the last time the known address was attempted.
+func (ka *KnownAddress) LastAttempt() time.Time {
+	return ka.lastattempt
+}
+
+// chance returns the selection probability for a known address. The priority
+// depends upon how recently the address has been seen, how recently it was last
+// attempted and how often attempts to connect to it have failed.
+func (ka *KnownAddress) chance() float64 {
+	now := time.Now()
+	lastAttempt := now.Sub(ka.lastattempt)
+
+	if lastAttempt < 0 {
+		lastAttempt = 0
+	}
+
+	c := 1.0
+
+	// Very recent attempts are less likely to be retried.
+	if lastAttempt < 10*time.Minute {
+		c *= 0.01
+	}
+
+	// Failed attempts deprioritise.
+	for i := ka.attempts; i > 0; i-- {
+		c /= 1.5
+	}
+
+	return c
+}
+
+// isBad returns true if the address in question has not been tried in the last
+// minute and meets one of the following criteria:
+// 1) It claims to be from the future
+// 2) It hasn't been seen in over a month
+// 3) It has failed at least three times and never succeeded
+// 4) It has failed ten times in the last week
+// All addresses that meet these criteria are assumed to be worthless and not
+// worth keeping hold of.
+func (ka *KnownAddress) isBad() bool {
+	if ka.lastattempt.After(time.Now().Add(-1 * time.Minute)) {
+		return false
+	}
+
+	// From the future?
+	if ka.na.Timestamp.After(time.Now().Add(10 * time.Minute)) {
+		return true
+	}
+
+	// Over a month old?
+	if ka.na.Timestamp.Before(time.Now().Add(-1 * numMissingDays * time.Hour * 24)) {
+		return true
+	}
+
+	// Never succeeded?
+	if ka.lastsuccess.IsZero() && ka.attempts >= numRetries {
+		return true
+	}
+
+	// Hasn't succeeded in too long?
+	if !ka.lastsuccess.After(time.Now().Add(-1*minBadDays*time.Hour*24)) &&
+		ka.attempts >= maxFailures {
+		return true
+	}
+
+	return false
+}

addrmgr/knownaddress_test.go

@@ -0,0 +1,114 @@
+// Copyright (c) 2013-2015 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package addrmgr_test
+
+import (
+	"math"
+	"testing"
+	"time"
+
+	"github.com/kaspanet/kaspad/addrmgr"
+	"github.com/kaspanet/kaspad/wire"
+)
+
+func TestChance(t *testing.T) {
+	now := time.Unix(time.Now().Unix(), 0)
+	var tests = []struct {
+		addr     *addrmgr.KnownAddress
+		expected float64
+	}{
+		{
+			//Test normal case
+			addrmgr.TstNewKnownAddress(&wire.NetAddress{Timestamp: now.Add(-35 * time.Second)},
+				0, time.Now().Add(-30*time.Minute), time.Now(), false, 0),
+			1.0,
+		}, {
+			//Test case in which lastseen < 0
+			addrmgr.TstNewKnownAddress(&wire.NetAddress{Timestamp: now.Add(20 * time.Second)},
+				0, time.Now().Add(-30*time.Minute), time.Now(), false, 0),
+			1.0,
+		}, {
+			//Test case in which lastattempt < 0
+			addrmgr.TstNewKnownAddress(&wire.NetAddress{Timestamp: now.Add(-35 * time.Second)},
+				0, time.Now().Add(30*time.Minute), time.Now(), false, 0),
+			1.0 * .01,
+		}, {
+			//Test case in which lastattempt < ten minutes
+			addrmgr.TstNewKnownAddress(&wire.NetAddress{Timestamp: now.Add(-35 * time.Second)},
+				0, time.Now().Add(-5*time.Minute), time.Now(), false, 0),
+			1.0 * .01,
+		}, {
+			//Test case with several failed attempts.
+			addrmgr.TstNewKnownAddress(&wire.NetAddress{Timestamp: now.Add(-35 * time.Second)},
+				2, time.Now().Add(-30*time.Minute), time.Now(), false, 0),
+			1 / 1.5 / 1.5,
+		},
+	}
+
+	err := .0001
+	for i, test := range tests {
+		chance := addrmgr.TstKnownAddressChance(test.addr)
+		if math.Abs(test.expected-chance) >= err {
+			t.Errorf("case %d: got %f, expected %f", i, chance, test.expected)
+		}
+	}
+}
+
+func TestIsBad(t *testing.T) {
+	now := time.Unix(time.Now().Unix(), 0)
+	future := now.Add(35 * time.Minute)
+	monthOld := now.Add(-43 * time.Hour * 24)
+	secondsOld := now.Add(-2 * time.Second)
+	minutesOld := now.Add(-27 * time.Minute)
+	hoursOld := now.Add(-5 * time.Hour)
+	zeroTime := time.Time{}
+
+	futureNa := &wire.NetAddress{Timestamp: future}
+	minutesOldNa := &wire.NetAddress{Timestamp: minutesOld}
+	monthOldNa := &wire.NetAddress{Timestamp: monthOld}
+	currentNa := &wire.NetAddress{Timestamp: secondsOld}
+
+	//Test addresses that have been tried in the last minute.
+	if addrmgr.TstKnownAddressIsBad(addrmgr.TstNewKnownAddress(futureNa, 3, secondsOld, zeroTime, false, 0)) {
+		t.Errorf("test case 1: addresses that have been tried in the last minute are not bad.")
+	}
+	if addrmgr.TstKnownAddressIsBad(addrmgr.TstNewKnownAddress(monthOldNa, 3, secondsOld, zeroTime, false, 0)) {
+		t.Errorf("test case 2: addresses that have been tried in the last minute are not bad.")
+	}
+	if addrmgr.TstKnownAddressIsBad(addrmgr.TstNewKnownAddress(currentNa, 3, secondsOld, zeroTime, false, 0)) {
+		t.Errorf("test case 3: addresses that have been tried in the last minute are not bad.")
+	}
+	if addrmgr.TstKnownAddressIsBad(addrmgr.TstNewKnownAddress(currentNa, 3, secondsOld, monthOld, true, 0)) {
+		t.Errorf("test case 4: addresses that have been tried in the last minute are not bad.")
+	}
+	if addrmgr.TstKnownAddressIsBad(addrmgr.TstNewKnownAddress(currentNa, 2, secondsOld, secondsOld, true, 0)) {
+		t.Errorf("test case 5: addresses that have been tried in the last minute are not bad.")
+	}
+
+	//Test address that claims to be from the future.
+	if !addrmgr.TstKnownAddressIsBad(addrmgr.TstNewKnownAddress(futureNa, 0, minutesOld, hoursOld, true, 0)) {
+		t.Errorf("test case 6: addresses that claim to be from the future are bad.")
+	}
+
+	//Test address that has not been seen in over a month.
+	if !addrmgr.TstKnownAddressIsBad(addrmgr.TstNewKnownAddress(monthOldNa, 0, minutesOld, hoursOld, true, 0)) {
+		t.Errorf("test case 7: addresses more than a month old are bad.")
+	}
+
+	//It has failed at least three times and never succeeded.
+	if !addrmgr.TstKnownAddressIsBad(addrmgr.TstNewKnownAddress(minutesOldNa, 3, minutesOld, zeroTime, true, 0)) {
+		t.Errorf("test case 8: addresses that have never succeeded are bad.")
+	}
+
+	//It has failed ten times in the last week
+	if !addrmgr.TstKnownAddressIsBad(addrmgr.TstNewKnownAddress(minutesOldNa, 10, minutesOld, monthOld, true, 0)) {
+		t.Errorf("test case 9: addresses that have not succeeded in too long are bad.")
+	}
+
+	//Test an address that should work.
+	if addrmgr.TstKnownAddressIsBad(addrmgr.TstNewKnownAddress(minutesOldNa, 2, minutesOld, hoursOld, true, 0)) {
+		t.Errorf("test case 10: This should be a valid address.")
+	}
+}

addrmgr/log.go

@@ -0,0 +1,13 @@
+// Copyright (c) 2013-2014 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package addrmgr
+
+import (
+	"github.com/kaspanet/kaspad/logger"
+	"github.com/kaspanet/kaspad/util/panics"
+)
+
+var log, _ = logger.Get(logger.SubsystemTags.ADXR)
+var spawn = panics.GoroutineWrapperFuncWithPanicHandler(log)

addrmgr/network.go

@@ -0,0 +1,260 @@
+// Copyright (c) 2013-2014 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package addrmgr
+
+import (
+	"net"
+
+	"github.com/kaspanet/kaspad/config"
+
+	"github.com/kaspanet/kaspad/wire"
+)
+
+var (
+	// rfc1918Nets specifies the IPv4 private address blocks as defined by
+	// by RFC1918 (10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16).
+	rfc1918Nets = []net.IPNet{
+		ipNet("10.0.0.0", 8, 32),
+		ipNet("172.16.0.0", 12, 32),
+		ipNet("192.168.0.0", 16, 32),
+	}
+
+	// rfc2544Net specifies the the IPv4 block as defined by RFC2544
+	// (198.18.0.0/15)
+	rfc2544Net = ipNet("198.18.0.0", 15, 32)
+
+	// rfc3849Net specifies the IPv6 documentation address block as defined
+	// by RFC3849 (2001:DB8::/32).
+	rfc3849Net = ipNet("2001:DB8::", 32, 128)
+
+	// rfc3927Net specifies the IPv4 auto configuration address block as
+	// defined by RFC3927 (169.254.0.0/16).
+	rfc3927Net = ipNet("169.254.0.0", 16, 32)
+
+	// rfc3964Net specifies the IPv6 to IPv4 encapsulation address block as
+	// defined by RFC3964 (2002::/16).
+	rfc3964Net = ipNet("2002::", 16, 128)
+
+	// rfc4193Net specifies the IPv6 unique local address block as defined
+	// by RFC4193 (FC00::/7).
+	rfc4193Net = ipNet("FC00::", 7, 128)
+
+	// rfc4380Net specifies the IPv6 teredo tunneling over UDP address block
+	// as defined by RFC4380 (2001::/32).
+	rfc4380Net = ipNet("2001::", 32, 128)
+
+	// rfc4843Net specifies the IPv6 ORCHID address block as defined by
+	// RFC4843 (2001:10::/28).
+	rfc4843Net = ipNet("2001:10::", 28, 128)
+
+	// rfc4862Net specifies the IPv6 stateless address autoconfiguration
+	// address block as defined by RFC4862 (FE80::/64).
+	rfc4862Net = ipNet("FE80::", 64, 128)
+
+	// rfc5737Net specifies the IPv4 documentation address blocks as defined
+	// by RFC5737 (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24)
+	rfc5737Net = []net.IPNet{
+		ipNet("192.0.2.0", 24, 32),
+		ipNet("198.51.100.0", 24, 32),
+		ipNet("203.0.113.0", 24, 32),
+	}
+
+	// rfc6052Net specifies the IPv6 well-known prefix address block as
+	// defined by RFC6052 (64:FF9B::/96).
+	rfc6052Net = ipNet("64:FF9B::", 96, 128)
+
+	// rfc6145Net specifies the IPv6 to IPv4 translated address range as
+	// defined by RFC6145 (::FFFF:0:0:0/96).
+	rfc6145Net = ipNet("::FFFF:0:0:0", 96, 128)
+
+	// rfc6598Net specifies the IPv4 block as defined by RFC6598 (100.64.0.0/10)
+	rfc6598Net = ipNet("100.64.0.0", 10, 32)
+
+	// zero4Net defines the IPv4 address block for address staring with 0
+	// (0.0.0.0/8).
+	zero4Net = ipNet("0.0.0.0", 8, 32)
+
+	// heNet defines the Hurricane Electric IPv6 address block.
+	heNet = ipNet("2001:470::", 32, 128)
+)
+
+// ipNet returns a net.IPNet struct given the passed IP address string, number
+// of one bits to include at the start of the mask, and the total number of bits
+// for the mask.
+func ipNet(ip string, ones, bits int) net.IPNet {
+	return net.IPNet{IP: net.ParseIP(ip), Mask: net.CIDRMask(ones, bits)}
+}
+
+// IsIPv4 returns whether or not the given address is an IPv4 address.
+func IsIPv4(na *wire.NetAddress) bool {
+	return na.IP.To4() != nil
+}
+
+// IsLocal returns whether or not the given address is a local address.
+func IsLocal(na *wire.NetAddress) bool {
+	return na.IP.IsLoopback() || zero4Net.Contains(na.IP)
+}
+
+// IsRFC1918 returns whether or not the passed address is part of the IPv4
+// private network address space as defined by RFC1918 (10.0.0.0/8,
+// 172.16.0.0/12, or 192.168.0.0/16).
+func IsRFC1918(na *wire.NetAddress) bool {
+	for _, rfc := range rfc1918Nets {
+		if rfc.Contains(na.IP) {
+			return true
+		}
+	}
+	return false
+}
+
+// IsRFC2544 returns whether or not the passed address is part of the IPv4
+// address space as defined by RFC2544 (198.18.0.0/15)
+func IsRFC2544(na *wire.NetAddress) bool {
+	return rfc2544Net.Contains(na.IP)
+}
+
+// IsRFC3849 returns whether or not the passed address is part of the IPv6
+// documentation range as defined by RFC3849 (2001:DB8::/32).
+func IsRFC3849(na *wire.NetAddress) bool {
+	return rfc3849Net.Contains(na.IP)
+}
+
+// IsRFC3927 returns whether or not the passed address is part of the IPv4
+// autoconfiguration range as defined by RFC3927 (169.254.0.0/16).
+func IsRFC3927(na *wire.NetAddress) bool {
+	return rfc3927Net.Contains(na.IP)
+}
+
+// IsRFC3964 returns whether or not the passed address is part of the IPv6 to
+// IPv4 encapsulation range as defined by RFC3964 (2002::/16).
+func IsRFC3964(na *wire.NetAddress) bool {
+	return rfc3964Net.Contains(na.IP)
+}
+
+// IsRFC4193 returns whether or not the passed address is part of the IPv6
+// unique local range as defined by RFC4193 (FC00::/7).
+func IsRFC4193(na *wire.NetAddress) bool {
+	return rfc4193Net.Contains(na.IP)
+}
+
+// IsRFC4380 returns whether or not the passed address is part of the IPv6
+// teredo tunneling over UDP range as defined by RFC4380 (2001::/32).
+func IsRFC4380(na *wire.NetAddress) bool {
+	return rfc4380Net.Contains(na.IP)
+}
+
+// IsRFC4843 returns whether or not the passed address is part of the IPv6
+// ORCHID range as defined by RFC4843 (2001:10::/28).
+func IsRFC4843(na *wire.NetAddress) bool {
+	return rfc4843Net.Contains(na.IP)
+}
+
+// IsRFC4862 returns whether or not the passed address is part of the IPv6
+// stateless address autoconfiguration range as defined by RFC4862 (FE80::/64).
+func IsRFC4862(na *wire.NetAddress) bool {
+	return rfc4862Net.Contains(na.IP)
+}
+
+// IsRFC5737 returns whether or not the passed address is part of the IPv4
+// documentation address space as defined by RFC5737 (192.0.2.0/24,
+// 198.51.100.0/24, 203.0.113.0/24)
+func IsRFC5737(na *wire.NetAddress) bool {
+	for _, rfc := range rfc5737Net {
+		if rfc.Contains(na.IP) {
+			return true
+		}
+	}
+
+	return false
+}
+
+// IsRFC6052 returns whether or not the passed address is part of the IPv6
+// well-known prefix range as defined by RFC6052 (64:FF9B::/96).
+func IsRFC6052(na *wire.NetAddress) bool {
+	return rfc6052Net.Contains(na.IP)
+}
+
+// IsRFC6145 returns whether or not the passed address is part of the IPv6 to
+// IPv4 translated address range as defined by RFC6145 (::FFFF:0:0:0/96).
+func IsRFC6145(na *wire.NetAddress) bool {
+	return rfc6145Net.Contains(na.IP)
+}
+
+// IsRFC6598 returns whether or not the passed address is part of the IPv4
+// shared address space specified by RFC6598 (100.64.0.0/10)
+func IsRFC6598(na *wire.NetAddress) bool {
+	return rfc6598Net.Contains(na.IP)
+}
+
+// IsValid returns whether or not the passed address is valid. The address is
+// considered invalid under the following circumstances:
+// IPv4: It is either a zero or all bits set address.
+// IPv6: It is either a zero or RFC3849 documentation address.
+func IsValid(na *wire.NetAddress) bool {
+	// IsUnspecified returns if address is 0, so only all bits set, and
+	// RFC3849 need to be explicitly checked.
+	return na.IP != nil && !(na.IP.IsUnspecified() ||
+		na.IP.Equal(net.IPv4bcast))
+}
+
+// IsRoutable returns whether or not the passed address is routable over
+// the public internet. This is true as long as the address is valid and is not
+// in any reserved ranges.
+func IsRoutable(na *wire.NetAddress) bool {
+	if config.ActiveConfig().NetParams().AcceptUnroutable {
+		return !IsLocal(na)
+	}
+
+	return IsValid(na) && !(IsRFC1918(na) || IsRFC2544(na) ||
+		IsRFC3927(na) || IsRFC4862(na) || IsRFC3849(na) ||
+		IsRFC4843(na) || IsRFC5737(na) || IsRFC6598(na) ||
+		IsLocal(na) || (IsRFC4193(na)))
+}
+
+// GroupKey returns a string representing the network group an address is part
+// of. This is the /16 for IPv4, the /32 (/36 for he.net) for IPv6, the string
+// "local" for a local address, and the string "unroutable" for an unroutable
+// address.
+func GroupKey(na *wire.NetAddress) string {
+	if IsLocal(na) {
+		return "local"
+	}
+	if !IsRoutable(na) {
+		return "unroutable"
+	}
+	if IsIPv4(na) {
+		return na.IP.Mask(net.CIDRMask(16, 32)).String()
+	}
+	if IsRFC6145(na) || IsRFC6052(na) {
+		// last four bytes are the ip address
+		ip := na.IP[12:16]
+		return ip.Mask(net.CIDRMask(16, 32)).String()
+	}
+
+	if IsRFC3964(na) {
+		ip := na.IP[2:6]
+		return ip.Mask(net.CIDRMask(16, 32)).String()
+
+	}
+	if IsRFC4380(na) {
+		// teredo tunnels have the last 4 bytes as the v4 address XOR
+		// 0xff.
+		ip := net.IP(make([]byte, 4))
+		for i, byte := range na.IP[12:16] {
+			ip[i] = byte ^ 0xff
+		}
+		return ip.Mask(net.CIDRMask(16, 32)).String()
+	}
+
+	// OK, so now we know ourselves to be a IPv6 address.
+	// We use /32 for everything, except for Hurricane Electric's
+	// (he.net) IP range, which we use /36 for.
+	bits := 32
+	if heNet.Contains(na.IP) {
+		bits = 36
+	}
+
+	return na.IP.Mask(net.CIDRMask(bits, 128)).String()
+}

blockdag/README.md

@@ -0,0 +1,41 @@
+blockchain
+==========
+
+[![ISC License](http://img.shields.io/badge/license-ISC-blue.svg)](http://copyfree.org)
+[![GoDoc](https://img.shields.io/badge/godoc-reference-blue.svg)](http://godoc.org/github.com/kaspanet/kaspad/blockchain)
+
+Package blockdag implements Kaspa block handling, organization of the blockDAG, 
+block sorting and UTXO-set maintenance.
+The test coverage is currently only around 75%, but will be increasing over
+time.
+
+## Kaspad BlockDAG Processing Overview
+
+Before a block is allowed into the block DAG, it must go through an intensive
+series of validation rules. The following list serves as a general outline of
+those rules to provide some intuition into what is going on under the hood, but
+is by no means exhaustive:
+
+ - Reject duplicate blocks
+ - Perform a series of sanity checks on the block and its transactions such as
+   verifying proof of work, timestamps, number and character of transactions,
+   transaction amounts, script complexity, and merkle root calculations
+ - Save the most recent orphan blocks for a limited time in case their parent
+   blocks become available.
+ - Save blocks from the future for delayed processing
+ - Stop processing if the block is an orphan or delayed as the rest of the 
+   processing depends on the block's position within the block chain
+ - Make sure the block does not violate finality rules
+ - Perform a series of more thorough checks that depend on the block's position
+   within the blockDAG such as verifying block difficulties adhere to
+   difficulty retarget rules, timestamps are after the median of the last
+   several blocks, all transactions are finalized, checkpoint blocks match, and
+   block versions are in line with the previous blocks
+ - Determine how the block fits into the DAG and perform different actions
+   accordingly 
+ - Run the transaction scripts to verify the spender is allowed to spend the
+   coins
+ - Run GhostDAG to fit the block in a canonical sorting
+ - Build the block's UTXO Set, as well as update the global UTXO Set accordingly
+ - Insert the block into the block database
+

blockdag/accept.go

@@ -0,0 +1,130 @@
+// Copyright (c) 2013-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"fmt"
+	"github.com/kaspanet/kaspad/database"
+	"github.com/kaspanet/kaspad/util"
+)
+
+func (dag *BlockDAG) addNodeToIndexWithInvalidAncestor(block *util.Block) error {
+	blockHeader := &block.MsgBlock().Header
+	newNode, _ := dag.newBlockNode(blockHeader, newSet())
+	newNode.status = statusInvalidAncestor
+	dag.index.AddNode(newNode)
+	return dag.index.flushToDB()
+}
+
+// maybeAcceptBlock potentially accepts a block into the block DAG. It
+// performs several validation checks which depend on its position within
+// the block DAG before adding it. The block is expected to have already
+// gone through ProcessBlock before calling this function with it.
+//
+// The flags are also passed to checkBlockContext and connectToDAG. See
+// their documentation for how the flags modify their behavior.
+//
+// This function MUST be called with the dagLock held (for writes).
+func (dag *BlockDAG) maybeAcceptBlock(block *util.Block, flags BehaviorFlags) error {
+	parents, err := lookupParentNodes(block, dag)
+	if err != nil {
+		if rErr, ok := err.(RuleError); ok && rErr.ErrorCode == ErrInvalidAncestorBlock {
+			err := dag.addNodeToIndexWithInvalidAncestor(block)
+			if err != nil {
+				return err
+			}
+		}
+		return err
+	}
+
+	// The block must pass all of the validation rules which depend on the
+	// position of the block within the block DAG.
+	err = dag.checkBlockContext(block, parents, flags)
+	if err != nil {
+		return err
+	}
+
+	// Create a new block node for the block and add it to the node index.
+	newNode, selectedParentAnticone := dag.newBlockNode(&block.MsgBlock().Header, parents)
+	newNode.status = statusDataStored
+	dag.index.AddNode(newNode)
+
+	// Insert the block into the database if it's not already there. Even
+	// though it is possible the block will ultimately fail to connect, it
+	// has already passed all proof-of-work and validity tests which means
+	// it would be prohibitively expensive for an attacker to fill up the
+	// disk with a bunch of blocks that fail to connect. This is necessary
+	// since it allows block download to be decoupled from the much more
+	// expensive connection logic. It also has some other nice properties
+	// such as making blocks that never become part of the DAG or
+	// blocks that fail to connect available for further analysis.
+	err = dag.db.Update(func(dbTx database.Tx) error {
+		err := dbStoreBlock(dbTx, block)
+		if err != nil {
+			return err
+		}
+		return dag.index.flushToDBWithTx(dbTx)
+	})
+	if err != nil {
+		return err
+	}
+
+	// Make sure that all the block's transactions are finalized
+	fastAdd := flags&BFFastAdd == BFFastAdd
+	bluestParent := parents.bluest()
+	if !fastAdd {
+		if err := dag.validateAllTxsFinalized(block, newNode, bluestParent); err != nil {
+			return err
+		}
+	}
+
+	block.SetBlueScore(newNode.blueScore)
+
+	// Connect the passed block to the DAG. This also handles validation of the
+	// transaction scripts.
+	chainUpdates, err := dag.addBlock(newNode, block, selectedParentAnticone, flags)
+	if err != nil {
+		return err
+	}
+
+	// Notify the caller that the new block was accepted into the block
+	// DAG. The caller would typically want to react by relaying the
+	// inventory to other peers.
+	dag.dagLock.Unlock()
+	dag.sendNotification(NTBlockAdded, &BlockAddedNotificationData{
+		Block:         block,
+		WasUnorphaned: flags&BFWasUnorphaned != 0,
+	})
+	if len(chainUpdates.addedChainBlockHashes) > 0 {
+		dag.sendNotification(NTChainChanged, &ChainChangedNotificationData{
+			RemovedChainBlockHashes: chainUpdates.removedChainBlockHashes,
+			AddedChainBlockHashes:   chainUpdates.addedChainBlockHashes,
+		})
+	}
+	dag.dagLock.Lock()
+
+	return nil
+}
+
+func lookupParentNodes(block *util.Block, blockDAG *BlockDAG) (blockSet, error) {
+	header := block.MsgBlock().Header
+	parentHashes := header.ParentHashes
+
+	nodes := newSet()
+	for _, parentHash := range parentHashes {
+		node := blockDAG.index.LookupNode(parentHash)
+		if node == nil {
+			str := fmt.Sprintf("parent block %s is unknown", parentHashes)
+			return nil, ruleError(ErrParentBlockUnknown, str)
+		} else if blockDAG.index.NodeStatus(node).KnownInvalid() {
+			str := fmt.Sprintf("parent block %s is known to be invalid", parentHashes)
+			return nil, ruleError(ErrInvalidAncestorBlock, str)
+		}
+
+		nodes.add(node)
+	}
+
+	return nodes, nil
+}

blockdag/accept_test.go

@@ -0,0 +1,105 @@
+package blockdag
+
+import (
+	"path/filepath"
+	"testing"
+
+	"github.com/kaspanet/kaspad/dagconfig"
+)
+
+func TestMaybeAcceptBlockErrors(t *testing.T) {
+	// Create a new database and DAG instance to run tests against.
+	dag, teardownFunc, err := DAGSetup("TestMaybeAcceptBlockErrors", Config{
+		DAGParams: &dagconfig.SimnetParams,
+	})
+	if err != nil {
+		t.Fatalf("TestMaybeAcceptBlockErrors: Failed to setup DAG instance: %v", err)
+	}
+	defer teardownFunc()
+
+	dag.TestSetCoinbaseMaturity(0)
+
+	// Test rejecting the block if its parents are missing
+	orphanBlockFile := "blk_3B.dat"
+	loadedBlocks, err := LoadBlocks(filepath.Join("testdata/", orphanBlockFile))
+	if err != nil {
+		t.Fatalf("TestMaybeAcceptBlockErrors: "+
+			"Error loading file '%s': %s\n", orphanBlockFile, err)
+	}
+	block := loadedBlocks[0]
+
+	err = dag.maybeAcceptBlock(block, BFNone)
+	if err == nil {
+		t.Errorf("TestMaybeAcceptBlockErrors: rejecting the block if its parents are missing: "+
+			"Expected: %s, got: <nil>", ErrParentBlockUnknown)
+	}
+	ruleErr, ok := err.(RuleError)
+	if !ok {
+		t.Errorf("TestMaybeAcceptBlockErrors: rejecting the block if its parents are missing: "+
+			"Expected RuleError but got %s", err)
+	} else if ruleErr.ErrorCode != ErrParentBlockUnknown {
+		t.Errorf("TestMaybeAcceptBlockErrors: rejecting the block if its parents are missing: "+
+			"Unexpected error code. Want: %s, got: %s", ErrParentBlockUnknown, ruleErr.ErrorCode)
+	}
+
+	// Test rejecting the block if its parents are invalid
+	blocksFile := "blk_0_to_4.dat"
+	blocks, err := LoadBlocks(filepath.Join("testdata/", blocksFile))
+	if err != nil {
+		t.Fatalf("TestMaybeAcceptBlockErrors: "+
+			"Error loading file '%s': %s\n", blocksFile, err)
+	}
+
+	// Add a valid block and mark it as invalid
+	block1 := blocks[1]
+	isOrphan, isDelayed, err := dag.ProcessBlock(block1, BFNone)
+	if err != nil {
+		t.Fatalf("TestMaybeAcceptBlockErrors: Valid block unexpectedly returned an error: %s", err)
+	}
+	if isDelayed {
+		t.Fatalf("TestMaybeAcceptBlockErrors: block 1 is too far in the future")
+	}
+	if isOrphan {
+		t.Fatalf("TestMaybeAcceptBlockErrors: incorrectly returned block 1 is an orphan")
+	}
+	blockNode1 := dag.index.LookupNode(block1.Hash())
+	dag.index.SetStatusFlags(blockNode1, statusValidateFailed)
+
+	block2 := blocks[2]
+	err = dag.maybeAcceptBlock(block2, BFNone)
+	if err == nil {
+		t.Errorf("TestMaybeAcceptBlockErrors: rejecting the block if its parents are invalid: "+
+			"Expected: %s, got: <nil>", ErrInvalidAncestorBlock)
+	}
+	ruleErr, ok = err.(RuleError)
+	if !ok {
+		t.Errorf("TestMaybeAcceptBlockErrors: rejecting the block if its parents are invalid: "+
+			"Expected RuleError but got %s", err)
+	} else if ruleErr.ErrorCode != ErrInvalidAncestorBlock {
+		t.Errorf("TestMaybeAcceptBlockErrors: rejecting the block if its parents are invalid: "+
+			"Unexpected error. Want: %s, got: %s", ErrInvalidAncestorBlock, ruleErr.ErrorCode)
+	}
+
+	// Set block1's status back to valid for next tests
+	dag.index.UnsetStatusFlags(blockNode1, statusValidateFailed)
+
+	// Test rejecting the block due to bad context
+	originalBits := block2.MsgBlock().Header.Bits
+	block2.MsgBlock().Header.Bits = 0
+	err = dag.maybeAcceptBlock(block2, BFNone)
+	if err == nil {
+		t.Errorf("TestMaybeAcceptBlockErrors: rejecting the block due to bad context: "+
+			"Expected: %s, got: <nil>", ErrUnexpectedDifficulty)
+	}
+	ruleErr, ok = err.(RuleError)
+	if !ok {
+		t.Errorf("TestMaybeAcceptBlockErrors: rejecting the block due to bad context: "+
+			"Expected RuleError but got %s", err)
+	} else if ruleErr.ErrorCode != ErrUnexpectedDifficulty {
+		t.Errorf("TestMaybeAcceptBlockErrors: rejecting the block due to bad context: "+
+			"Unexpected error. Want: %s, got: %s", ErrUnexpectedDifficulty, ruleErr.ErrorCode)
+	}
+
+	// Set block2's bits back to valid for next tests
+	block2.MsgBlock().Header.Bits = originalBits
+}

blockdag/blockheap.go

@@ -0,0 +1,78 @@
+package blockdag
+
+import (
+	"container/heap"
+)
+
+// baseHeap is an implementation for heap.Interface that sorts blocks by their height
+type baseHeap []*blockNode
+
+func (h baseHeap) Len() int      { return len(h) }
+func (h baseHeap) Swap(i, j int) { h[i], h[j] = h[j], h[i] }
+
+func (h *baseHeap) Push(x interface{}) {
+	*h = append(*h, x.(*blockNode))
+}
+
+func (h *baseHeap) Pop() interface{} {
+	oldHeap := *h
+	oldLength := len(oldHeap)
+	popped := oldHeap[oldLength-1]
+	*h = oldHeap[0 : oldLength-1]
+	return popped
+}
+
+// upHeap extends baseHeap to include Less operation that traverses from bottom to top
+type upHeap struct{ baseHeap }
+
+func (h upHeap) Less(i, j int) bool {
+	return h.baseHeap[i].less(h.baseHeap[j])
+}
+
+// downHeap extends baseHeap to include Less operation that traverses from top to bottom
+type downHeap struct{ baseHeap }
+
+func (h downHeap) Less(i, j int) bool {
+	return !h.baseHeap[i].less(h.baseHeap[j])
+}
+
+// blockHeap represents a mutable heap of Blocks, sorted by their height
+type blockHeap struct {
+	impl heap.Interface
+}
+
+// newDownHeap initializes and returns a new blockHeap
+func newDownHeap() blockHeap {
+	h := blockHeap{impl: &downHeap{}}
+	heap.Init(h.impl)
+	return h
+}
+
+// newUpHeap initializes and returns a new blockHeap
+func newUpHeap() blockHeap {
+	h := blockHeap{impl: &upHeap{}}
+	heap.Init(h.impl)
+	return h
+}
+
+// pop removes the block with lowest height from this heap and returns it
+func (bh blockHeap) pop() *blockNode {
+	return heap.Pop(bh.impl).(*blockNode)
+}
+
+// Push pushes the block onto the heap
+func (bh blockHeap) Push(block *blockNode) {
+	heap.Push(bh.impl, block)
+}
+
+// pushSet pushes a blockset to the heap.
+func (bh blockHeap) pushSet(bs blockSet) {
+	for block := range bs {
+		heap.Push(bh.impl, block)
+	}
+}
+
+// Len returns the length of this heap
+func (bh blockHeap) Len() int {
+	return bh.impl.Len()
+}

blockdag/blockheap_test.go

@@ -0,0 +1,129 @@
+package blockdag
+
+import (
+	"testing"
+
+	"github.com/kaspanet/kaspad/dagconfig"
+	"github.com/kaspanet/kaspad/util/daghash"
+)
+
+// TestBlockHeap tests pushing, popping, and determining the length of the heap.
+func TestBlockHeap(t *testing.T) {
+	// Create a new database and DAG instance to run tests against.
+	dag, teardownFunc, err := DAGSetup("TestBlockHeap", Config{
+		DAGParams: &dagconfig.MainnetParams,
+	})
+	if err != nil {
+		t.Fatalf("TestBlockHeap: Failed to setup DAG instance: %s", err)
+	}
+	defer teardownFunc()
+
+	block0Header := dagconfig.MainnetParams.GenesisBlock.Header
+	block0, _ := dag.newBlockNode(&block0Header, newSet())
+
+	block100000Header := Block100000.Header
+	block100000, _ := dag.newBlockNode(&block100000Header, setFromSlice(block0))
+
+	block0smallHash, _ := dag.newBlockNode(&block0Header, newSet())
+	block0smallHash.hash = &daghash.Hash{}
+
+	tests := []struct {
+		name            string
+		toPush          []*blockNode
+		expectedLength  int
+		expectedPopUp   *blockNode
+		expectedPopDown *blockNode
+	}{
+		{
+			name:            "empty heap must have length 0",
+			toPush:          []*blockNode{},
+			expectedLength:  0,
+			expectedPopDown: nil,
+			expectedPopUp:   nil,
+		},
+		{
+			name:            "heap with one push must have length 1",
+			toPush:          []*blockNode{block0},
+			expectedLength:  1,
+			expectedPopDown: nil,
+			expectedPopUp:   nil,
+		},
+		{
+			name:            "heap with one push and one pop",
+			toPush:          []*blockNode{block0},
+			expectedLength:  0,
+			expectedPopDown: block0,
+			expectedPopUp:   block0,
+		},
+		{
+			name: "push two blocks with different heights, heap shouldn't have to rebalance " +
+				"for down direction, but will have to rebalance for up direction",
+			toPush:          []*blockNode{block100000, block0},
+			expectedLength:  1,
+			expectedPopDown: block100000,
+			expectedPopUp:   block0,
+		},
+		{
+			name: "push two blocks with different heights, heap shouldn't have to rebalance " +
+				"for up direction, but will have to rebalance for down direction",
+			toPush:          []*blockNode{block0, block100000},
+			expectedLength:  1,
+			expectedPopDown: block100000,
+			expectedPopUp:   block0,
+		},
+		{
+			name: "push two blocks with equal heights but different hashes, heap shouldn't have to rebalance " +
+				"for down direction, but will have to rebalance for up direction",
+			toPush:          []*blockNode{block0, block0smallHash},
+			expectedLength:  1,
+			expectedPopDown: block0,
+			expectedPopUp:   block0smallHash,
+		},
+		{
+			name: "push two blocks with equal heights but different hashes, heap shouldn't have to rebalance " +
+				"for up direction, but will have to rebalance for down direction",
+			toPush:          []*blockNode{block0smallHash, block0},
+			expectedLength:  1,
+			expectedPopDown: block0,
+			expectedPopUp:   block0smallHash,
+		},
+	}
+
+	for _, test := range tests {
+		dHeap := newDownHeap()
+		for _, block := range test.toPush {
+			dHeap.Push(block)
+		}
+
+		var poppedBlock *blockNode
+		if test.expectedPopDown != nil {
+			poppedBlock = dHeap.pop()
+		}
+		if dHeap.Len() != test.expectedLength {
+			t.Errorf("unexpected down heap length in test \"%s\". "+
+				"Expected: %v, got: %v", test.name, test.expectedLength, dHeap.Len())
+		}
+		if poppedBlock != test.expectedPopDown {
+			t.Errorf("unexpected popped block for down heap in test \"%s\". "+
+				"Expected: %v, got: %v", test.name, test.expectedPopDown, poppedBlock)
+		}
+
+		uHeap := newUpHeap()
+		for _, block := range test.toPush {
+			uHeap.Push(block)
+		}
+
+		poppedBlock = nil
+		if test.expectedPopUp != nil {
+			poppedBlock = uHeap.pop()
+		}
+		if uHeap.Len() != test.expectedLength {
+			t.Errorf("unexpected up heap length in test \"%s\". "+
+				"Expected: %v, got: %v", test.name, test.expectedLength, uHeap.Len())
+		}
+		if poppedBlock != test.expectedPopUp {
+			t.Errorf("unexpected popped block for up heap in test \"%s\". "+
+				"Expected: %v, got: %v", test.name, test.expectedPopDown, poppedBlock)
+		}
+	}
+}

blockdag/blockidhash.go

@@ -0,0 +1,136 @@
+package blockdag
+
+import (
+	"github.com/kaspanet/kaspad/database"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/pkg/errors"
+)
+
+var (
+	// idByHashIndexBucketName is the name of the db bucket used to house
+	// the block hash -> block id index.
+	idByHashIndexBucketName = []byte("idbyhashidx")
+
+	// hashByIDIndexBucketName is the name of the db bucket used to house
+	// the block id -> block hash index.
+	hashByIDIndexBucketName = []byte("hashbyididx")
+
+	currentBlockIDKey = []byte("currentblockid")
+)
+
+// -----------------------------------------------------------------------------
+// This is a mapping between block hashes and unique IDs. The ID
+// is simply a sequentially incremented uint64 that is used instead of block hash
+// for the indexers. This is useful because it is only 8 bytes versus 32 bytes
+// hashes and thus saves a ton of space when a block is referenced in an index.
+// It consists of three buckets: the first bucket maps the hash of each
+// block to the unique ID and the second maps that ID back to the block hash.
+// The third bucket contains the last received block ID, and is used
+// when starting the node to check that the enabled indexes are up to date
+// with the latest received block, and if not, initiate recovery process.
+//
+// The serialized format for keys and values in the block hash to ID bucket is:
+//   <hash> = <ID>
+//
+//   Field           Type              Size
+//   hash            daghash.Hash     32 bytes
+//   ID              uint64            8 bytes
+//   -----
+//   Total: 40 bytes
+//
+// The serialized format for keys and values in the ID to block hash bucket is:
+//   <ID> = <hash>
+//
+//   Field           Type              Size
+//   ID              uint64            8 bytes
+//   hash            daghash.Hash     32 bytes
+//   -----
+//   Total: 40 bytes
+//
+// -----------------------------------------------------------------------------
+
+const blockIDSize = 8 // 8 bytes for block ID
+
+// DBFetchBlockIDByHash uses an existing database transaction to retrieve the
+// block id for the provided hash from the index.
+func DBFetchBlockIDByHash(dbTx database.Tx, hash *daghash.Hash) (uint64, error) {
+	hashIndex := dbTx.Metadata().Bucket(idByHashIndexBucketName)
+	serializedID := hashIndex.Get(hash[:])
+	if serializedID == nil {
+		return 0, errors.Errorf("no entry in the block ID index for block with hash %s", hash)
+	}
+
+	return DeserializeBlockID(serializedID), nil
+}
+
+// DBFetchBlockHashBySerializedID uses an existing database transaction to
+// retrieve the hash for the provided serialized block id from the index.
+func DBFetchBlockHashBySerializedID(dbTx database.Tx, serializedID []byte) (*daghash.Hash, error) {
+	idIndex := dbTx.Metadata().Bucket(hashByIDIndexBucketName)
+	hashBytes := idIndex.Get(serializedID)
+	if hashBytes == nil {
+		return nil, errors.Errorf("no entry in the block ID index for block with id %d", byteOrder.Uint64(serializedID))
+	}
+
+	var hash daghash.Hash
+	copy(hash[:], hashBytes)
+	return &hash, nil
+}
+
+// dbPutBlockIDIndexEntry uses an existing database transaction to update or add
+// the index entries for the hash to id and id to hash mappings for the provided
+// values.
+func dbPutBlockIDIndexEntry(dbTx database.Tx, hash *daghash.Hash, serializedID []byte) error {
+	// Add the block hash to ID mapping to the index.
+	meta := dbTx.Metadata()
+	hashIndex := meta.Bucket(idByHashIndexBucketName)
+	if err := hashIndex.Put(hash[:], serializedID[:]); err != nil {
+		return err
+	}
+
+	// Add the block ID to hash mapping to the index.
+	idIndex := meta.Bucket(hashByIDIndexBucketName)
+	return idIndex.Put(serializedID[:], hash[:])
+}
+
+// DBFetchCurrentBlockID returns the last known block ID.
+func DBFetchCurrentBlockID(dbTx database.Tx) uint64 {
+	serializedID := dbTx.Metadata().Get(currentBlockIDKey)
+	if serializedID == nil {
+		return 0
+	}
+	return DeserializeBlockID(serializedID)
+}
+
+// DeserializeBlockID returns a deserialized block id
+func DeserializeBlockID(serializedID []byte) uint64 {
+	return byteOrder.Uint64(serializedID)
+}
+
+// SerializeBlockID returns a serialized block id
+func SerializeBlockID(blockID uint64) []byte {
+	serializedBlockID := make([]byte, blockIDSize)
+	byteOrder.PutUint64(serializedBlockID, blockID)
+	return serializedBlockID
+}
+
+// DBFetchBlockHashByID uses an existing database transaction to retrieve the
+// hash for the provided block id from the index.
+func DBFetchBlockHashByID(dbTx database.Tx, id uint64) (*daghash.Hash, error) {
+	return DBFetchBlockHashBySerializedID(dbTx, SerializeBlockID(id))
+}
+
+func createBlockID(dbTx database.Tx, blockHash *daghash.Hash) (uint64, error) {
+	currentBlockID := DBFetchCurrentBlockID(dbTx)
+	newBlockID := currentBlockID + 1
+	serializedNewBlockID := SerializeBlockID(newBlockID)
+	err := dbTx.Metadata().Put(currentBlockIDKey, serializedNewBlockID)
+	if err != nil {
+		return 0, err
+	}
+	err = dbPutBlockIDIndexEntry(dbTx, blockHash, serializedNewBlockID)
+	if err != nil {
+		return 0, err
+	}
+	return newBlockID, nil
+}

blockdag/blockindex.go

@@ -0,0 +1,141 @@
+// Copyright (c) 2015-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"sync"
+
+	"github.com/kaspanet/kaspad/dagconfig"
+	"github.com/kaspanet/kaspad/database"
+	"github.com/kaspanet/kaspad/util/daghash"
+)
+
+// blockIndex provides facilities for keeping track of an in-memory index of the
+// block DAG.
+type blockIndex struct {
+	// The following fields are set when the instance is created and can't
+	// be changed afterwards, so there is no need to protect them with a
+	// separate mutex.
+	db        database.DB
+	dagParams *dagconfig.Params
+
+	sync.RWMutex
+	index map[daghash.Hash]*blockNode
+	dirty map[*blockNode]struct{}
+}
+
+// newBlockIndex returns a new empty instance of a block index. The index will
+// be dynamically populated as block nodes are loaded from the database and
+// manually added.
+func newBlockIndex(db database.DB, dagParams *dagconfig.Params) *blockIndex {
+	return &blockIndex{
+		db:        db,
+		dagParams: dagParams,
+		index:     make(map[daghash.Hash]*blockNode),
+		dirty:     make(map[*blockNode]struct{}),
+	}
+}
+
+// HaveBlock returns whether or not the block index contains the provided hash.
+//
+// This function is safe for concurrent access.
+func (bi *blockIndex) HaveBlock(hash *daghash.Hash) bool {
+	bi.RLock()
+	_, hasBlock := bi.index[*hash]
+	bi.RUnlock()
+	return hasBlock
+}
+
+// LookupNode returns the block node identified by the provided hash. It will
+// return nil if there is no entry for the hash.
+//
+// This function is safe for concurrent access.
+func (bi *blockIndex) LookupNode(hash *daghash.Hash) *blockNode {
+	bi.RLock()
+	node := bi.index[*hash]
+	bi.RUnlock()
+	return node
+}
+
+// AddNode adds the provided node to the block index and marks it as dirty.
+// Duplicate entries are not checked so it is up to caller to avoid adding them.
+//
+// This function is safe for concurrent access.
+func (bi *blockIndex) AddNode(node *blockNode) {
+	bi.Lock()
+	bi.addNode(node)
+	bi.dirty[node] = struct{}{}
+	bi.Unlock()
+}
+
+// addNode adds the provided node to the block index, but does not mark it as
+// dirty. This can be used while initializing the block index.
+//
+// This function is NOT safe for concurrent access.
+func (bi *blockIndex) addNode(node *blockNode) {
+	bi.index[*node.hash] = node
+}
+
+// NodeStatus provides concurrent-safe access to the status field of a node.
+//
+// This function is safe for concurrent access.
+func (bi *blockIndex) NodeStatus(node *blockNode) blockStatus {
+	bi.RLock()
+	status := node.status
+	bi.RUnlock()
+	return status
+}
+
+// SetStatusFlags flips the provided status flags on the block node to on,
+// regardless of whether they were on or off previously. This does not unset any
+// flags currently on.
+//
+// This function is safe for concurrent access.
+func (bi *blockIndex) SetStatusFlags(node *blockNode, flags blockStatus) {
+	bi.Lock()
+	node.status |= flags
+	bi.dirty[node] = struct{}{}
+	bi.Unlock()
+}
+
+// UnsetStatusFlags flips the provided status flags on the block node to off,
+// regardless of whether they were on or off previously.
+//
+// This function is safe for concurrent access.
+func (bi *blockIndex) UnsetStatusFlags(node *blockNode, flags blockStatus) {
+	bi.Lock()
+	node.status &^= flags
+	bi.dirty[node] = struct{}{}
+	bi.Unlock()
+}
+
+// flushToDB writes all dirty block nodes to the database. If all writes
+// succeed, this clears the dirty set.
+func (bi *blockIndex) flushToDB() error {
+	return bi.db.Update(func(dbTx database.Tx) error {
+		return bi.flushToDBWithTx(dbTx)
+	})
+}
+
+// flushToDBWithTx writes all dirty block nodes to the database. If all
+// writes succeed, this clears the dirty set.
+func (bi *blockIndex) flushToDBWithTx(dbTx database.Tx) error {
+	bi.Lock()
+	defer bi.Unlock()
+	if len(bi.dirty) == 0 {
+		return nil
+	}
+
+	for node := range bi.dirty {
+		err := dbStoreBlockNode(dbTx, node)
+		if err != nil {
+			return err
+		}
+	}
+
+	// If write was successful, clear the dirty set.
+	bi.dirty = make(map[*blockNode]struct{})
+	return nil
+}

blockdag/blockindex_test.go

@@ -0,0 +1,27 @@
+package blockdag
+
+import (
+	"testing"
+	"time"
+
+	"github.com/kaspanet/kaspad/dagconfig"
+)
+
+func TestAncestorErrors(t *testing.T) {
+	// Create a new database and DAG instance to run tests against.
+	params := dagconfig.SimnetParams
+	dag, teardownFunc, err := DAGSetup("TestAncestorErrors", Config{
+		DAGParams: &params,
+	})
+	if err != nil {
+		t.Fatalf("TestAncestorErrors: Failed to setup DAG instance: %s", err)
+	}
+	defer teardownFunc()
+
+	node := newTestNode(dag, newSet(), int32(0x10000000), 0, time.Unix(0, 0))
+	node.blueScore = 2
+	ancestor := node.SelectedAncestor(3)
+	if ancestor != nil {
+		t.Errorf("TestAncestorErrors: Ancestor() unexpectedly returned a node. Expected: <nil>")
+	}
+}

blockdag/blocklocator.go

@@ -0,0 +1,102 @@
+package blockdag
+
+import (
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/pkg/errors"
+)
+
+// BlockLocator is used to help locate a specific block. The algorithm for
+// building the block locator is to add block hashes in reverse order on the
+// block's selected parent chain until the desired stop block is reached.
+// In order to keep the list of locator hashes to a reasonable number of entries,
+// the step between each entry is doubled each loop iteration to exponentially
+// decrease the number of hashes as a function of the distance from the block
+// being located.
+//
+// For example, assume a selected parent chain with IDs as depicted below, and the
+// stop block is genesis:
+// 	genesis -> 1 -> 2 -> ... -> 15 -> 16  -> 17  -> 18
+//
+// The block locator for block 17 would be the hashes of blocks:
+//  [17 16 14 11 7 2 genesis]
+type BlockLocator []*daghash.Hash
+
+// BlockLocatorFromHashes returns a block locator from high and low hash.
+// See BlockLocator for details on the algorithm used to create a block locator.
+//
+// This function is safe for concurrent access.
+func (dag *BlockDAG) BlockLocatorFromHashes(highHash, lowHash *daghash.Hash) (BlockLocator, error) {
+	dag.dagLock.RLock()
+	defer dag.dagLock.RUnlock()
+
+	highNode := dag.index.LookupNode(highHash)
+	lowNode := dag.index.LookupNode(lowHash)
+
+	return dag.blockLocator(highNode, lowNode)
+}
+
+// blockLocator returns a block locator for the passed high and low nodes.
+// See the BlockLocator type comments for more details.
+//
+// This function MUST be called with the DAG state lock held (for reads).
+func (dag *BlockDAG) blockLocator(highNode, lowNode *blockNode) (BlockLocator, error) {
+	// We use the selected parent of the high node, so the
+	// block locator won't contain the high node.
+	highNode = highNode.selectedParent
+
+	node := highNode
+	step := uint64(1)
+	locator := make(BlockLocator, 0)
+	for node != nil {
+		locator = append(locator, node.hash)
+
+		// Nothing more to add once the low node has been added.
+		if node.blueScore <= lowNode.blueScore {
+			if node != lowNode {
+				return nil, errors.Errorf("highNode and lowNode are " +
+					"not in the same selected parent chain.")
+			}
+			break
+		}
+
+		// Calculate blueScore of previous node to include ensuring the
+		// final node is lowNode.
+		nextBlueScore := node.blueScore - step
+		if nextBlueScore < lowNode.blueScore {
+			nextBlueScore = lowNode.blueScore
+		}
+
+		// walk backwards through the nodes to the correct ancestor.
+		node = node.SelectedAncestor(nextBlueScore)
+
+		// Double the distance between included hashes.
+		step *= 2
+	}
+
+	return locator, nil
+}
+
+// FindNextLocatorBoundaries returns the lowest unknown block locator, hash
+// and the highest known block locator hash. This is used to create the
+// next block locator to find the highest shared known chain block with the
+// sync peer.
+//
+// This function MUST be called with the DAG state lock held (for reads).
+func (dag *BlockDAG) FindNextLocatorBoundaries(locator BlockLocator) (highHash, lowHash *daghash.Hash) {
+	// Find the most recent locator block hash in the DAG. In the case none of
+	// the hashes in the locator are in the DAG, fall back to the genesis block.
+	lowNode := dag.genesis
+	nextBlockLocatorIndex := int64(len(locator) - 1)
+	for i, hash := range locator {
+		node := dag.index.LookupNode(hash)
+		if node != nil {
+			lowNode = node
+			nextBlockLocatorIndex = int64(i) - 1
+			break
+		}
+	}
+	if nextBlockLocatorIndex < 0 {
+		return nil, lowNode.hash
+	}
+	return locator[nextBlockLocatorIndex], lowNode.hash
+}

blockdag/blocknode.go

@@ -0,0 +1,237 @@
+// Copyright (c) 2015-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"fmt"
+	"github.com/kaspanet/kaspad/dagconfig"
+	"github.com/pkg/errors"
+	"math"
+	"time"
+
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/wire"
+)
+
+// blockStatus is a bit field representing the validation state of the block.
+type blockStatus byte
+
+const (
+	// statusDataStored indicates that the block's payload is stored on disk.
+	statusDataStored blockStatus = 1 << iota
+
+	// statusValid indicates that the block has been fully validated.
+	statusValid
+
+	// statusValidateFailed indicates that the block has failed validation.
+	statusValidateFailed
+
+	// statusInvalidAncestor indicates that one of the block's ancestors has
+	// has failed validation, thus the block is also invalid.
+	statusInvalidAncestor
+
+	// statusNone indicates that the block has no validation state flags set.
+	//
+	// NOTE: This must be defined last in order to avoid influencing iota.
+	statusNone blockStatus = 0
+)
+
+// KnownValid returns whether the block is known to be valid. This will return
+// false for a valid block that has not been fully validated yet.
+func (status blockStatus) KnownValid() bool {
+	return status&statusValid != 0
+}
+
+// KnownInvalid returns whether the block is known to be invalid. This may be
+// because the block itself failed validation or any of its ancestors is
+// invalid. This will return false for invalid blocks that have not been proven
+// invalid yet.
+func (status blockStatus) KnownInvalid() bool {
+	return status&(statusValidateFailed|statusInvalidAncestor) != 0
+}
+
+// blockNode represents a block within the block DAG. The DAG is stored into
+// the block database.
+type blockNode struct {
+	// NOTE: Additions, deletions, or modifications to the order of the
+	// definitions in this struct should not be changed without considering
+	// how it affects alignment on 64-bit platforms. The current order is
+	// specifically crafted to result in minimal padding. There will be
+	// hundreds of thousands of these in memory, so a few extra bytes of
+	// padding adds up.
+
+	// parents is the parent blocks for this node.
+	parents blockSet
+
+	// selectedParent is the selected parent for this node.
+	// The selected parent is the parent that if chosen will maximize the blue score of this block
+	selectedParent *blockNode
+
+	// children are all the blocks that refer to this block as a parent
+	children blockSet
+
+	// blues are all blue blocks in this block's worldview that are in its selected parent anticone
+	blues []*blockNode
+
+	// blueScore is the count of all the blue blocks in this block's past
+	blueScore uint64
+
+	// bluesAnticoneSizes is a map holding the set of blues affected by this block and their
+	// modified blue anticone size.
+	bluesAnticoneSizes map[daghash.Hash]dagconfig.KType
+
+	// hash is the double sha 256 of the block.
+	hash *daghash.Hash
+
+	// Some fields from block headers to aid in  reconstructing headers
+	// from memory. These must be treated as immutable and are intentionally
+	// ordered to avoid padding on 64-bit platforms.
+	version              int32
+	bits                 uint32
+	nonce                uint64
+	timestamp            int64
+	hashMerkleRoot       *daghash.Hash
+	acceptedIDMerkleRoot *daghash.Hash
+	utxoCommitment       *daghash.Hash
+
+	// status is a bitfield representing the validation state of the block. The
+	// status field, unlike the other fields, may be written to and so should
+	// only be accessed using the concurrent-safe NodeStatus method on
+	// blockIndex once the node has been added to the global index.
+	status blockStatus
+
+	// isFinalized determines whether the node is below the finality point.
+	isFinalized bool
+}
+
+// newBlockNode returns a new block node for the given block header and parents, and the
+// anticone of its selected parent (parent with highest blue score).
+// selectedParentAnticone is used to update reachability data we store for future reachability queries.
+// This function is NOT safe for concurrent access.
+func (dag *BlockDAG) newBlockNode(blockHeader *wire.BlockHeader, parents blockSet) (node *blockNode, selectedParentAnticone []*blockNode) {
+	node = &blockNode{
+		parents:            parents,
+		children:           make(blockSet),
+		blueScore:          math.MaxUint64, // Initialized to the max value to avoid collisions with the genesis block
+		timestamp:          dag.AdjustedTime().Unix(),
+		bluesAnticoneSizes: make(map[daghash.Hash]dagconfig.KType),
+	}
+
+	// blockHeader is nil only for the virtual block
+	if blockHeader != nil {
+		node.hash = blockHeader.BlockHash()
+		node.version = blockHeader.Version
+		node.bits = blockHeader.Bits
+		node.nonce = blockHeader.Nonce
+		node.timestamp = blockHeader.Timestamp.Unix()
+		node.hashMerkleRoot = blockHeader.HashMerkleRoot
+		node.acceptedIDMerkleRoot = blockHeader.AcceptedIDMerkleRoot
+		node.utxoCommitment = blockHeader.UTXOCommitment
+	} else {
+		node.hash = &daghash.ZeroHash
+	}
+
+	if len(parents) == 0 {
+		// The genesis block is defined to have a blueScore of 0
+		node.blueScore = 0
+		return node, nil
+	}
+
+	selectedParentAnticone, err := dag.ghostdag(node)
+	if err != nil {
+		panic(errors.Wrap(err, "unexpected error in GHOSTDAG"))
+	}
+	return node, selectedParentAnticone
+}
+
+// updateParentsChildren updates the node's parents to point to new node
+func (node *blockNode) updateParentsChildren() {
+	for parent := range node.parents {
+		parent.children.add(node)
+	}
+}
+
+func (node *blockNode) less(other *blockNode) bool {
+	if node.blueScore == other.blueScore {
+		return daghash.Less(node.hash, other.hash)
+	}
+
+	return node.blueScore < other.blueScore
+}
+
+// Header constructs a block header from the node and returns it.
+//
+// This function is safe for concurrent access.
+func (node *blockNode) Header() *wire.BlockHeader {
+	// No lock is needed because all accessed fields are immutable.
+	return &wire.BlockHeader{
+		Version:              node.version,
+		ParentHashes:         node.ParentHashes(),
+		HashMerkleRoot:       node.hashMerkleRoot,
+		AcceptedIDMerkleRoot: node.acceptedIDMerkleRoot,
+		UTXOCommitment:       node.utxoCommitment,
+		Timestamp:            time.Unix(node.timestamp, 0),
+		Bits:                 node.bits,
+		Nonce:                node.nonce,
+	}
+}
+
+// SelectedAncestor returns the ancestor block node at the provided blue score by following
+// the selected-parents chain backwards from this node. The returned block will be nil when a
+// blue score is requested that is higher than the blue score of the passed node.
+//
+// This function is safe for concurrent access.
+func (node *blockNode) SelectedAncestor(blueScore uint64) *blockNode {
+	if blueScore < 0 || blueScore > node.blueScore {
+		return nil
+	}
+
+	n := node
+	for n != nil && n.blueScore > blueScore {
+		n = n.selectedParent
+	}
+
+	return n
+}
+
+// RelativeAncestor returns the ancestor block node a relative 'distance' of
+// blue blocks before this node. This is equivalent to calling Ancestor with
+// the node's blue score minus provided distance.
+//
+// This function is safe for concurrent access.
+func (node *blockNode) RelativeAncestor(distance uint64) *blockNode {
+	return node.SelectedAncestor(node.blueScore - distance)
+}
+
+// CalcPastMedianTime returns the median time of the previous few blocks
+// prior to, and including, the block node.
+//
+// This function is safe for concurrent access.
+func (node *blockNode) PastMedianTime(dag *BlockDAG) time.Time {
+	window := blueBlockWindow(node, 2*dag.TimestampDeviationTolerance-1)
+	medianTimestamp, err := window.medianTimestamp()
+	if err != nil {
+		panic(fmt.Sprintf("blueBlockWindow: %s", err))
+	}
+	return time.Unix(medianTimestamp, 0)
+}
+
+func (node *blockNode) ParentHashes() []*daghash.Hash {
+	return node.parents.hashes()
+}
+
+// isGenesis returns if the current block is the genesis block
+func (node *blockNode) isGenesis() bool {
+	return len(node.parents) == 0
+}
+
+func (node *blockNode) finalityScore(dag *BlockDAG) uint64 {
+	return node.blueScore / uint64(dag.dagParams.FinalityInterval)
+}
+
+// String returns a string that contains the block hash.
+func (node blockNode) String() string {
+	return node.hash.String()
+}

blockdag/blocknode_test.go

@@ -0,0 +1,40 @@
+package blockdag
+
+import (
+	"github.com/kaspanet/kaspad/dagconfig"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"testing"
+)
+
+// This test is to ensure the size BlueAnticoneSizesSize is serialized to the size of KType.
+// We verify that by serializing and deserializing the block while making sure that we stay within the expected range.
+func TestBlueAnticoneSizesSize(t *testing.T) {
+	dag, teardownFunc, err := DAGSetup("TestBlueAnticoneSizesSize", Config{
+		DAGParams: &dagconfig.SimnetParams,
+	})
+	if err != nil {
+		t.Fatalf("TestBlueAnticoneSizesSize: Failed to setup DAG instance: %s", err)
+	}
+	defer teardownFunc()
+
+	k := dagconfig.KType(0)
+	k--
+
+	if k < dagconfig.KType(0) {
+		t.Fatalf("KType must be unsigned")
+	}
+
+	blockHeader := dagconfig.SimnetParams.GenesisBlock.Header
+	node, _ := dag.newBlockNode(&blockHeader, newSet())
+	hash := daghash.Hash{1}
+	// Setting maxKType to maximum value of KType.
+	// As we verify above that KType is unsigned we can be sure that maxKType is indeed the maximum value of KType.
+	maxKType := ^dagconfig.KType(0)
+	node.bluesAnticoneSizes[hash] = maxKType
+	serializedNode, _ := serializeBlockNode(node)
+	deserializedNode, _ := dag.deserializeBlockNode(serializedNode)
+	if deserializedNode.bluesAnticoneSizes[hash] != maxKType {
+		t.Fatalf("TestBlueAnticoneSizesSize: BlueAnticoneSize should not change when deserializing. Expected: %v but got %v",
+			maxKType, deserializedNode.bluesAnticoneSizes[hash])
+	}
+}

blockdag/blockset.go

@@ -0,0 +1,128 @@
+package blockdag
+
+import (
+	"strings"
+
+	"github.com/kaspanet/kaspad/util/daghash"
+)
+
+// blockSet implements a basic unsorted set of blocks
+type blockSet map[*blockNode]struct{}
+
+// newSet creates a new, empty BlockSet
+func newSet() blockSet {
+	return map[*blockNode]struct{}{}
+}
+
+// setFromSlice converts a slice of blockNodes into an unordered set represented as map
+func setFromSlice(nodes ...*blockNode) blockSet {
+	set := newSet()
+	for _, node := range nodes {
+		set.add(node)
+	}
+	return set
+}
+
+// add adds a blockNode to this BlockSet
+func (bs blockSet) add(node *blockNode) {
+	bs[node] = struct{}{}
+}
+
+// remove removes a blockNode from this BlockSet, if exists
+// Does nothing if this set does not contain the blockNode
+func (bs blockSet) remove(node *blockNode) {
+	delete(bs, node)
+}
+
+// clone clones thie block set
+func (bs blockSet) clone() blockSet {
+	clone := newSet()
+	for node := range bs {
+		clone.add(node)
+	}
+	return clone
+}
+
+// subtract returns the difference between the BlockSet and another BlockSet
+func (bs blockSet) subtract(other blockSet) blockSet {
+	diff := newSet()
+	for node := range bs {
+		if !other.contains(node) {
+			diff.add(node)
+		}
+	}
+	return diff
+}
+
+// addSet adds all blockNodes in other set to this set
+func (bs blockSet) addSet(other blockSet) {
+	for node := range other {
+		bs.add(node)
+	}
+}
+
+// addSlice adds provided slice to this set
+func (bs blockSet) addSlice(slice []*blockNode) {
+	for _, node := range slice {
+		bs.add(node)
+	}
+}
+
+// union returns a BlockSet that contains all blockNodes included in this set,
+// the other set, or both
+func (bs blockSet) union(other blockSet) blockSet {
+	union := bs.clone()
+
+	union.addSet(other)
+
+	return union
+}
+
+// contains returns true iff this set contains node
+func (bs blockSet) contains(node *blockNode) bool {
+	_, ok := bs[node]
+	return ok
+}
+
+// hashes returns the hashes of the blockNodes in this set.
+func (bs blockSet) hashes() []*daghash.Hash {
+	hashes := make([]*daghash.Hash, 0, len(bs))
+	for node := range bs {
+		hashes = append(hashes, node.hash)
+	}
+	daghash.Sort(hashes)
+	return hashes
+}
+
+func (bs blockSet) String() string {
+	nodeStrs := make([]string, 0, len(bs))
+	for node := range bs {
+		nodeStrs = append(nodeStrs, node.String())
+	}
+	return strings.Join(nodeStrs, ",")
+}
+
+// anyChildInSet returns true iff any child of node is contained within this set
+func (bs blockSet) anyChildInSet(node *blockNode) bool {
+	for child := range node.children {
+		if bs.contains(child) {
+			return true
+		}
+	}
+
+	return false
+}
+
+func (bs blockSet) bluest() *blockNode {
+	var bluestNode *blockNode
+	var maxScore uint64
+	for node := range bs {
+		if bluestNode == nil ||
+			node.blueScore > maxScore ||
+			(node.blueScore == maxScore && daghash.Less(node.hash, bluestNode.hash)) {
+			bluestNode = node
+			maxScore = node.blueScore
+		}
+	}
+	return bluestNode
+}

blockdag/blockset_test.go

@@ -0,0 +1,245 @@
+package blockdag
+
+import (
+	"reflect"
+	"testing"
+
+	"github.com/kaspanet/kaspad/util/daghash"
+)
+
+func TestHashes(t *testing.T) {
+	bs := setFromSlice(
+		&blockNode{
+			hash: &daghash.Hash{3},
+		},
+		&blockNode{
+			hash: &daghash.Hash{1},
+		},
+		&blockNode{
+			hash: &daghash.Hash{0},
+		},
+		&blockNode{
+			hash: &daghash.Hash{2},
+		},
+	)
+
+	expected := []*daghash.Hash{
+		{0},
+		{1},
+		{2},
+		{3},
+	}
+
+	hashes := bs.hashes()
+	if !daghash.AreEqual(hashes, expected) {
+		t.Errorf("TestHashes: hashes order is %s but expected %s", hashes, expected)
+	}
+}
+
+func TestBlockSetSubtract(t *testing.T) {
+	node1 := &blockNode{hash: &daghash.Hash{10}}
+	node2 := &blockNode{hash: &daghash.Hash{20}}
+	node3 := &blockNode{hash: &daghash.Hash{30}}
+
+	tests := []struct {
+		name           string
+		setA           blockSet
+		setB           blockSet
+		expectedResult blockSet
+	}{
+		{
+			name:           "both sets empty",
+			setA:           setFromSlice(),
+			setB:           setFromSlice(),
+			expectedResult: setFromSlice(),
+		},
+		{
+			name:           "subtract an empty set",
+			setA:           setFromSlice(node1),
+			setB:           setFromSlice(),
+			expectedResult: setFromSlice(node1),
+		},
+		{
+			name:           "subtract from empty set",
+			setA:           setFromSlice(),
+			setB:           setFromSlice(node1),
+			expectedResult: setFromSlice(),
+		},
+		{
+			name:           "subtract unrelated set",
+			setA:           setFromSlice(node1),
+			setB:           setFromSlice(node2),
+			expectedResult: setFromSlice(node1),
+		},
+		{
+			name:           "typical case",
+			setA:           setFromSlice(node1, node2),
+			setB:           setFromSlice(node2, node3),
+			expectedResult: setFromSlice(node1),
+		},
+	}
+
+	for _, test := range tests {
+		result := test.setA.subtract(test.setB)
+		if !reflect.DeepEqual(result, test.expectedResult) {
+			t.Errorf("blockSet.subtract: unexpected result in test '%s'. "+
+				"Expected: %v, got: %v", test.name, test.expectedResult, result)
+		}
+	}
+}
+
+func TestBlockSetAddSet(t *testing.T) {
+	node1 := &blockNode{hash: &daghash.Hash{10}}
+	node2 := &blockNode{hash: &daghash.Hash{20}}
+	node3 := &blockNode{hash: &daghash.Hash{30}}
+
+	tests := []struct {
+		name           string
+		setA           blockSet
+		setB           blockSet
+		expectedResult blockSet
+	}{
+		{
+			name:           "both sets empty",
+			setA:           setFromSlice(),
+			setB:           setFromSlice(),
+			expectedResult: setFromSlice(),
+		},
+		{
+			name:           "add an empty set",
+			setA:           setFromSlice(node1),
+			setB:           setFromSlice(),
+			expectedResult: setFromSlice(node1),
+		},
+		{
+			name:           "add to empty set",
+			setA:           setFromSlice(),
+			setB:           setFromSlice(node1),
+			expectedResult: setFromSlice(node1),
+		},
+		{
+			name:           "add already added member",
+			setA:           setFromSlice(node1, node2),
+			setB:           setFromSlice(node1),
+			expectedResult: setFromSlice(node1, node2),
+		},
+		{
+			name:           "typical case",
+			setA:           setFromSlice(node1, node2),
+			setB:           setFromSlice(node2, node3),
+			expectedResult: setFromSlice(node1, node2, node3),
+		},
+	}
+
+	for _, test := range tests {
+		test.setA.addSet(test.setB)
+		if !reflect.DeepEqual(test.setA, test.expectedResult) {
+			t.Errorf("blockSet.addSet: unexpected result in test '%s'. "+
+				"Expected: %v, got: %v", test.name, test.expectedResult, test.setA)
+		}
+	}
+}
+
+func TestBlockSetAddSlice(t *testing.T) {
+	node1 := &blockNode{hash: &daghash.Hash{10}}
+	node2 := &blockNode{hash: &daghash.Hash{20}}
+	node3 := &blockNode{hash: &daghash.Hash{30}}
+
+	tests := []struct {
+		name           string
+		set            blockSet
+		slice          []*blockNode
+		expectedResult blockSet
+	}{
+		{
+			name:           "add empty slice to empty set",
+			set:            setFromSlice(),
+			slice:          []*blockNode{},
+			expectedResult: setFromSlice(),
+		},
+		{
+			name:           "add an empty slice",
+			set:            setFromSlice(node1),
+			slice:          []*blockNode{},
+			expectedResult: setFromSlice(node1),
+		},
+		{
+			name:           "add to empty set",
+			set:            setFromSlice(),
+			slice:          []*blockNode{node1},
+			expectedResult: setFromSlice(node1),
+		},
+		{
+			name:           "add already added member",
+			set:            setFromSlice(node1, node2),
+			slice:          []*blockNode{node1},
+			expectedResult: setFromSlice(node1, node2),
+		},
+		{
+			name:           "typical case",
+			set:            setFromSlice(node1, node2),
+			slice:          []*blockNode{node2, node3},
+			expectedResult: setFromSlice(node1, node2, node3),
+		},
+	}
+
+	for _, test := range tests {
+		test.set.addSlice(test.slice)
+		if !reflect.DeepEqual(test.set, test.expectedResult) {
+			t.Errorf("blockSet.addSlice: unexpected result in test '%s'. "+
+				"Expected: %v, got: %v", test.name, test.expectedResult, test.set)
+		}
+	}
+}
+
+func TestBlockSetUnion(t *testing.T) {
+	node1 := &blockNode{hash: &daghash.Hash{10}}
+	node2 := &blockNode{hash: &daghash.Hash{20}}
+	node3 := &blockNode{hash: &daghash.Hash{30}}
+
+	tests := []struct {
+		name           string
+		setA           blockSet
+		setB           blockSet
+		expectedResult blockSet
+	}{
+		{
+			name:           "both sets empty",
+			setA:           setFromSlice(),
+			setB:           setFromSlice(),
+			expectedResult: setFromSlice(),
+		},
+		{
+			name:           "union against an empty set",
+			setA:           setFromSlice(node1),
+			setB:           setFromSlice(),
+			expectedResult: setFromSlice(node1),
+		},
+		{
+			name:           "union from an empty set",
+			setA:           setFromSlice(),
+			setB:           setFromSlice(node1),
+			expectedResult: setFromSlice(node1),
+		},
+		{
+			name:           "union with subset",
+			setA:           setFromSlice(node1, node2),
+			setB:           setFromSlice(node1),
+			expectedResult: setFromSlice(node1, node2),
+		},
+		{
+			name:           "typical case",
+			setA:           setFromSlice(node1, node2),
+			setB:           setFromSlice(node2, node3),
+			expectedResult: setFromSlice(node1, node2, node3),
+		},
+	}
+
+	for _, test := range tests {
+		result := test.setA.union(test.setB)
+		if !reflect.DeepEqual(result, test.expectedResult) {
+			t.Errorf("blockSet.union: unexpected result in test '%s'. "+
+				"Expected: %v, got: %v", test.name, test.expectedResult, result)
+		}
+	}
+}

blockdag/blockwindow.go

@@ -0,0 +1,75 @@
+package blockdag
+
+import (
+	"github.com/kaspanet/kaspad/util"
+	"github.com/pkg/errors"
+	"math"
+	"math/big"
+	"sort"
+)
+
+type blockWindow []*blockNode
+
+// blueBlockWindow returns a blockWindow of the given size that contains the
+// blues in the past of startindNode, sorted by GHOSTDAG order.
+// If the number of blues in the past of startingNode is less then windowSize,
+// the window will be padded by genesis blocks to achieve a size of windowSize.
+func blueBlockWindow(startingNode *blockNode, windowSize uint64) blockWindow {
+	window := make(blockWindow, 0, windowSize)
+	currentNode := startingNode
+	for uint64(len(window)) < windowSize && currentNode.selectedParent != nil {
+		if currentNode.selectedParent != nil {
+			for _, blue := range currentNode.blues {
+				window = append(window, blue)
+				if uint64(len(window)) == windowSize {
+					break
+				}
+			}
+			currentNode = currentNode.selectedParent
+		}
+	}
+
+	if uint64(len(window)) < windowSize {
+		genesis := currentNode
+		for uint64(len(window)) < windowSize {
+			window = append(window, genesis)
+		}
+	}
+
+	return window
+}
+
+func (window blockWindow) minMaxTimestamps() (min, max int64) {
+	min = math.MaxInt64
+	max = 0
+	for _, node := range window {
+		if node.timestamp < min {
+			min = node.timestamp
+		}
+		if node.timestamp > max {
+			max = node.timestamp
+		}
+	}
+	return
+}
+
+func (window blockWindow) averageTarget() *big.Int {
+	averageTarget := big.NewInt(0)
+	for _, node := range window {
+		target := util.CompactToBig(node.bits)
+		averageTarget.Add(averageTarget, target)
+	}
+	return averageTarget.Div(averageTarget, big.NewInt(int64(len(window))))
+}
+
+func (window blockWindow) medianTimestamp() (int64, error) {
+	if len(window) == 0 {
+		return 0, errors.New("Cannot calculate median timestamp for an empty block window")
+	}
+	timestamps := make([]int64, len(window))
+	for i, node := range window {
+		timestamps[i] = node.timestamp
+	}
+	sort.Sort(timeSorter(timestamps))
+	return timestamps[len(timestamps)/2], nil
+}

blockdag/blockwindow_test.go

@@ -0,0 +1,157 @@
+package blockdag
+
+import (
+	"github.com/kaspanet/kaspad/dagconfig"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/pkg/errors"
+	"reflect"
+	"testing"
+	"time"
+)
+
+func TestBlueBlockWindow(t *testing.T) {
+	params := dagconfig.SimnetParams
+	params.K = 1
+	dag, teardownFunc, err := DAGSetup("TestBlueBlockWindow", Config{
+		DAGParams: &params,
+	})
+	if err != nil {
+		t.Fatalf("Failed to setup dag instance: %v", err)
+	}
+	defer teardownFunc()
+
+	resetExtraNonceForTest()
+
+	windowSize := uint64(10)
+	genesisNode := dag.genesis
+	blockTime := genesisNode.Header().Timestamp
+	blockByIDMap := make(map[string]*blockNode)
+	idByBlockMap := make(map[*blockNode]string)
+	blockByIDMap["A"] = genesisNode
+	idByBlockMap[genesisNode] = "A"
+
+	blocksData := []*struct {
+		parents                          []string
+		id                               string //id is a virtual entity that is used only for tests so we can define relations between blocks without knowing their hash
+		expectedWindowWithGenesisPadding []string
+	}{
+		{
+			parents:                          []string{"A"},
+			id:                               "B",
+			expectedWindowWithGenesisPadding: []string{"A", "A", "A", "A", "A", "A", "A", "A", "A", "A"},
+		},
+		{
+			parents:                          []string{"B"},
+			id:                               "C",
+			expectedWindowWithGenesisPadding: []string{"B", "A", "A", "A", "A", "A", "A", "A", "A", "A"},
+		},
+		{
+			parents:                          []string{"B"},
+			id:                               "D",
+			expectedWindowWithGenesisPadding: []string{"B", "A", "A", "A", "A", "A", "A", "A", "A", "A"},
+		},
+		{
+			parents:                          []string{"C", "D"},
+			id:                               "E",
+			expectedWindowWithGenesisPadding: []string{"C", "D", "B", "A", "A", "A", "A", "A", "A", "A"},
+		},
+		{
+			parents:                          []string{"C", "D"},
+			id:                               "F",
+			expectedWindowWithGenesisPadding: []string{"C", "D", "B", "A", "A", "A", "A", "A", "A", "A"},
+		},
+		{
+			parents:                          []string{"A"},
+			id:                               "G",
+			expectedWindowWithGenesisPadding: []string{"A", "A", "A", "A", "A", "A", "A", "A", "A", "A"},
+		},
+		{
+			parents:                          []string{"G"},
+			id:                               "H",
+			expectedWindowWithGenesisPadding: []string{"G", "A", "A", "A", "A", "A", "A", "A", "A", "A"},
+		},
+		{
+			parents:                          []string{"H", "F"},
+			id:                               "I",
+			expectedWindowWithGenesisPadding: []string{"F", "C", "D", "B", "A", "A", "A", "A", "A", "A"},
+		},
+		{
+			parents:                          []string{"I"},
+			id:                               "J",
+			expectedWindowWithGenesisPadding: []string{"I", "F", "C", "D", "B", "A", "A", "A", "A", "A"},
+		},
+		{
+			parents:                          []string{"J"},
+			id:                               "K",
+			expectedWindowWithGenesisPadding: []string{"J", "I", "F", "C", "D", "B", "A", "A", "A", "A"},
+		},
+		{
+			parents:                          []string{"K"},
+			id:                               "L",
+			expectedWindowWithGenesisPadding: []string{"K", "J", "I", "F", "C", "D", "B", "A", "A", "A"},
+		},
+		{
+			parents:                          []string{"L"},
+			id:                               "M",
+			expectedWindowWithGenesisPadding: []string{"L", "K", "J", "I", "F", "C", "D", "B", "A", "A"},
+		},
+		{
+			parents:                          []string{"M"},
+			id:                               "N",
+			expectedWindowWithGenesisPadding: []string{"M", "L", "K", "J", "I", "F", "C", "D", "B", "A"},
+		},
+		{
+			parents:                          []string{"N"},
+			id:                               "O",
+			expectedWindowWithGenesisPadding: []string{"N", "M", "L", "K", "J", "I", "F", "C", "D", "B"},
+		},
+	}
+
+	for _, blockData := range blocksData {
+		blockTime = blockTime.Add(time.Second)
+		parents := blockSet{}
+		for _, parentID := range blockData.parents {
+			parent := blockByIDMap[parentID]
+			parents.add(parent)
+		}
+
+		block, err := PrepareBlockForTest(dag, parents.hashes(), nil)
+		if err != nil {
+			t.Fatalf("block %v got unexpected error from PrepareBlockForTest: %v", blockData.id, err)
+		}
+
+		utilBlock := util.NewBlock(block)
+		isOrphan, isDelayed, err := dag.ProcessBlock(utilBlock, BFNoPoWCheck)
+		if err != nil {
+			t.Fatalf("dag.ProcessBlock got unexpected error for block %v: %v", blockData.id, err)
+		}
+		if isDelayed {
+			t.Fatalf("block %s "+
+				"is too far in the future", blockData.id)
+		}
+		if isOrphan {
+			t.Fatalf("block %v was unexpectedly orphan", blockData.id)
+		}
+
+		node := dag.index.LookupNode(utilBlock.Hash())
+
+		blockByIDMap[blockData.id] = node
+		idByBlockMap[node] = blockData.id
+
+		window := blueBlockWindow(node, windowSize)
+		if err := checkWindowIDs(window, blockData.expectedWindowWithGenesisPadding, idByBlockMap); err != nil {
+			t.Errorf("Unexpected values for window for block %s: %s", blockData.id, err)
+		}
+	}
+}
+
+func checkWindowIDs(window []*blockNode, expectedIDs []string, idByBlockMap map[*blockNode]string) error {
+	ids := make([]string, len(window))
+	for i, node := range window {
+		ids[i] = idByBlockMap[node]
+	}
+	if !reflect.DeepEqual(ids, expectedIDs) {
+		return errors.Errorf("window expected to have blocks %s but got %s", expectedIDs, ids)
+	}
+	return nil
+}

blockdag/coinbase.go

@@ -0,0 +1,278 @@
+package blockdag
+
+import (
+	"bufio"
+	"bytes"
+	"encoding/binary"
+	"github.com/kaspanet/kaspad/util/subnetworkid"
+	"github.com/pkg/errors"
+	"io"
+	"math"
+
+	"github.com/kaspanet/kaspad/database"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/util/txsort"
+	"github.com/kaspanet/kaspad/wire"
+)
+
+// compactFeeData is a specialized data type to store a compact list of fees
+// inside a block.
+// Every transaction gets a single uint64 value, stored as a plain binary list.
+// The transactions are ordered the same way they are ordered inside the block, making it easy
+// to traverse every transaction in a block and extract its fee.
+//
+// compactFeeFactory is used to create such a list.
+// compactFeeIterator is used to iterate over such a list.
+
+type compactFeeData []byte
+
+func (cfd compactFeeData) Len() int {
+	return len(cfd) / 8
+}
+
+type compactFeeFactory struct {
+	buffer *bytes.Buffer
+	writer *bufio.Writer
+}
+
+func newCompactFeeFactory() *compactFeeFactory {
+	buffer := bytes.NewBuffer([]byte{})
+	return &compactFeeFactory{
+		buffer: buffer,
+		writer: bufio.NewWriter(buffer),
+	}
+}
+
+func (cfw *compactFeeFactory) add(txFee uint64) error {
+	return binary.Write(cfw.writer, binary.LittleEndian, txFee)
+}
+
+func (cfw *compactFeeFactory) data() (compactFeeData, error) {
+	err := cfw.writer.Flush()
+
+	return compactFeeData(cfw.buffer.Bytes()), err
+}
+
+type compactFeeIterator struct {
+	reader io.Reader
+}
+
+func (cfd compactFeeData) iterator() *compactFeeIterator {
+	return &compactFeeIterator{
+		reader: bufio.NewReader(bytes.NewBuffer(cfd)),
+	}
+}
+
+func (cfr *compactFeeIterator) next() (uint64, error) {
+	var txFee uint64
+
+	err := binary.Read(cfr.reader, binary.LittleEndian, &txFee)
+
+	return txFee, err
+}
+
+// The following functions relate to storing and retrieving fee data from the database
+var feeBucket = []byte("fees")
+
+// getBluesFeeData returns the compactFeeData for all nodes's blues,
+// used to calculate the fees this blockNode needs to pay
+func (node *blockNode) getBluesFeeData(dag *BlockDAG) (map[daghash.Hash]compactFeeData, error) {
+	bluesFeeData := make(map[daghash.Hash]compactFeeData)
+
+	err := dag.db.View(func(dbTx database.Tx) error {
+		for _, blueBlock := range node.blues {
+			feeData, err := dbFetchFeeData(dbTx, blueBlock.hash)
+			if err != nil {
+				return errors.Errorf("Error getting fee data for block %s: %s", blueBlock.hash, err)
+			}
+
+			bluesFeeData[*blueBlock.hash] = feeData
+		}
+
+		return nil
+	})
+	if err != nil {
+		return nil, err
+	}
+
+	return bluesFeeData, nil
+}
+
+func dbStoreFeeData(dbTx database.Tx, blockHash *daghash.Hash, feeData compactFeeData) error {
+	feeBucket, err := dbTx.Metadata().CreateBucketIfNotExists(feeBucket)
+	if err != nil {
+		return errors.Errorf("Error creating or retrieving fee bucket: %s", err)
+	}
+
+	return feeBucket.Put(blockHash.CloneBytes(), feeData)
+}
+
+func dbFetchFeeData(dbTx database.Tx, blockHash *daghash.Hash) (compactFeeData, error) {
+	feeBucket := dbTx.Metadata().Bucket(feeBucket)
+	if feeBucket == nil {
+		return nil, errors.New("Fee bucket does not exist")
+	}
+
+	feeData := feeBucket.Get(blockHash.CloneBytes())
+	if feeData == nil {
+		return nil, errors.Errorf("No fee data found for block %s", blockHash)
+	}
+
+	return feeData, nil
+}
+
+// The following functions deal with building and validating the coinbase transaction
+
+func (node *blockNode) validateCoinbaseTransaction(dag *BlockDAG, block *util.Block, txsAcceptanceData MultiBlockTxsAcceptanceData) error {
+	if node.isGenesis() {
+		return nil
+	}
+	blockCoinbaseTx := block.CoinbaseTransaction().MsgTx()
+	scriptPubKey, extraData, err := DeserializeCoinbasePayload(blockCoinbaseTx)
+	if err != nil {
+		return err
+	}
+	expectedCoinbaseTransaction, err := node.expectedCoinbaseTransaction(dag, txsAcceptanceData, scriptPubKey, extraData)
+	if err != nil {
+		return err
+	}
+
+	if !expectedCoinbaseTransaction.Hash().IsEqual(block.CoinbaseTransaction().Hash()) {
+		return ruleError(ErrBadCoinbaseTransaction, "Coinbase transaction is not built as expected")
+	}
+
+	return nil
+}
+
+// expectedCoinbaseTransaction returns the coinbase transaction for the current block
+func (node *blockNode) expectedCoinbaseTransaction(dag *BlockDAG, txsAcceptanceData MultiBlockTxsAcceptanceData, scriptPubKey []byte, extraData []byte) (*util.Tx, error) {
+	bluesFeeData, err := node.getBluesFeeData(dag)
+	if err != nil {
+		return nil, err
+	}
+
+	txIns := []*wire.TxIn{}
+	txOuts := []*wire.TxOut{}
+
+	for _, blue := range node.blues {
+		txIn, txOut, err := coinbaseInputAndOutputForBlueBlock(dag, blue, txsAcceptanceData, bluesFeeData)
+		if err != nil {
+			return nil, err
+		}
+		txIns = append(txIns, txIn)
+		if txOut != nil {
+			txOuts = append(txOuts, txOut)
+		}
+	}
+	payload, err := SerializeCoinbasePayload(scriptPubKey, extraData)
+	if err != nil {
+		return nil, err
+	}
+	coinbaseTx := wire.NewSubnetworkMsgTx(wire.TxVersion, txIns, txOuts, subnetworkid.SubnetworkIDCoinbase, 0, payload)
+	sortedCoinbaseTx := txsort.Sort(coinbaseTx)
+	return util.NewTx(sortedCoinbaseTx), nil
+}
+
+// SerializeCoinbasePayload builds the coinbase payload based on the provided scriptPubKey and extra data.
+func SerializeCoinbasePayload(scriptPubKey []byte, extraData []byte) ([]byte, error) {
+	w := &bytes.Buffer{}
+	err := wire.WriteVarInt(w, uint64(len(scriptPubKey)))
+	if err != nil {
+		return nil, err
+	}
+	_, err = w.Write(scriptPubKey)
+	if err != nil {
+		return nil, err
+	}
+	_, err = w.Write(extraData)
+	if err != nil {
+		return nil, err
+	}
+	return w.Bytes(), nil
+}
+
+// DeserializeCoinbasePayload deserialize the coinbase payload to its component (scriptPubKey and extra data).
+func DeserializeCoinbasePayload(tx *wire.MsgTx) (scriptPubKey []byte, extraData []byte, err error) {
+	r := bytes.NewReader(tx.Payload)
+	scriptPubKeyLen, err := wire.ReadVarInt(r)
+	if err != nil {
+		return nil, nil, err
+	}
+	scriptPubKey = make([]byte, scriptPubKeyLen)
+	_, err = r.Read(scriptPubKey)
+	if err != nil {
+		return nil, nil, err
+	}
+	extraData = make([]byte, r.Len())
+	if r.Len() != 0 {
+		_, err = r.Read(extraData)
+		if err != nil {
+			return nil, nil, err
+		}
+	}
+	return scriptPubKey, extraData, nil
+}
+
+// feeInputAndOutputForBlueBlock calculates the input and output that should go into the coinbase transaction of blueBlock
+// If blueBlock gets no fee - returns only txIn and nil for txOut
+func coinbaseInputAndOutputForBlueBlock(dag *BlockDAG, blueBlock *blockNode,
+	txsAcceptanceData MultiBlockTxsAcceptanceData, feeData map[daghash.Hash]compactFeeData) (
+	*wire.TxIn, *wire.TxOut, error) {
+
+	blockTxsAcceptanceData, ok := txsAcceptanceData.FindAcceptanceData(blueBlock.hash)
+	if !ok {
+		return nil, nil, errors.Errorf("No txsAcceptanceData for block %s", blueBlock.hash)
+	}
+	blockFeeData, ok := feeData[*blueBlock.hash]
+	if !ok {
+		return nil, nil, errors.Errorf("No feeData for block %s", blueBlock.hash)
+	}
+
+	if len(blockTxsAcceptanceData.TxAcceptanceData) != blockFeeData.Len() {
+		return nil, nil, errors.Errorf(
+			"length of accepted transaction data(%d) and fee data(%d) is not equal for block %s",
+			len(blockTxsAcceptanceData.TxAcceptanceData), blockFeeData.Len(), blueBlock.hash)
+	}
+
+	txIn := &wire.TxIn{
+		SignatureScript: []byte{},
+		PreviousOutpoint: wire.Outpoint{
+			TxID:  daghash.TxID(*blueBlock.hash),
+			Index: math.MaxUint32,
+		},
+		Sequence: wire.MaxTxInSequenceNum,
+	}
+
+	totalFees := uint64(0)
+	feeIterator := blockFeeData.iterator()
+
+	for _, txAcceptanceData := range blockTxsAcceptanceData.TxAcceptanceData {
+		fee, err := feeIterator.next()
+		if err != nil {
+			return nil, nil, errors.Errorf("Error retrieving fee from compactFeeData iterator: %s", err)
+		}
+		if txAcceptanceData.IsAccepted {
+			totalFees += fee
+		}
+	}
+
+	totalReward := CalcBlockSubsidy(blueBlock.blueScore, dag.dagParams) + totalFees
+
+	if totalReward == 0 {
+		return txIn, nil, nil
+	}
+
+	// the ScriptPubKey for the coinbase is parsed from the coinbase payload
+	scriptPubKey, _, err := DeserializeCoinbasePayload(blockTxsAcceptanceData.TxAcceptanceData[0].Tx.MsgTx())
+	if err != nil {
+		return nil, nil, err
+	}
+
+	txOut := &wire.TxOut{
+		Value:        totalReward,
+		ScriptPubKey: scriptPubKey,
+	}
+
+	return txIn, txOut, nil
+}

blockdag/coinbase_test.go

@@ -0,0 +1,60 @@
+package blockdag
+
+import (
+	"io"
+	"reflect"
+	"testing"
+)
+
+func TestFeeAccumulators(t *testing.T) {
+	fees := []uint64{1, 2, 3, 4, 5, 6, 7, 0xffffffffffffffff}
+
+	factory := newCompactFeeFactory()
+
+	for _, fee := range fees {
+		err := factory.add(fee)
+		if err != nil {
+			t.Fatalf("Error writing %d as tx fee: %s", fee, err)
+		}
+	}
+
+	expectedData := compactFeeData{
+		1, 0, 0, 0, 0, 0, 0, 0,
+		2, 0, 0, 0, 0, 0, 0, 0,
+		3, 0, 0, 0, 0, 0, 0, 0,
+		4, 0, 0, 0, 0, 0, 0, 0,
+		5, 0, 0, 0, 0, 0, 0, 0,
+		6, 0, 0, 0, 0, 0, 0, 0,
+		7, 0, 0, 0, 0, 0, 0, 0,
+		255, 255, 255, 255, 255, 255, 255, 255,
+	}
+	actualData, err := factory.data()
+
+	if err != nil {
+		t.Fatalf("Error getting bytes from writer: %s", err)
+	}
+	if !reflect.DeepEqual(expectedData, actualData) {
+		t.Errorf("Expected bytes: %v, but got: %v", expectedData, actualData)
+	}
+
+	iterator := actualData.iterator()
+
+	for i, expectedFee := range fees {
+		actualFee, err := iterator.next()
+		if err != nil {
+			t.Fatalf("Error getting fee for Tx#%d: %s", i, err)
+		}
+
+		if actualFee != expectedFee {
+			t.Errorf("Tx #%d: Expected fee: %d, but got %d", i, expectedFee, actualFee)
+		}
+	}
+
+	_, err = iterator.next()
+	if err == nil {
+		t.Fatal("No error from iterator.nextTxFee after done reading all transactions")
+	}
+	if err != io.EOF {
+		t.Fatalf("Error from iterator.nextTxFee after done reading all transactions is not io.EOF: %s", err)
+	}
+}

blockdag/common_test.go

@@ -0,0 +1,224 @@
+// Copyright (c) 2013-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"compress/bzip2"
+	"encoding/binary"
+	"github.com/pkg/errors"
+	"io"
+	"os"
+	"path/filepath"
+	"reflect"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/kaspanet/kaspad/dagconfig"
+	_ "github.com/kaspanet/kaspad/database/ffldb"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/wire"
+)
+
+func loadBlocksWithLog(t *testing.T, filename string) ([]*util.Block, error) {
+	blocks, err := LoadBlocks(filename)
+	if err == nil {
+		t.Logf("Loaded %d blocks from file %s", len(blocks), filename)
+		for i, b := range blocks {
+			t.Logf("Block #%d: %s", i, b.Hash())
+		}
+	}
+	return blocks, err
+}
+
+// loadUTXOSet returns a utxo view loaded from a file.
+func loadUTXOSet(filename string) (UTXOSet, error) {
+	// The utxostore file format is:
+	// <tx hash><output index><serialized utxo len><serialized utxo>
+	//
+	// The output index and serialized utxo len are little endian uint32s
+	// and the serialized utxo uses the format described in dagio.go.
+
+	filename = filepath.Join("testdata", filename)
+	fi, err := os.Open(filename)
+	if err != nil {
+		return nil, err
+	}
+
+	// Choose read based on whether the file is compressed or not.
+	var r io.Reader
+	if strings.HasSuffix(filename, ".bz2") {
+		r = bzip2.NewReader(fi)
+	} else {
+		r = fi
+	}
+	defer fi.Close()
+
+	utxoSet := NewFullUTXOSet()
+	for {
+		// Tx ID of the utxo entry.
+		var txID daghash.TxID
+		_, err := io.ReadAtLeast(r, txID[:], len(txID[:]))
+		if err != nil {
+			// Expected EOF at the right offset.
+			if err == io.EOF {
+				break
+			}
+			return nil, err
+		}
+
+		// Output index of the utxo entry.
+		var index uint32
+		err = binary.Read(r, binary.LittleEndian, &index)
+		if err != nil {
+			return nil, err
+		}
+
+		// Num of serialized utxo entry bytes.
+		var numBytes uint32
+		err = binary.Read(r, binary.LittleEndian, &numBytes)
+		if err != nil {
+			return nil, err
+		}
+
+		// Serialized utxo entry.
+		serialized := make([]byte, numBytes)
+		_, err = io.ReadAtLeast(r, serialized, int(numBytes))
+		if err != nil {
+			return nil, err
+		}
+
+		// Deserialize it and add it to the view.
+		entry, err := deserializeUTXOEntry(serialized)
+		if err != nil {
+			return nil, err
+		}
+		utxoSet.utxoCollection[wire.Outpoint{TxID: txID, Index: index}] = entry
+	}
+
+	return utxoSet, nil
+}
+
+// TestSetCoinbaseMaturity makes the ability to set the coinbase maturity
+// available when running tests.
+func (dag *BlockDAG) TestSetCoinbaseMaturity(maturity uint64) {
+	dag.dagParams.BlockCoinbaseMaturity = maturity
+}
+
+// newTestDAG returns a DAG that is usable for syntetic tests. It is
+// important to note that this DAG has no database associated with it, so
+// it is not usable with all functions and the tests must take care when making
+// use of it.
+func newTestDAG(params *dagconfig.Params) *BlockDAG {
+	index := newBlockIndex(nil, params)
+	targetTimePerBlock := int64(params.TargetTimePerBlock / time.Second)
+	dag := &BlockDAG{
+		dagParams:                      params,
+		timeSource:                     NewMedianTime(),
+		targetTimePerBlock:             targetTimePerBlock,
+		difficultyAdjustmentWindowSize: params.DifficultyAdjustmentWindowSize,
+		TimestampDeviationTolerance:    params.TimestampDeviationTolerance,
+		powMaxBits:                     util.BigToCompact(params.PowMax),
+		index:                          index,
+		warningCaches:                  newThresholdCaches(vbNumBits),
+		deploymentCaches:               newThresholdCaches(dagconfig.DefinedDeployments),
+	}
+
+	// Create a genesis block node and block index index populated with it
+	// on the above fake DAG.
+	dag.genesis, _ = dag.newBlockNode(&params.GenesisBlock.Header, newSet())
+	index.AddNode(dag.genesis)
+
+	dag.virtual = newVirtualBlock(dag, setFromSlice(dag.genesis))
+	return dag
+}
+
+// newTestNode creates a block node connected to the passed parent with the
+// provided fields populated and fake values for the other fields.
+func newTestNode(dag *BlockDAG, parents blockSet, blockVersion int32, bits uint32, timestamp time.Time) *blockNode {
+	// Make up a header and create a block node from it.
+	header := &wire.BlockHeader{
+		Version:              blockVersion,
+		ParentHashes:         parents.hashes(),
+		Bits:                 bits,
+		Timestamp:            timestamp,
+		HashMerkleRoot:       &daghash.ZeroHash,
+		AcceptedIDMerkleRoot: &daghash.ZeroHash,
+		UTXOCommitment:       &daghash.ZeroHash,
+	}
+	node, _ := dag.newBlockNode(header, parents)
+	return node
+}
+
+func addNodeAsChildToParents(node *blockNode) {
+	for parent := range node.parents {
+		parent.children.add(node)
+	}
+}
+
+// checkRuleError ensures the type of the two passed errors are of the
+// same type (either both nil or both of type RuleError) and their error codes
+// match when not nil.
+func checkRuleError(gotErr, wantErr error) error {
+	// Ensure the error code is of the expected type and the error
+	// code matches the value specified in the test instance.
+	if reflect.TypeOf(gotErr) != reflect.TypeOf(wantErr) {
+		return errors.Errorf("wrong error - got %T (%[1]v), want %T",
+			gotErr, wantErr)
+	}
+	if gotErr == nil {
+		return nil
+	}
+
+	// Ensure the want error type is a script error.
+	werr, ok := wantErr.(RuleError)
+	if !ok {
+		return errors.Errorf("unexpected test error type %T", wantErr)
+	}
+
+	// Ensure the error codes match. It's safe to use a raw type assert
+	// here since the code above already proved they are the same type and
+	// the want error is a script error.
+	gotErrorCode := gotErr.(RuleError).ErrorCode
+	if gotErrorCode != werr.ErrorCode {
+		return errors.Errorf("mismatched error code - got %v (%v), want %v",
+			gotErrorCode, gotErr, werr.ErrorCode)
+	}
+
+	return nil
+}
+
+func prepareAndProcessBlock(t *testing.T, dag *BlockDAG, parents ...*wire.MsgBlock) *wire.MsgBlock {
+	parentHashes := make([]*daghash.Hash, len(parents))
+	for i, parent := range parents {
+		parentHashes[i] = parent.BlockHash()
+	}
+	daghash.Sort(parentHashes)
+	block, err := PrepareBlockForTest(dag, parentHashes, nil)
+	if err != nil {
+		t.Fatalf("error in PrepareBlockForTest: %s", err)
+	}
+	utilBlock := util.NewBlock(block)
+	isOrphan, isDelayed, err := dag.ProcessBlock(utilBlock, BFNoPoWCheck)
+	if err != nil {
+		t.Fatalf("unexpected error in ProcessBlock: %s", err)
+	}
+	if isDelayed {
+		t.Fatalf("block is too far in the future")
+	}
+	if isOrphan {
+		t.Fatalf("block was unexpectedly orphan")
+	}
+	return block
+}
+
+func nodeByMsgBlock(t *testing.T, dag *BlockDAG, block *wire.MsgBlock) *blockNode {
+	node := dag.index.LookupNode(block.BlockHash())
+	if node == nil {
+		t.Fatalf("couldn't find block node with hash %s", block.BlockHash())
+	}
+	return node
+}

blockdag/compress.go

@@ -0,0 +1,584 @@
+// Copyright (c) 2015-2016 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"github.com/kaspanet/kaspad/ecc"
+	"github.com/kaspanet/kaspad/txscript"
+)
+
+// -----------------------------------------------------------------------------
+// A variable length quantity (VLQ) is an encoding that uses an arbitrary number
+// of binary octets to represent an arbitrarily large integer. The scheme
+// employs a most significant byte (MSB) base-128 encoding where the high bit in
+// each byte indicates whether or not the byte is the final one. In addition,
+// to ensure there are no redundant encodings, an offset is subtracted every
+// time a group of 7 bits is shifted out. Therefore each integer can be
+// represented in exactly one way, and each representation stands for exactly
+// one integer.
+//
+// Another nice property of this encoding is that it provides a compact
+// representation of values that are typically used to indicate sizes. For
+// example, the values 0 - 127 are represented with a single byte, 128 - 16511
+// with two bytes, and 16512 - 2113663 with three bytes.
+//
+// While the encoding allows arbitrarily large integers, it is artificially
+// limited in this code to an unsigned 64-bit integer for efficiency purposes.
+//
+// Example encodings:
+//           0 -> [0x00]
+//         127 -> [0x7f]                 * Max 1-byte value
+//         128 -> [0x80 0x00]
+//         129 -> [0x80 0x01]
+//         255 -> [0x80 0x7f]
+//         256 -> [0x81 0x00]
+//       16511 -> [0xff 0x7f]            * Max 2-byte value
+//       16512 -> [0x80 0x80 0x00]
+//       32895 -> [0x80 0xff 0x7f]
+//     2113663 -> [0xff 0xff 0x7f]       * Max 3-byte value
+//   270549119 -> [0xff 0xff 0xff 0x7f]  * Max 4-byte value
+//      2^64-1 -> [0x80 0xfe 0xfe 0xfe 0xfe 0xfe 0xfe 0xfe 0xfe 0x7f]
+//
+// References:
+//   https://en.wikipedia.org/wiki/Variable-length_quantity
+//   http://www.codecodex.com/wiki/Variable-Length_Integers
+// -----------------------------------------------------------------------------
+
+// serializeSizeVLQ returns the number of bytes it would take to serialize the
+// passed number as a variable-length quantity according to the format described
+// above.
+func serializeSizeVLQ(n uint64) int {
+	size := 1
+	for ; n > 0x7f; n = (n >> 7) - 1 {
+		size++
+	}
+
+	return size
+}
+
+// putVLQ serializes the provided number to a variable-length quantity according
+// to the format described above and returns the number of bytes of the encoded
+// value. The result is placed directly into the passed byte slice which must
+// be at least large enough to handle the number of bytes returned by the
+// serializeSizeVLQ function or it will panic.
+func putVLQ(target []byte, n uint64) int {
+	offset := 0
+	for ; ; offset++ {
+		// The high bit is set when another byte follows.
+		highBitMask := byte(0x80)
+		if offset == 0 {
+			highBitMask = 0x00
+		}
+
+		target[offset] = byte(n&0x7f) | highBitMask
+		if n <= 0x7f {
+			break
+		}
+		n = (n >> 7) - 1
+	}
+
+	// Reverse the bytes so it is MSB-encoded.
+	for i, j := 0, offset; i < j; i, j = i+1, j-1 {
+		target[i], target[j] = target[j], target[i]
+	}
+
+	return offset + 1
+}
+
+// deserializeVLQ deserializes the provided variable-length quantity according
+// to the format described above. It also returns the number of bytes
+// deserialized.
+func deserializeVLQ(serialized []byte) (uint64, int) {
+	var n uint64
+	var size int
+	for _, val := range serialized {
+		size++
+		n = (n << 7) | uint64(val&0x7f)
+		if val&0x80 != 0x80 {
+			break
+		}
+		n++
+	}
+
+	return n, size
+}
+
+// -----------------------------------------------------------------------------
+// In order to reduce the size of stored scripts, a domain specific compression
+// algorithm is used which recognizes standard scripts and stores them using
+// less bytes than the original script.
+//
+// The general serialized format is:
+//
+//   <script size or type><script data>
+//
+//   Field                 Type     Size
+//   script size or type   VLQ      variable
+//   script data           []byte   variable
+//
+// The specific serialized format for each recognized standard script is:
+//
+// - Pay-to-pubkey-hash: (21 bytes) - <0><20-byte pubkey hash>
+// - Pay-to-script-hash: (21 bytes) - <1><20-byte script hash>
+// - Pay-to-pubkey**:    (33 bytes) - <2, 3, 4, or 5><32-byte pubkey X value>
+//   2, 3 = compressed pubkey with bit 0 specifying the y coordinate to use
+//   4, 5 = uncompressed pubkey with bit 0 specifying the y coordinate to use
+//   ** Only valid public keys starting with 0x02, 0x03, and 0x04 are supported.
+//
+// Any scripts which are not recognized as one of the aforementioned standard
+// scripts are encoded using the general serialized format and encode the script
+// size as the sum of the actual size of the script and the number of special
+// cases.
+// -----------------------------------------------------------------------------
+
+// The following constants specify the special constants used to identify a
+// special script type in the domain-specific compressed script encoding.
+//
+// NOTE: This section specifically does not use iota since these values are
+// serialized and must be stable for long-term storage.
+const (
+	// cstPayToPubKeyHash identifies a compressed pay-to-pubkey-hash script.
+	cstPayToPubKeyHash = 0
+
+	// cstPayToScriptHash identifies a compressed pay-to-script-hash script.
+	cstPayToScriptHash = 1
+
+	// cstPayToPubKeyComp2 identifies a compressed pay-to-pubkey script to
+	// a compressed pubkey. Bit 0 specifies which y-coordinate to use
+	// to reconstruct the full uncompressed pubkey.
+	cstPayToPubKeyComp2 = 2
+
+	// cstPayToPubKeyComp3 identifies a compressed pay-to-pubkey script to
+	// a compressed pubkey. Bit 0 specifies which y-coordinate to use
+	// to reconstruct the full uncompressed pubkey.
+	cstPayToPubKeyComp3 = 3
+
+	// cstPayToPubKeyUncomp4 identifies a compressed pay-to-pubkey script to
+	// an uncompressed pubkey. Bit 0 specifies which y-coordinate to use
+	// to reconstruct the full uncompressed pubkey.
+	cstPayToPubKeyUncomp4 = 4
+
+	// cstPayToPubKeyUncomp5 identifies a compressed pay-to-pubkey script to
+	// an uncompressed pubkey. Bit 0 specifies which y-coordinate to use
+	// to reconstruct the full uncompressed pubkey.
+	cstPayToPubKeyUncomp5 = 5
+
+	// numSpecialScripts is the number of special scripts recognized by the
+	// domain-specific script compression algorithm.
+	numSpecialScripts = 6
+)
+
+// isPubKeyHash returns whether or not the passed public key script is a
+// standard pay-to-pubkey-hash script along with the pubkey hash it is paying to
+// if it is.
+func isPubKeyHash(script []byte) (bool, []byte) {
+	if len(script) == 25 && script[0] == txscript.OpDup &&
+		script[1] == txscript.OpHash160 &&
+		script[2] == txscript.OpData20 &&
+		script[23] == txscript.OpEqualVerify &&
+		script[24] == txscript.OpCheckSig {
+
+		return true, script[3:23]
+	}
+
+	return false, nil
+}
+
+// isScriptHash returns whether or not the passed public key script is a
+// standard pay-to-script-hash script along with the script hash it is paying to
+// if it is.
+func isScriptHash(script []byte) (bool, []byte) {
+	if len(script) == 23 && script[0] == txscript.OpHash160 &&
+		script[1] == txscript.OpData20 &&
+		script[22] == txscript.OpEqual {
+
+		return true, script[2:22]
+	}
+
+	return false, nil
+}
+
+// isPubKey returns whether or not the passed public key script is a standard
+// pay-to-pubkey script that pays to a valid compressed or uncompressed public
+// key along with the serialized pubkey it is paying to if it is.
+//
+// NOTE: This function ensures the public key is actually valid since the
+// compression algorithm requires valid pubkeys. It does not support hybrid
+// pubkeys. This means that even if the script has the correct form for a
+// pay-to-pubkey script, this function will only return true when it is paying
+// to a valid compressed or uncompressed pubkey.
+func isPubKey(script []byte) (bool, []byte) {
+	// Pay-to-compressed-pubkey script.
+	if len(script) == 35 && script[0] == txscript.OpData33 &&
+		script[34] == txscript.OpCheckSig && (script[1] == 0x02 ||
+		script[1] == 0x03) {
+
+		// Ensure the public key is valid.
+		serializedPubKey := script[1:34]
+		_, err := ecc.ParsePubKey(serializedPubKey, ecc.S256())
+		if err == nil {
+			return true, serializedPubKey
+		}
+	}
+
+	// Pay-to-uncompressed-pubkey script.
+	if len(script) == 67 && script[0] == txscript.OpData65 &&
+		script[66] == txscript.OpCheckSig && script[1] == 0x04 {
+
+		// Ensure the public key is valid.
+		serializedPubKey := script[1:66]
+		_, err := ecc.ParsePubKey(serializedPubKey, ecc.S256())
+		if err == nil {
+			return true, serializedPubKey
+		}
+	}
+
+	return false, nil
+}
+
+// compressedScriptSize returns the number of bytes the passed script would take
+// when encoded with the domain specific compression algorithm described above.
+func compressedScriptSize(scriptPubKey []byte) int {
+	// Pay-to-pubkey-hash script.
+	if valid, _ := isPubKeyHash(scriptPubKey); valid {
+		return 21
+	}
+
+	// Pay-to-script-hash script.
+	if valid, _ := isScriptHash(scriptPubKey); valid {
+		return 21
+	}
+
+	// Pay-to-pubkey (compressed or uncompressed) script.
+	if valid, _ := isPubKey(scriptPubKey); valid {
+		return 33
+	}
+
+	// When none of the above special cases apply, encode the script as is
+	// preceded by the sum of its size and the number of special cases
+	// encoded as a variable length quantity.
+	return serializeSizeVLQ(uint64(len(scriptPubKey)+numSpecialScripts)) +
+		len(scriptPubKey)
+}
+
+// decodeCompressedScriptSize treats the passed serialized bytes as a compressed
+// script, possibly followed by other data, and returns the number of bytes it
+// occupies taking into account the special encoding of the script size by the
+// domain specific compression algorithm described above.
+func decodeCompressedScriptSize(serialized []byte) int {
+	scriptSize, bytesRead := deserializeVLQ(serialized)
+	if bytesRead == 0 {
+		return 0
+	}
+
+	switch scriptSize {
+	case cstPayToPubKeyHash:
+		return 21
+
+	case cstPayToScriptHash:
+		return 21
+
+	case cstPayToPubKeyComp2, cstPayToPubKeyComp3, cstPayToPubKeyUncomp4,
+		cstPayToPubKeyUncomp5:
+		return 33
+	}
+
+	scriptSize -= numSpecialScripts
+	scriptSize += uint64(bytesRead)
+	return int(scriptSize)
+}
+
+// putCompressedScript compresses the passed script according to the domain
+// specific compression algorithm described above directly into the passed
+// target byte slice. The target byte slice must be at least large enough to
+// handle the number of bytes returned by the compressedScriptSize function or
+// it will panic.
+func putCompressedScript(target, scriptPubKey []byte) int {
+	// Pay-to-pubkey-hash script.
+	if valid, hash := isPubKeyHash(scriptPubKey); valid {
+		target[0] = cstPayToPubKeyHash
+		copy(target[1:21], hash)
+		return 21
+	}
+
+	// Pay-to-script-hash script.
+	if valid, hash := isScriptHash(scriptPubKey); valid {
+		target[0] = cstPayToScriptHash
+		copy(target[1:21], hash)
+		return 21
+	}
+
+	// Pay-to-pubkey (compressed or uncompressed) script.
+	if valid, serializedPubKey := isPubKey(scriptPubKey); valid {
+		pubKeyFormat := serializedPubKey[0]
+		switch pubKeyFormat {
+		case 0x02, 0x03:
+			target[0] = pubKeyFormat
+			copy(target[1:33], serializedPubKey[1:33])
+			return 33
+		case 0x04:
+			// Encode the oddness of the serialized pubkey into the
+			// compressed script type.
+			target[0] = pubKeyFormat | (serializedPubKey[64] & 0x01)
+			copy(target[1:33], serializedPubKey[1:33])
+			return 33
+		}
+	}
+
+	// When none of the above special cases apply, encode the unmodified
+	// script preceded by the sum of its size and the number of special
+	// cases encoded as a variable length quantity.
+	encodedSize := uint64(len(scriptPubKey) + numSpecialScripts)
+	vlqSizeLen := putVLQ(target, encodedSize)
+	copy(target[vlqSizeLen:], scriptPubKey)
+	return vlqSizeLen + len(scriptPubKey)
+}
+
+// decompressScript returns the original script obtained by decompressing the
+// passed compressed script according to the domain specific compression
+// algorithm described above.
+//
+// NOTE: The script parameter must already have been proven to be long enough
+// to contain the number of bytes returned by decodeCompressedScriptSize or it
+// will panic. This is acceptable since it is only an internal function.
+func decompressScript(compressedScriptPubKey []byte) []byte {
+	// In practice this function will not be called with a zero-length or
+	// nil script since the nil script encoding includes the length, however
+	// the code below assumes the length exists, so just return nil now if
+	// the function ever ends up being called with a nil script in the
+	// future.
+	if len(compressedScriptPubKey) == 0 {
+		return nil
+	}
+
+	// Decode the script size and examine it for the special cases.
+	encodedScriptSize, bytesRead := deserializeVLQ(compressedScriptPubKey)
+	switch encodedScriptSize {
+	// Pay-to-pubkey-hash script. The resulting script is:
+	// <OP_DUP><OP_HASH160><20 byte hash><OP_EQUALVERIFY><OP_CHECKSIG>
+	case cstPayToPubKeyHash:
+		scriptPubKey := make([]byte, 25)
+		scriptPubKey[0] = txscript.OpDup
+		scriptPubKey[1] = txscript.OpHash160
+		scriptPubKey[2] = txscript.OpData20
+		copy(scriptPubKey[3:], compressedScriptPubKey[bytesRead:bytesRead+20])
+		scriptPubKey[23] = txscript.OpEqualVerify
+		scriptPubKey[24] = txscript.OpCheckSig
+		return scriptPubKey
+
+	// Pay-to-script-hash script. The resulting script is:
+	// <OP_HASH160><20 byte script hash><OP_EQUAL>
+	case cstPayToScriptHash:
+		scriptPubKey := make([]byte, 23)
+		scriptPubKey[0] = txscript.OpHash160
+		scriptPubKey[1] = txscript.OpData20
+		copy(scriptPubKey[2:], compressedScriptPubKey[bytesRead:bytesRead+20])
+		scriptPubKey[22] = txscript.OpEqual
+		return scriptPubKey
+
+	// Pay-to-compressed-pubkey script. The resulting script is:
+	// <OP_DATA_33><33 byte compressed pubkey><OP_CHECKSIG>
+	case cstPayToPubKeyComp2, cstPayToPubKeyComp3:
+		scriptPubKey := make([]byte, 35)
+		scriptPubKey[0] = txscript.OpData33
+		scriptPubKey[1] = byte(encodedScriptSize)
+		copy(scriptPubKey[2:], compressedScriptPubKey[bytesRead:bytesRead+32])
+		scriptPubKey[34] = txscript.OpCheckSig
+		return scriptPubKey
+
+	// Pay-to-uncompressed-pubkey script. The resulting script is:
+	// <OP_DATA_65><65 byte uncompressed pubkey><OP_CHECKSIG>
+	case cstPayToPubKeyUncomp4, cstPayToPubKeyUncomp5:
+		// Change the leading byte to the appropriate compressed pubkey
+		// identifier (0x02 or 0x03) so it can be decoded as a
+		// compressed pubkey. This really should never fail since the
+		// encoding ensures it is valid before compressing to this type.
+		compressedKey := make([]byte, 33)
+		compressedKey[0] = byte(encodedScriptSize - 2)
+		copy(compressedKey[1:], compressedScriptPubKey[1:])
+		key, err := ecc.ParsePubKey(compressedKey, ecc.S256())
+		if err != nil {
+			return nil
+		}
+
+		scriptPubKey := make([]byte, 67)
+		scriptPubKey[0] = txscript.OpData65
+		copy(scriptPubKey[1:], key.SerializeUncompressed())
+		scriptPubKey[66] = txscript.OpCheckSig
+		return scriptPubKey
+	}
+
+	// When none of the special cases apply, the script was encoded using
+	// the general format, so reduce the script size by the number of
+	// special cases and return the unmodified script.
+	scriptSize := int(encodedScriptSize - numSpecialScripts)
+	scriptPubKey := make([]byte, scriptSize)
+	copy(scriptPubKey, compressedScriptPubKey[bytesRead:bytesRead+scriptSize])
+	return scriptPubKey
+}
+
+// -----------------------------------------------------------------------------
+// In order to reduce the size of stored amounts, a domain specific compression
+// algorithm is used which relies on there typically being a lot of zeroes at
+// end of the amounts.
+//
+// While this is simply exchanging one uint64 for another, the resulting value
+// for typical amounts has a much smaller magnitude which results in fewer bytes
+// when encoded as variable length quantity. For example, consider the amount
+// of 0.1 KAS which is 10000000 sompi. Encoding 10000000 as a VLQ would take
+// 4 bytes while encoding the compressed value of 8 as a VLQ only takes 1 byte.
+//
+// Essentially the compression is achieved by splitting the value into an
+// exponent in the range [0-9] and a digit in the range [1-9], when possible,
+// and encoding them in a way that can be decoded. More specifically, the
+// encoding is as follows:
+// - 0 is 0
+// - Find the exponent, e, as the largest power of 10 that evenly divides the
+//   value up to a maximum of 9
+// - When e < 9, the final digit can't be 0 so store it as d and remove it by
+//   dividing the value by 10 (call the result n). The encoded value is thus:
+//   1 + 10*(9*n + d-1) + e
+// - When e==9, the only thing known is the amount is not 0. The encoded value
+//   is thus:
+//   1 + 10*(n-1) + e   ==   10 + 10*(n-1)
+//
+// Example encodings:
+// (The numbers in parenthesis are the number of bytes when serialized as a VLQ)
+//            0 (1) -> 0        (1)           *  0.00000000 KAS
+//         1000 (2) -> 4        (1)           *  0.00001000 KAS
+//        10000 (2) -> 5        (1)           *  0.00010000 KAS
+//     12345678 (4) -> 111111101(4)           *  0.12345678 KAS
+//     50000000 (4) -> 47       (1)           *  0.50000000 KAS
+//    100000000 (4) -> 9        (1)           *  1.00000000 KAS
+//    500000000 (5) -> 49       (1)           *  5.00000000 KAS
+//   1000000000 (5) -> 10       (1)           * 10.00000000 KAS
+// -----------------------------------------------------------------------------
+
+// compressTxOutAmount compresses the passed amount according to the domain
+// specific compression algorithm described above.
+func compressTxOutAmount(amount uint64) uint64 {
+	// No need to do any work if it's zero.
+	if amount == 0 {
+		return 0
+	}
+
+	// Find the largest power of 10 (max of 9) that evenly divides the
+	// value.
+	exponent := uint64(0)
+	for amount%10 == 0 && exponent < 9 {
+		amount /= 10
+		exponent++
+	}
+
+	// The compressed result for exponents less than 9 is:
+	// 1 + 10*(9*n + d-1) + e
+	if exponent < 9 {
+		lastDigit := amount % 10
+		amount /= 10
+		return 1 + 10*(9*amount+lastDigit-1) + exponent
+	}
+
+	// The compressed result for an exponent of 9 is:
+	// 1 + 10*(n-1) + e   ==   10 + 10*(n-1)
+	return 10 + 10*(amount-1)
+}
+
+// decompressTxOutAmount returns the original amount the passed compressed
+// amount represents according to the domain specific compression algorithm
+// described above.
+func decompressTxOutAmount(amount uint64) uint64 {
+	// No need to do any work if it's zero.
+	if amount == 0 {
+		return 0
+	}
+
+	// The decompressed amount is either of the following two equations:
+	// x = 1 + 10*(9*n + d - 1) + e
+	// x = 1 + 10*(n - 1)       + 9
+	amount--
+
+	// The decompressed amount is now one of the following two equations:
+	// x = 10*(9*n + d - 1) + e
+	// x = 10*(n - 1)       + 9
+	exponent := amount % 10
+	amount /= 10
+
+	// The decompressed amount is now one of the following two equations:
+	// x = 9*n + d - 1  | where e < 9
+	// x = n - 1        | where e = 9
+	n := uint64(0)
+	if exponent < 9 {
+		lastDigit := amount%9 + 1
+		amount /= 9
+		n = amount*10 + lastDigit
+	} else {
+		n = amount + 1
+	}
+
+	// Apply the exponent.
+	for ; exponent > 0; exponent-- {
+		n *= 10
+	}
+
+	return n
+}
+
+// -----------------------------------------------------------------------------
+// Compressed transaction outputs consist of an amount and a public key script
+// both compressed using the domain specific compression algorithms previously
+// described.
+//
+// The serialized format is:
+//
+//   <compressed amount><compressed script>
+//
+//   Field                 Type     Size
+//     compressed amount   VLQ      variable
+//     compressed script   []byte   variable
+// -----------------------------------------------------------------------------
+
+// compressedTxOutSize returns the number of bytes the passed transaction output
+// fields would take when encoded with the format described above.
+func compressedTxOutSize(amount uint64, scriptPubKey []byte) int {
+	return serializeSizeVLQ(compressTxOutAmount(amount)) +
+		compressedScriptSize(scriptPubKey)
+}
+
+// putCompressedTxOut compresses the passed amount and script according to their
+// domain specific compression algorithms and encodes them directly into the
+// passed target byte slice with the format described above. The target byte
+// slice must be at least large enough to handle the number of bytes returned by
+// the compressedTxOutSize function or it will panic.
+func putCompressedTxOut(target []byte, amount uint64, scriptPubKey []byte) int {
+	offset := putVLQ(target, compressTxOutAmount(amount))
+	offset += putCompressedScript(target[offset:], scriptPubKey)
+	return offset
+}
+
+// decodeCompressedTxOut decodes the passed compressed txout, possibly followed
+// by other data, into its uncompressed amount and script and returns them along
+// with the number of bytes they occupied prior to decompression.
+func decodeCompressedTxOut(serialized []byte) (uint64, []byte, int, error) {
+	// Deserialize the compressed amount and ensure there are bytes
+	// remaining for the compressed script.
+	compressedAmount, bytesRead := deserializeVLQ(serialized)
+	if bytesRead >= len(serialized) {
+		return 0, nil, bytesRead, errDeserialize("unexpected end of " +
+			"data after compressed amount")
+	}
+
+	// Decode the compressed script size and ensure there are enough bytes
+	// left in the slice for it.
+	scriptSize := decodeCompressedScriptSize(serialized[bytesRead:])
+	if len(serialized[bytesRead:]) < scriptSize {
+		return 0, nil, bytesRead, errDeserialize("unexpected end of " +
+			"data after script size")
+	}
+
+	// Decompress and return the amount and script.
+	amount := decompressTxOutAmount(compressedAmount)
+	script := decompressScript(serialized[bytesRead : bytesRead+scriptSize])
+	return amount, script, bytesRead + scriptSize, nil
+}

blockdag/compress_test.go

@@ -0,0 +1,436 @@
+// Copyright (c) 2015-2016 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"bytes"
+	"encoding/hex"
+	"testing"
+)
+
+// hexToBytes converts the passed hex string into bytes and will panic if there
+// is an error. This is only provided for the hard-coded constants so errors in
+// the source code can be detected. It will only (and must only) be called with
+// hard-coded values.
+func hexToBytes(s string) []byte {
+	b, err := hex.DecodeString(s)
+	if err != nil {
+		panic("invalid hex in source file: " + s)
+	}
+	return b
+}
+
+// TestVLQ ensures the variable length quantity serialization, deserialization,
+// and size calculation works as expected.
+func TestVLQ(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		val        uint64
+		serialized []byte
+	}{
+		{0, hexToBytes("00")},
+		{1, hexToBytes("01")},
+		{127, hexToBytes("7f")},
+		{128, hexToBytes("8000")},
+		{129, hexToBytes("8001")},
+		{255, hexToBytes("807f")},
+		{256, hexToBytes("8100")},
+		{16383, hexToBytes("fe7f")},
+		{16384, hexToBytes("ff00")},
+		{16511, hexToBytes("ff7f")}, // Max 2-byte value
+		{16512, hexToBytes("808000")},
+		{16513, hexToBytes("808001")},
+		{16639, hexToBytes("80807f")},
+		{32895, hexToBytes("80ff7f")},
+		{2113663, hexToBytes("ffff7f")}, // Max 3-byte value
+		{2113664, hexToBytes("80808000")},
+		{270549119, hexToBytes("ffffff7f")}, // Max 4-byte value
+		{270549120, hexToBytes("8080808000")},
+		{2147483647, hexToBytes("86fefefe7f")},
+		{2147483648, hexToBytes("86fefeff00")},
+		{4294967295, hexToBytes("8efefefe7f")}, // Max uint32, 5 bytes
+		// Max uint64, 10 bytes
+		{18446744073709551615, hexToBytes("80fefefefefefefefe7f")},
+	}
+
+	for _, test := range tests {
+		// Ensure the function to calculate the serialized size without
+		// actually serializing the value is calculated properly.
+		gotSize := serializeSizeVLQ(test.val)
+		if gotSize != len(test.serialized) {
+			t.Errorf("serializeSizeVLQ: did not get expected size "+
+				"for %d - got %d, want %d", test.val, gotSize,
+				len(test.serialized))
+			continue
+		}
+
+		// Ensure the value serializes to the expected bytes.
+		gotBytes := make([]byte, gotSize)
+		gotBytesWritten := putVLQ(gotBytes, test.val)
+		if !bytes.Equal(gotBytes, test.serialized) {
+			t.Errorf("putVLQUnchecked: did not get expected bytes "+
+				"for %d - got %x, want %x", test.val, gotBytes,
+				test.serialized)
+			continue
+		}
+		if gotBytesWritten != len(test.serialized) {
+			t.Errorf("putVLQUnchecked: did not get expected number "+
+				"of bytes written for %d - got %d, want %d",
+				test.val, gotBytesWritten, len(test.serialized))
+			continue
+		}
+
+		// Ensure the serialized bytes deserialize to the expected
+		// value.
+		gotVal, gotBytesRead := deserializeVLQ(test.serialized)
+		if gotVal != test.val {
+			t.Errorf("deserializeVLQ: did not get expected value "+
+				"for %x - got %d, want %d", test.serialized,
+				gotVal, test.val)
+			continue
+		}
+		if gotBytesRead != len(test.serialized) {
+			t.Errorf("deserializeVLQ: did not get expected number "+
+				"of bytes read for %d - got %d, want %d",
+				test.serialized, gotBytesRead,
+				len(test.serialized))
+			continue
+		}
+	}
+}
+
+// TestScriptCompression ensures the domain-specific script compression and
+// decompression works as expected.
+func TestScriptCompression(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name         string
+		uncompressed []byte
+		compressed   []byte
+	}{
+		{
+			name:         "nil",
+			uncompressed: nil,
+			compressed:   hexToBytes("06"),
+		},
+		{
+			name:         "pay-to-pubkey-hash 1",
+			uncompressed: hexToBytes("76a9141018853670f9f3b0582c5b9ee8ce93764ac32b9388ac"),
+			compressed:   hexToBytes("001018853670f9f3b0582c5b9ee8ce93764ac32b93"),
+		},
+		{
+			name:         "pay-to-pubkey-hash 2",
+			uncompressed: hexToBytes("76a914e34cce70c86373273efcc54ce7d2a491bb4a0e8488ac"),
+			compressed:   hexToBytes("00e34cce70c86373273efcc54ce7d2a491bb4a0e84"),
+		},
+		{
+			name:         "pay-to-script-hash 1",
+			uncompressed: hexToBytes("a914da1745e9b549bd0bfa1a569971c77eba30cd5a4b87"),
+			compressed:   hexToBytes("01da1745e9b549bd0bfa1a569971c77eba30cd5a4b"),
+		},
+		{
+			name:         "pay-to-script-hash 2",
+			uncompressed: hexToBytes("a914f815b036d9bbbce5e9f2a00abd1bf3dc91e9551087"),
+			compressed:   hexToBytes("01f815b036d9bbbce5e9f2a00abd1bf3dc91e95510"),
+		},
+		{
+			name:         "pay-to-pubkey compressed 0x02",
+			uncompressed: hexToBytes("2102192d74d0cb94344c9569c2e77901573d8d7903c3ebec3a957724895dca52c6b4ac"),
+			compressed:   hexToBytes("02192d74d0cb94344c9569c2e77901573d8d7903c3ebec3a957724895dca52c6b4"),
+		},
+		{
+			name:         "pay-to-pubkey compressed 0x03",
+			uncompressed: hexToBytes("2103b0bd634234abbb1ba1e986e884185c61cf43e001f9137f23c2c409273eb16e65ac"),
+			compressed:   hexToBytes("03b0bd634234abbb1ba1e986e884185c61cf43e001f9137f23c2c409273eb16e65"),
+		},
+		{
+			name:         "pay-to-pubkey uncompressed 0x04 even",
+			uncompressed: hexToBytes("4104192d74d0cb94344c9569c2e77901573d8d7903c3ebec3a957724895dca52c6b40d45264838c0bd96852662ce6a847b197376830160c6d2eb5e6a4c44d33f453eac"),
+			compressed:   hexToBytes("04192d74d0cb94344c9569c2e77901573d8d7903c3ebec3a957724895dca52c6b4"),
+		},
+		{
+			name:         "pay-to-pubkey uncompressed 0x04 odd",
+			uncompressed: hexToBytes("410411db93e1dcdb8a016b49840f8c53bc1eb68a382e97b1482ecad7b148a6909a5cb2e0eaddfb84ccf9744464f82e160bfa9b8b64f9d4c03f999b8643f656b412a3ac"),
+			compressed:   hexToBytes("0511db93e1dcdb8a016b49840f8c53bc1eb68a382e97b1482ecad7b148a6909a5c"),
+		},
+		{
+			name:         "pay-to-pubkey invalid pubkey",
+			uncompressed: hexToBytes("3302aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaac"),
+			compressed:   hexToBytes("293302aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaac"),
+		},
+		{
+			name:         "requires 2 size bytes - data push 200 bytes",
+			uncompressed: append(hexToBytes("4cc8"), bytes.Repeat([]byte{0x00}, 200)...),
+			// [0x80, 0x50] = 208 as a variable length quantity
+			// [0x4c, 0xc8] = OP_PUSHDATA1 200
+			compressed: append(hexToBytes("80504cc8"), bytes.Repeat([]byte{0x00}, 200)...),
+		},
+	}
+
+	for _, test := range tests {
+		// Ensure the function to calculate the serialized size without
+		// actually serializing the value is calculated properly.
+		gotSize := compressedScriptSize(test.uncompressed)
+		if gotSize != len(test.compressed) {
+			t.Errorf("compressedScriptSize (%s): did not get "+
+				"expected size - got %d, want %d", test.name,
+				gotSize, len(test.compressed))
+			continue
+		}
+
+		// Ensure the script compresses to the expected bytes.
+		gotCompressed := make([]byte, gotSize)
+		gotBytesWritten := putCompressedScript(gotCompressed,
+			test.uncompressed)
+		if !bytes.Equal(gotCompressed, test.compressed) {
+			t.Errorf("putCompressedScript (%s): did not get "+
+				"expected bytes - got %x, want %x", test.name,
+				gotCompressed, test.compressed)
+			continue
+		}
+		if gotBytesWritten != len(test.compressed) {
+			t.Errorf("putCompressedScript (%s): did not get "+
+				"expected number of bytes written - got %d, "+
+				"want %d", test.name, gotBytesWritten,
+				len(test.compressed))
+			continue
+		}
+
+		// Ensure the compressed script size is properly decoded from
+		// the compressed script.
+		gotDecodedSize := decodeCompressedScriptSize(test.compressed)
+		if gotDecodedSize != len(test.compressed) {
+			t.Errorf("decodeCompressedScriptSize (%s): did not get "+
+				"expected size - got %d, want %d", test.name,
+				gotDecodedSize, len(test.compressed))
+			continue
+		}
+
+		// Ensure the script decompresses to the expected bytes.
+		gotDecompressed := decompressScript(test.compressed)
+		if !bytes.Equal(gotDecompressed, test.uncompressed) {
+			t.Errorf("decompressScript (%s): did not get expected "+
+				"bytes - got %x, want %x", test.name,
+				gotDecompressed, test.uncompressed)
+			continue
+		}
+	}
+}
+
+// TestScriptCompressionErrors ensures calling various functions related to
+// script compression with incorrect data returns the expected results.
+func TestScriptCompressionErrors(t *testing.T) {
+	t.Parallel()
+
+	// A nil script must result in a decoded size of 0.
+	if gotSize := decodeCompressedScriptSize(nil); gotSize != 0 {
+		t.Fatalf("decodeCompressedScriptSize with nil script did not "+
+			"return 0 - got %d", gotSize)
+	}
+
+	// A nil script must result in a nil decompressed script.
+	if gotScript := decompressScript(nil); gotScript != nil {
+		t.Fatalf("decompressScript with nil script did not return nil "+
+			"decompressed script - got %x", gotScript)
+	}
+
+	// A compressed script for a pay-to-pubkey (uncompressed) that results
+	// in an invalid pubkey must result in a nil decompressed script.
+	compressedScript := hexToBytes("04012d74d0cb94344c9569c2e77901573d8d" +
+		"7903c3ebec3a957724895dca52c6b4")
+	if gotScript := decompressScript(compressedScript); gotScript != nil {
+		t.Fatalf("decompressScript with compressed pay-to-"+
+			"uncompressed-pubkey that is invalid did not return "+
+			"nil decompressed script - got %x", gotScript)
+	}
+}
+
+// TestAmountCompression ensures the domain-specific transaction output amount
+// compression and decompression works as expected.
+func TestAmountCompression(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name         string
+		uncompressed uint64
+		compressed   uint64
+	}{
+		{
+			name:         "0 KAS",
+			uncompressed: 0,
+			compressed:   0,
+		},
+		{
+			name:         "546 Sompi (current network dust value)",
+			uncompressed: 546,
+			compressed:   4911,
+		},
+		{
+			name:         "0.00001 KAS (typical transaction fee)",
+			uncompressed: 1000,
+			compressed:   4,
+		},
+		{
+			name:         "0.0001 KAS (typical transaction fee)",
+			uncompressed: 10000,
+			compressed:   5,
+		},
+		{
+			name:         "0.12345678 KAS",
+			uncompressed: 12345678,
+			compressed:   111111101,
+		},
+		{
+			name:         "0.5 KAS",
+			uncompressed: 50000000,
+			compressed:   48,
+		},
+		{
+			name:         "1 KAS",
+			uncompressed: 100000000,
+			compressed:   9,
+		},
+		{
+			name:         "5 KAS",
+			uncompressed: 500000000,
+			compressed:   49,
+		},
+		{
+			name:         "21000000 KAS (max minted coins)",
+			uncompressed: 2100000000000000,
+			compressed:   21000000,
+		},
+	}
+
+	for _, test := range tests {
+		// Ensure the amount compresses to the expected value.
+		gotCompressed := compressTxOutAmount(test.uncompressed)
+		if gotCompressed != test.compressed {
+			t.Errorf("compressTxOutAmount (%s): did not get "+
+				"expected value - got %d, want %d", test.name,
+				gotCompressed, test.compressed)
+			continue
+		}
+
+		// Ensure the value decompresses to the expected value.
+		gotDecompressed := decompressTxOutAmount(test.compressed)
+		if gotDecompressed != test.uncompressed {
+			t.Errorf("decompressTxOutAmount (%s): did not get "+
+				"expected value - got %d, want %d", test.name,
+				gotDecompressed, test.uncompressed)
+			continue
+		}
+	}
+}
+
+// TestCompressedTxOut ensures the transaction output serialization and
+// deserialization works as expected.
+func TestCompressedTxOut(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name         string
+		amount       uint64
+		scriptPubKey []byte
+		compressed   []byte
+	}{
+		{
+			name:         "pay-to-pubkey-hash dust",
+			amount:       546,
+			scriptPubKey: hexToBytes("76a9141018853670f9f3b0582c5b9ee8ce93764ac32b9388ac"),
+			compressed:   hexToBytes("a52f001018853670f9f3b0582c5b9ee8ce93764ac32b93"),
+		},
+		{
+			name:         "pay-to-pubkey uncompressed 1 KAS",
+			amount:       100000000,
+			scriptPubKey: hexToBytes("4104192d74d0cb94344c9569c2e77901573d8d7903c3ebec3a957724895dca52c6b40d45264838c0bd96852662ce6a847b197376830160c6d2eb5e6a4c44d33f453eac"),
+			compressed:   hexToBytes("0904192d74d0cb94344c9569c2e77901573d8d7903c3ebec3a957724895dca52c6b4"),
+		},
+	}
+
+	for _, test := range tests {
+		// Ensure the function to calculate the serialized size without
+		// actually serializing the txout is calculated properly.
+		gotSize := compressedTxOutSize(test.amount, test.scriptPubKey)
+		if gotSize != len(test.compressed) {
+			t.Errorf("compressedTxOutSize (%s): did not get "+
+				"expected size - got %d, want %d", test.name,
+				gotSize, len(test.compressed))
+			continue
+		}
+
+		// Ensure the txout compresses to the expected value.
+		gotCompressed := make([]byte, gotSize)
+		gotBytesWritten := putCompressedTxOut(gotCompressed,
+			test.amount, test.scriptPubKey)
+		if !bytes.Equal(gotCompressed, test.compressed) {
+			t.Errorf("compressTxOut (%s): did not get expected "+
+				"bytes - got %x, want %x", test.name,
+				gotCompressed, test.compressed)
+			continue
+		}
+		if gotBytesWritten != len(test.compressed) {
+			t.Errorf("compressTxOut (%s): did not get expected "+
+				"number of bytes written - got %d, want %d",
+				test.name, gotBytesWritten,
+				len(test.compressed))
+			continue
+		}
+
+		// Ensure the serialized bytes are decoded back to the expected
+		// uncompressed values.
+		gotAmount, gotScript, gotBytesRead, err := decodeCompressedTxOut(
+			test.compressed)
+		if err != nil {
+			t.Errorf("decodeCompressedTxOut (%s): unexpected "+
+				"error: %v", test.name, err)
+			continue
+		}
+		if gotAmount != test.amount {
+			t.Errorf("decodeCompressedTxOut (%s): did not get "+
+				"expected amount - got %d, want %d",
+				test.name, gotAmount, test.amount)
+			continue
+		}
+		if !bytes.Equal(gotScript, test.scriptPubKey) {
+			t.Errorf("decodeCompressedTxOut (%s): did not get "+
+				"expected script - got %x, want %x",
+				test.name, gotScript, test.scriptPubKey)
+			continue
+		}
+		if gotBytesRead != len(test.compressed) {
+			t.Errorf("decodeCompressedTxOut (%s): did not get "+
+				"expected number of bytes read - got %d, want %d",
+				test.name, gotBytesRead, len(test.compressed))
+			continue
+		}
+	}
+}
+
+// TestTxOutCompressionErrors ensures calling various functions related to
+// txout compression with incorrect data returns the expected results.
+func TestTxOutCompressionErrors(t *testing.T) {
+	t.Parallel()
+
+	// A compressed txout with missing compressed script must error.
+	compressedTxOut := hexToBytes("00")
+	_, _, _, err := decodeCompressedTxOut(compressedTxOut)
+	if !isDeserializeErr(err) {
+		t.Fatalf("decodeCompressedTxOut with missing compressed script "+
+			"did not return expected error type - got %T, want "+
+			"errDeserialize", err)
+	}
+
+	// A compressed txout with short compressed script must error.
+	compressedTxOut = hexToBytes("0010")
+	_, _, _, err = decodeCompressedTxOut(compressedTxOut)
+	if !isDeserializeErr(err) {
+		t.Fatalf("decodeCompressedTxOut with short compressed script "+
+			"did not return expected error type - got %T, want "+
+			"errDeserialize", err)
+	}
+}

blockdag/dagio.go

@@ -0,0 +1,907 @@
+// Copyright (c) 2015-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"bytes"
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"github.com/kaspanet/kaspad/dagconfig"
+	"github.com/pkg/errors"
+	"io"
+	"sync"
+
+	"github.com/kaspanet/kaspad/database"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/kaspanet/kaspad/util/binaryserializer"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/util/subnetworkid"
+	"github.com/kaspanet/kaspad/wire"
+)
+
+const (
+	// blockHdrSize is the size of a block header. This is simply the
+	// constant from wire and is only provided here for convenience since
+	// wire.MaxBlockHeaderPayload is quite long.
+	blockHdrSize = wire.MaxBlockHeaderPayload
+
+	// latestUTXOSetBucketVersion is the current version of the UTXO set
+	// bucket that is used to track all unspent outputs.
+	latestUTXOSetBucketVersion = 1
+)
+
+var (
+	// blockIndexBucketName is the name of the database bucket used to house the
+	// block headers and contextual information.
+	blockIndexBucketName = []byte("blockheaderidx")
+
+	// dagStateKeyName is the name of the db key used to store the DAG
+	// tip hashes.
+	dagStateKeyName = []byte("dagstate")
+
+	// utxoSetVersionKeyName is the name of the db key used to store the
+	// version of the utxo set currently in the database.
+	utxoSetVersionKeyName = []byte("utxosetversion")
+
+	// utxoSetBucketName is the name of the database bucket used to house the
+	// unspent transaction output set.
+	utxoSetBucketName = []byte("utxoset")
+
+	// utxoDiffsBucketName is the name of the database bucket used to house the
+	// diffs and diff children of blocks.
+	utxoDiffsBucketName = []byte("utxodiffs")
+
+	// reachabilityDataBucketName is the name of the database bucket used to house the
+	// reachability tree nodes and future covering sets of blocks.
+	reachabilityDataBucketName = []byte("reachability")
+
+	// subnetworksBucketName is the name of the database bucket used to store the
+	// subnetwork registry.
+	subnetworksBucketName = []byte("subnetworks")
+
+	// localSubnetworkKeyName is the name of the db key used to store the
+	// node's local subnetwork ID.
+	localSubnetworkKeyName = []byte("localsubnetworkidkey")
+
+	// byteOrder is the preferred byte order used for serializing numeric
+	// fields for storage in the database.
+	byteOrder = binary.LittleEndian
+)
+
+// errNotInDAG signifies that a block hash or height that is not in the
+// DAG was requested.
+type errNotInDAG string
+
+// Error implements the error interface.
+func (e errNotInDAG) Error() string {
+	return string(e)
+}
+
+// isNotInDAGErr returns whether or not the passed error is an
+// errNotInDAG error.
+func isNotInDAGErr(err error) bool {
+	_, ok := err.(errNotInDAG)
+	return ok
+}
+
+// errDeserialize signifies that a problem was encountered when deserializing
+// data.
+type errDeserialize string
+
+// Error implements the error interface.
+func (e errDeserialize) Error() string {
+	return string(e)
+}
+
+// isDeserializeErr returns whether or not the passed error is an errDeserialize
+// error.
+func isDeserializeErr(err error) bool {
+	_, ok := err.(errDeserialize)
+	return ok
+}
+
+// dbPutVersion uses an existing database transaction to update the provided
+// key in the metadata bucket to the given version. It is primarily used to
+// track versions on entities such as buckets.
+func dbPutVersion(dbTx database.Tx, key []byte, version uint32) error {
+	var serialized [4]byte
+	byteOrder.PutUint32(serialized[:], version)
+	return dbTx.Metadata().Put(key, serialized[:])
+}
+
+// -----------------------------------------------------------------------------
+// The unspent transaction output (UTXO) set consists of an entry for each
+// unspent output using a format that is optimized to reduce space using domain
+// specific compression algorithms.
+//
+// Each entry is keyed by an outpoint as specified below. It is important to
+// note that the key encoding uses a VLQ, which employs an MSB encoding so
+// iteration of UTXOs when doing byte-wise comparisons will produce them in
+// order.
+//
+// The serialized key format is:
+//   <hash><output index>
+//
+//   Field                Type             Size
+//   hash                 daghash.Hash   daghash.HashSize
+//   output index         VLQ              variable
+//
+// The serialized value format is:
+//
+//   <header code><compressed txout>
+//
+//   Field                Type     Size
+//   header code          VLQ      variable
+//   compressed txout
+//     compressed amount  VLQ      variable
+//     compressed script  []byte   variable
+//
+// The serialized header code format is:
+//   bit 0 - containing transaction is a coinbase
+//   bits 1-x - height of the block that contains the unspent txout
+//
+// Example 1:
+// b7c3332bc138e2c9429818f5fed500bcc1746544218772389054dc8047d7cd3f:0
+//
+//    03320496b538e853519c726a2c91e61ec11600ae1390813a627c66fb8be7947be63c52
+//    <><------------------------------------------------------------------>
+//     |                                          |
+//   header code                         compressed txout
+//
+//  - header code: 0x03 (coinbase, height 1)
+//  - compressed txout:
+//    - 0x32: VLQ-encoded compressed amount for 5000000000 (50 KAS)
+//    - 0x04: special script type pay-to-pubkey
+//    - 0x96...52: x-coordinate of the pubkey
+//
+// Example 2:
+// 4a16969aa4764dd7507fc1de7f0baa4850a246de90c45e59a3207f9a26b5036f:2
+//
+//    8cf316800900b8025be1b3efc63b0ad48e7f9f10e87544528d58
+//    <----><------------------------------------------>
+//      |                             |
+//   header code             compressed txout
+//
+//  - header code: 0x8cf316 (not coinbase, height 113931)
+//  - compressed txout:
+//    - 0x8009: VLQ-encoded compressed amount for 15000000 (0.15 KAS)
+//    - 0x00: special script type pay-to-pubkey-hash
+//    - 0xb8...58: pubkey hash
+//
+// Example 3:
+// 1b02d1c8cfef60a189017b9a420c682cf4a0028175f2f563209e4ff61c8c3620:22
+//
+//    a8a2588ba5b9e763011dd46a006572d820e448e12d2bbb38640bc718e6
+//    <----><-------------------------------------------------->
+//      |                             |
+//   header code             compressed txout
+//
+//  - header code: 0xa8a258 (not coinbase, height 338156)
+//  - compressed txout:
+//    - 0x8ba5b9e763: VLQ-encoded compressed amount for 366875659 (3.66875659 KAS)
+//    - 0x01: special script type pay-to-script-hash
+//    - 0x1d...e6: script hash
+// -----------------------------------------------------------------------------
+
+// maxUint32VLQSerializeSize is the maximum number of bytes a max uint32 takes
+// to serialize as a VLQ.
+var maxUint32VLQSerializeSize = serializeSizeVLQ(1<<32 - 1)
+
+// outpointKeyPool defines a concurrent safe free list of byte slices used to
+// provide temporary buffers for outpoint database keys.
+var outpointKeyPool = sync.Pool{
+	New: func() interface{} {
+		b := make([]byte, daghash.HashSize+maxUint32VLQSerializeSize)
+		return &b // Pointer to slice to avoid boxing alloc.
+	},
+}
+
+// outpointKey returns a key suitable for use as a database key in the UTXO set
+// while making use of a free list. A new buffer is allocated if there are not
+// already any available on the free list. The returned byte slice should be
+// returned to the free list by using the recycleOutpointKey function when the
+// caller is done with it _unless_ the slice will need to live for longer than
+// the caller can calculate such as when used to write to the database.
+func outpointKey(outpoint wire.Outpoint) *[]byte {
+	// A VLQ employs an MSB encoding, so they are useful not only to reduce
+	// the amount of storage space, but also so iteration of UTXOs when
+	// doing byte-wise comparisons will produce them in order.
+	key := outpointKeyPool.Get().(*[]byte)
+	idx := uint64(outpoint.Index)
+	*key = (*key)[:daghash.HashSize+serializeSizeVLQ(idx)]
+	copy(*key, outpoint.TxID[:])
+	putVLQ((*key)[daghash.HashSize:], idx)
+	return key
+}
+
+// recycleOutpointKey puts the provided byte slice, which should have been
+// obtained via the outpointKey function, back on the free list.
+func recycleOutpointKey(key *[]byte) {
+	outpointKeyPool.Put(key)
+}
+
+// dbPutUTXODiff uses an existing database transaction to update the UTXO set
+// in the database based on the provided UTXO view contents and state. In
+// particular, only the entries that have been marked as modified are written
+// to the database.
+func dbPutUTXODiff(dbTx database.Tx, diff *UTXODiff) error {
+	utxoBucket := dbTx.Metadata().Bucket(utxoSetBucketName)
+	for outpoint := range diff.toRemove {
+		key := outpointKey(outpoint)
+		err := utxoBucket.Delete(*key)
+		recycleOutpointKey(key)
+		if err != nil {
+			return err
+		}
+	}
+
+	for outpoint, entry := range diff.toAdd {
+		// Serialize and store the UTXO entry.
+		serialized := serializeUTXOEntry(entry)
+
+		key := outpointKey(outpoint)
+		err := utxoBucket.Put(*key, serialized)
+		// NOTE: The key is intentionally not recycled here since the
+		// database interface contract prohibits modifications. It will
+		// be garbage collected normally when the database is done with
+		// it.
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+type dagState struct {
+	TipHashes         []*daghash.Hash
+	LastFinalityPoint *daghash.Hash
+}
+
+// serializeDAGState returns the serialization of the DAG state.
+// This is data to be stored in the DAG state bucket.
+func serializeDAGState(state *dagState) ([]byte, error) {
+	return json.Marshal(state)
+}
+
+// deserializeDAGState deserializes the passed serialized DAG state.
+// This is data stored in the DAG state bucket and is updated after
+// every block is connected to the DAG.
+func deserializeDAGState(serializedData []byte) (*dagState, error) {
+	var state *dagState
+	err := json.Unmarshal(serializedData, &state)
+	if err != nil {
+		return nil, database.Error{
+			ErrorCode:   database.ErrCorruption,
+			Description: "corrupt DAG state",
+		}
+	}
+
+	return state, nil
+}
+
+// dbPutDAGState uses an existing database transaction to store the latest
+// tip hashes of the DAG.
+func dbPutDAGState(dbTx database.Tx, state *dagState) error {
+	serializedData, err := serializeDAGState(state)
+
+	if err != nil {
+		return err
+	}
+
+	return dbTx.Metadata().Put(dagStateKeyName, serializedData)
+}
+
+// createDAGState initializes both the database and the DAG state to the
+// genesis block. This includes creating the necessary buckets, so it
+// must only be called on an uninitialized database.
+func (dag *BlockDAG) createDAGState() error {
+	// Create the initial the database DAG state including creating the
+	// necessary index buckets and inserting the genesis block.
+	err := dag.db.Update(func(dbTx database.Tx) error {
+		meta := dbTx.Metadata()
+
+		// Create the bucket that houses the block index data.
+		_, err := meta.CreateBucket(blockIndexBucketName)
+		if err != nil {
+			return err
+		}
+
+		// Create the buckets that house the utxo set, the utxo diffs, and their
+		// version.
+		_, err = meta.CreateBucket(utxoSetBucketName)
+		if err != nil {
+			return err
+		}
+
+		_, err = meta.CreateBucket(utxoDiffsBucketName)
+		if err != nil {
+			return err
+		}
+
+		_, err = meta.CreateBucket(reachabilityDataBucketName)
+		if err != nil {
+			return err
+		}
+
+		err = dbPutVersion(dbTx, utxoSetVersionKeyName,
+			latestUTXOSetBucketVersion)
+		if err != nil {
+			return err
+		}
+
+		// Create the bucket that houses the registered subnetworks.
+		_, err = meta.CreateBucket(subnetworksBucketName)
+		if err != nil {
+			return err
+		}
+
+		if err := dbPutLocalSubnetworkID(dbTx, dag.subnetworkID); err != nil {
+			return err
+		}
+
+		if _, err := meta.CreateBucketIfNotExists(idByHashIndexBucketName); err != nil {
+			return err
+		}
+		if _, err := meta.CreateBucketIfNotExists(hashByIDIndexBucketName); err != nil {
+			return err
+		}
+		return nil
+	})
+
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+func (dag *BlockDAG) removeDAGState() error {
+	err := dag.db.Update(func(dbTx database.Tx) error {
+		meta := dbTx.Metadata()
+
+		err := meta.DeleteBucket(blockIndexBucketName)
+		if err != nil {
+			return err
+		}
+
+		err = meta.DeleteBucket(utxoSetBucketName)
+		if err != nil {
+			return err
+		}
+
+		err = meta.DeleteBucket(utxoDiffsBucketName)
+		if err != nil {
+			return err
+		}
+
+		err = meta.DeleteBucket(reachabilityDataBucketName)
+		if err != nil {
+			return err
+		}
+
+		err = dbTx.Metadata().Delete(utxoSetVersionKeyName)
+		if err != nil {
+			return err
+		}
+
+		err = meta.DeleteBucket(subnetworksBucketName)
+		if err != nil {
+			return err
+		}
+
+		err = dbTx.Metadata().Delete(localSubnetworkKeyName)
+		if err != nil {
+			return err
+		}
+
+		return nil
+	})
+
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+func dbPutLocalSubnetworkID(dbTx database.Tx, subnetworkID *subnetworkid.SubnetworkID) error {
+	if subnetworkID == nil {
+		return dbTx.Metadata().Put(localSubnetworkKeyName, []byte{})
+	}
+	return dbTx.Metadata().Put(localSubnetworkKeyName, subnetworkID[:])
+}
+
+// initDAGState attempts to load and initialize the DAG state from the
+// database. When the db does not yet contain any DAG state, both it and the
+// DAG state are initialized to the genesis block.
+func (dag *BlockDAG) initDAGState() error {
+	// Determine the state of the DAG database. We may need to initialize
+	// everything from scratch or upgrade certain buckets.
+	var initialized bool
+	err := dag.db.View(func(dbTx database.Tx) error {
+		initialized = dbTx.Metadata().Get(dagStateKeyName) != nil
+		if initialized {
+			var localSubnetworkID *subnetworkid.SubnetworkID
+			localSubnetworkIDBytes := dbTx.Metadata().Get(localSubnetworkKeyName)
+			if len(localSubnetworkIDBytes) != 0 {
+				localSubnetworkID = &subnetworkid.SubnetworkID{}
+				localSubnetworkID.SetBytes(localSubnetworkIDBytes)
+			}
+			if !localSubnetworkID.IsEqual(dag.subnetworkID) {
+				return errors.Errorf("Cannot start kaspad with subnetwork ID %s because"+
+					" its database is already built with subnetwork ID %s. If you"+
+					" want to switch to a new database, please reset the"+
+					" database by starting kaspad with --reset-db flag", dag.subnetworkID, localSubnetworkID)
+			}
+		}
+		return nil
+	})
+	if err != nil {
+		return err
+	}
+
+	if !initialized {
+		// At this point the database has not already been initialized, so
+		// initialize both it and the DAG state to the genesis block.
+		return dag.createDAGState()
+	}
+
+	// Attempt to load the DAG state from the database.
+	return dag.db.View(func(dbTx database.Tx) error {
+		// Fetch the stored DAG tipHashes from the database metadata.
+		// When it doesn't exist, it means the database hasn't been
+		// initialized for use with the DAG yet, so break out now to allow
+		// that to happen under a writable database transaction.
+		serializedData := dbTx.Metadata().Get(dagStateKeyName)
+		log.Tracef("Serialized DAG tip hashes: %x", serializedData)
+		state, err := deserializeDAGState(serializedData)
+		if err != nil {
+			return err
+		}
+
+		// Load all of the headers from the data for the known DAG
+		// and construct the block index accordingly. Since the
+		// number of nodes are already known, perform a single alloc
+		// for them versus a whole bunch of little ones to reduce
+		// pressure on the GC.
+		log.Infof("Loading block index...")
+
+		blockIndexBucket := dbTx.Metadata().Bucket(blockIndexBucketName)
+
+		var unprocessedBlockNodes []*blockNode
+		cursor := blockIndexBucket.Cursor()
+		for ok := cursor.First(); ok; ok = cursor.Next() {
+			node, err := dag.deserializeBlockNode(cursor.Value())
+			if err != nil {
+				return err
+			}
+
+			// Check to see if this node had been stored in the the block DB
+			// but not yet accepted. If so, add it to a slice to be processed later.
+			if node.status == statusDataStored {
+				unprocessedBlockNodes = append(unprocessedBlockNodes, node)
+				continue
+			}
+
+			// If the node is known to be invalid add it as-is to the block
+			// index and continue.
+			if node.status.KnownInvalid() {
+				dag.index.addNode(node)
+				continue
+			}
+
+			if dag.blockCount == 0 {
+				if !node.hash.IsEqual(dag.dagParams.GenesisHash) {
+					return AssertError(fmt.Sprintf("initDAGState: Expected "+
+						"first entry in block index to be genesis block, "+
+						"found %s", node.hash))
+				}
+			} else {
+				if len(node.parents) == 0 {
+					return AssertError(fmt.Sprintf("initDAGState: Could "+
+						"not find any parent for block %s", node.hash))
+				}
+			}
+
+			// Add the node to its parents children, connect it,
+			// and add it to the block index.
+			node.updateParentsChildren()
+			dag.index.addNode(node)
+
+			dag.blockCount++
+		}
+
+		// Load all of the known UTXO entries and construct the full
+		// UTXO set accordingly. Since the number of entries is already
+		// known, perform a single alloc for them versus a whole bunch
+		// of little ones to reduce pressure on the GC.
+		log.Infof("Loading UTXO set...")
+
+		utxoEntryBucket := dbTx.Metadata().Bucket(utxoSetBucketName)
+
+		// Determine how many UTXO entries will be loaded into the index so we can
+		// allocate the right amount.
+		var utxoEntryCount int32
+		cursor = utxoEntryBucket.Cursor()
+		for ok := cursor.First(); ok; ok = cursor.Next() {
+			utxoEntryCount++
+		}
+
+		fullUTXOCollection := make(utxoCollection, utxoEntryCount)
+		for ok := cursor.First(); ok; ok = cursor.Next() {
+			// Deserialize the outpoint
+			outpoint, err := deserializeOutpoint(cursor.Key())
+			if err != nil {
+				// Ensure any deserialization errors are returned as database
+				// corruption errors.
+				if isDeserializeErr(err) {
+					return database.Error{
+						ErrorCode:   database.ErrCorruption,
+						Description: fmt.Sprintf("corrupt outpoint: %s", err),
+					}
+				}
+
+				return err
+			}
+
+			// Deserialize the utxo entry
+			entry, err := deserializeUTXOEntry(cursor.Value())
+			if err != nil {
+				// Ensure any deserialization errors are returned as database
+				// corruption errors.
+				if isDeserializeErr(err) {
+					return database.Error{
+						ErrorCode:   database.ErrCorruption,
+						Description: fmt.Sprintf("corrupt utxo entry: %s", err),
+					}
+				}
+
+				return err
+			}
+
+			fullUTXOCollection[*outpoint] = entry
+		}
+
+		// Initialize the reachability store
+		err = dag.reachabilityStore.init(dbTx)
+		if err != nil {
+			return err
+		}
+
+		// Apply the loaded utxoCollection to the virtual block.
+		dag.virtual.utxoSet, err = newFullUTXOSetFromUTXOCollection(fullUTXOCollection)
+		if err != nil {
+			return AssertError(fmt.Sprintf("Error loading UTXOSet: %s", err))
+		}
+
+		// Apply the stored tips to the virtual block.
+		tips := newSet()
+		for _, tipHash := range state.TipHashes {
+			tip := dag.index.LookupNode(tipHash)
+			if tip == nil {
+				return AssertError(fmt.Sprintf("initDAGState: cannot find "+
+					"DAG tip %s in block index", state.TipHashes))
+			}
+			tips.add(tip)
+		}
+		dag.virtual.SetTips(tips)
+
+		// Set the last finality point
+		dag.lastFinalityPoint = dag.index.LookupNode(state.LastFinalityPoint)
+		dag.finalizeNodesBelowFinalityPoint(false)
+
+		// Go over any unprocessed blockNodes and process them now.
+		for _, node := range unprocessedBlockNodes {
+			// Check to see if the block exists in the block DB. If it
+			// doesn't, the database has certainly been corrupted.
+			blockExists, err := dbTx.HasBlock(node.hash)
+			if err != nil {
+				return AssertError(fmt.Sprintf("initDAGState: HasBlock "+
+					"for block %s failed: %s", node.hash, err))
+			}
+			if !blockExists {
+				return AssertError(fmt.Sprintf("initDAGState: block %s "+
+					"exists in block index but not in block db", node.hash))
+			}
+
+			// Attempt to accept the block.
+			block, err := dbFetchBlockByNode(dbTx, node)
+			isOrphan, isDelayed, err := dag.ProcessBlock(block, BFWasStored)
+			if err != nil {
+				log.Warnf("Block %s, which was not previously processed, "+
+					"failed to be accepted to the DAG: %s", node.hash, err)
+				continue
+			}
+
+			// If the block is an orphan or is delayed then it couldn't have
+			// possibly been written to the block index in the first place.
+			if isOrphan {
+				return AssertError(fmt.Sprintf("Block %s, which was not "+
+					"previously processed, turned out to be an orphan, which is "+
+					"impossible.", node.hash))
+			}
+			if isDelayed {
+				return AssertError(fmt.Sprintf("Block %s, which was not "+
+					"previously processed, turned out to be delayed, which is "+
+					"impossible.", node.hash))
+			}
+		}
+
+		return nil
+	})
+}
+
+// deserializeBlockNode parses a value in the block index bucket and returns a block node.
+func (dag *BlockDAG) deserializeBlockNode(blockRow []byte) (*blockNode, error) {
+	buffer := bytes.NewReader(blockRow)
+
+	var header wire.BlockHeader
+	err := header.Deserialize(buffer)
+	if err != nil {
+		return nil, err
+	}
+
+	node := &blockNode{
+		hash:                 header.BlockHash(),
+		version:              header.Version,
+		bits:                 header.Bits,
+		nonce:                header.Nonce,
+		timestamp:            header.Timestamp.Unix(),
+		hashMerkleRoot:       header.HashMerkleRoot,
+		acceptedIDMerkleRoot: header.AcceptedIDMerkleRoot,
+		utxoCommitment:       header.UTXOCommitment,
+	}
+
+	node.children = newSet()
+	node.parents = newSet()
+
+	for _, hash := range header.ParentHashes {
+		parent := dag.index.LookupNode(hash)
+		if parent == nil {
+			return nil, AssertError(fmt.Sprintf("deserializeBlockNode: Could "+
+				"not find parent %s for block %s", hash, header.BlockHash()))
+		}
+		node.parents.add(parent)
+	}
+
+	statusByte, err := buffer.ReadByte()
+	if err != nil {
+		return nil, err
+	}
+	node.status = blockStatus(statusByte)
+
+	selectedParentHash := &daghash.Hash{}
+	if _, err := io.ReadFull(buffer, selectedParentHash[:]); err != nil {
+		return nil, err
+	}
+
+	// Because genesis doesn't have selected parent, it's serialized as zero hash
+	if !selectedParentHash.IsEqual(&daghash.ZeroHash) {
+		node.selectedParent = dag.index.LookupNode(selectedParentHash)
+	}
+
+	node.blueScore, err = binaryserializer.Uint64(buffer, byteOrder)
+	if err != nil {
+		return nil, err
+	}
+
+	bluesCount, err := wire.ReadVarInt(buffer)
+	if err != nil {
+		return nil, err
+	}
+
+	node.blues = make([]*blockNode, bluesCount)
+	for i := uint64(0); i < bluesCount; i++ {
+		hash := &daghash.Hash{}
+		if _, err := io.ReadFull(buffer, hash[:]); err != nil {
+			return nil, err
+		}
+		node.blues[i] = dag.index.LookupNode(hash)
+	}
+
+	bluesAnticoneSizesLen, err := wire.ReadVarInt(buffer)
+	if err != nil {
+		return nil, err
+	}
+
+	node.bluesAnticoneSizes = make(map[daghash.Hash]dagconfig.KType)
+	for i := uint64(0); i < bluesAnticoneSizesLen; i++ {
+		hash := &daghash.Hash{}
+		if _, err := io.ReadFull(buffer, hash[:]); err != nil {
+			return nil, err
+		}
+		bluesAnticoneSize, err := binaryserializer.Uint8(buffer)
+		node.bluesAnticoneSizes[*hash] = dagconfig.KType(bluesAnticoneSize)
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	return node, nil
+}
+
+// dbFetchBlockByNode uses an existing database transaction to retrieve the
+// raw block for the provided node, deserialize it, and return a util.Block
+// of it.
+func dbFetchBlockByNode(dbTx database.Tx, node *blockNode) (*util.Block, error) {
+	// Load the raw block bytes from the database.
+	blockBytes, err := dbTx.FetchBlock(node.hash)
+	if err != nil {
+		return nil, err
+	}
+
+	// Create the encapsulated block.
+	block, err := util.NewBlockFromBytes(blockBytes)
+	if err != nil {
+		return nil, err
+	}
+	return block, nil
+}
+
+func serializeBlockNode(node *blockNode) ([]byte, error) {
+	w := bytes.NewBuffer(make([]byte, 0, blockHdrSize+1))
+	header := node.Header()
+	err := header.Serialize(w)
+	if err != nil {
+		return nil, err
+	}
+
+	err = w.WriteByte(byte(node.status))
+	if err != nil {
+		return nil, err
+	}
+
+	// Because genesis doesn't have selected parent, it's serialized as zero hash
+	selectedParentHash := &daghash.ZeroHash
+	if node.selectedParent != nil {
+		selectedParentHash = node.selectedParent.hash
+	}
+	_, err = w.Write(selectedParentHash[:])
+	if err != nil {
+		return nil, err
+	}
+
+	err = binaryserializer.PutUint64(w, byteOrder, node.blueScore)
+	if err != nil {
+		return nil, err
+	}
+
+	err = wire.WriteVarInt(w, uint64(len(node.blues)))
+	if err != nil {
+		return nil, err
+	}
+
+	for _, blue := range node.blues {
+		_, err = w.Write(blue.hash[:])
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	err = wire.WriteVarInt(w, uint64(len(node.bluesAnticoneSizes)))
+	if err != nil {
+		return nil, err
+	}
+	for blockHash, blueAnticoneSize := range node.bluesAnticoneSizes {
+		_, err = w.Write(blockHash[:])
+		if err != nil {
+			return nil, err
+		}
+
+		err = binaryserializer.PutUint8(w, uint8(blueAnticoneSize))
+		if err != nil {
+			return nil, err
+		}
+	}
+	return w.Bytes(), nil
+}
+
+// dbStoreBlockNode stores the block node data into the block
+// index bucket. This overwrites the current entry if there exists one.
+func dbStoreBlockNode(dbTx database.Tx, node *blockNode) error {
+	serializedNode, err := serializeBlockNode(node)
+	if err != nil {
+		return err
+	}
+	// Write block header data to block index bucket.
+	blockIndexBucket := dbTx.Metadata().Bucket(blockIndexBucketName)
+	key := BlockIndexKey(node.hash, node.blueScore)
+	return blockIndexBucket.Put(key, serializedNode)
+}
+
+// dbStoreBlock stores the provided block in the database if it is not already
+// there. The full block data is written to ffldb.
+func dbStoreBlock(dbTx database.Tx, block *util.Block) error {
+	hasBlock, err := dbTx.HasBlock(block.Hash())
+	if err != nil {
+		return err
+	}
+	if hasBlock {
+		return nil
+	}
+	return dbTx.StoreBlock(block)
+}
+
+// BlockIndexKey generates the binary key for an entry in the block index
+// bucket. The key is composed of the block blue score encoded as a big-endian
+// 64-bit unsigned int followed by the 32 byte block hash.
+// The blue score component is important for iteration order.
+func BlockIndexKey(blockHash *daghash.Hash, blueScore uint64) []byte {
+	indexKey := make([]byte, daghash.HashSize+8)
+	binary.BigEndian.PutUint64(indexKey[0:8], blueScore)
+	copy(indexKey[8:daghash.HashSize+8], blockHash[:])
+	return indexKey
+}
+
+func blockHashFromBlockIndexKey(BlockIndexKey []byte) (*daghash.Hash, error) {
+	return daghash.NewHash(BlockIndexKey[8 : daghash.HashSize+8])
+}
+
+// BlockByHash returns the block from the DAG with the given hash.
+//
+// This function is safe for concurrent access.
+func (dag *BlockDAG) BlockByHash(hash *daghash.Hash) (*util.Block, error) {
+	// Lookup the block hash in block index and ensure it is in the DAG
+	node := dag.index.LookupNode(hash)
+	if node == nil {
+		str := fmt.Sprintf("block %s is not in the DAG", hash)
+		return nil, errNotInDAG(str)
+	}
+
+	// Load the block from the database and return it.
+	var block *util.Block
+	err := dag.db.View(func(dbTx database.Tx) error {
+		var err error
+		block, err = dbFetchBlockByNode(dbTx, node)
+		return err
+	})
+	return block, err
+}
+
+// BlockHashesFrom returns a slice of blocks starting from lowHash
+// ordered by blueScore. If lowHash is nil then the genesis block is used.
+//
+// This method MUST be called with the DAG lock held
+func (dag *BlockDAG) BlockHashesFrom(lowHash *daghash.Hash, limit int) ([]*daghash.Hash, error) {
+	blockHashes := make([]*daghash.Hash, 0, limit)
+	if lowHash == nil {
+		lowHash = dag.genesis.hash
+
+		// If we're starting from the beginning we should include the
+		// genesis hash in the result
+		blockHashes = append(blockHashes, dag.genesis.hash)
+	}
+	if !dag.BlockExists(lowHash) {
+		return nil, errors.Errorf("block %s not found", lowHash)
+	}
+	blueScore, err := dag.BlueScoreByBlockHash(lowHash)
+	if err != nil {
+		return nil, err
+	}
+
+	err = dag.index.db.View(func(dbTx database.Tx) error {
+		blockIndexBucket := dbTx.Metadata().Bucket(blockIndexBucketName)
+		lowKey := BlockIndexKey(lowHash, blueScore)
+
+		cursor := blockIndexBucket.Cursor()
+		cursor.Seek(lowKey)
+		for ok := cursor.Next(); ok; ok = cursor.Next() {
+			key := cursor.Key()
+			blockHash, err := blockHashFromBlockIndexKey(key)
+			if err != nil {
+				return err
+			}
+			blockHashes = append(blockHashes, blockHash)
+			if len(blockHashes) == limit {
+				break
+			}
+		}
+		return nil
+	})
+	if err != nil {
+		return nil, err
+	}
+	return blockHashes, nil
+}

blockdag/dagio_test.go

@@ -0,0 +1,274 @@
+// Copyright (c) 2015-2016 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"bytes"
+	"github.com/pkg/errors"
+	"reflect"
+	"testing"
+
+	"github.com/kaspanet/kaspad/database"
+	"github.com/kaspanet/kaspad/util/daghash"
+)
+
+// TestErrNotInDAG ensures the functions related to errNotInDAG work
+// as expected.
+func TestErrNotInDAG(t *testing.T) {
+	errStr := "no block at height 1 exists"
+	err := error(errNotInDAG(errStr))
+
+	// Ensure the stringized output for the error is as expected.
+	if err.Error() != errStr {
+		t.Fatalf("errNotInDAG retuned unexpected error string - "+
+			"got %q, want %q", err.Error(), errStr)
+	}
+
+	// Ensure error is detected as the correct type.
+	if !isNotInDAGErr(err) {
+		t.Fatalf("isNotInDAGErr did not detect as expected type")
+	}
+	err = errors.New("something else")
+	if isNotInDAGErr(err) {
+		t.Fatalf("isNotInDAGErr detected incorrect type")
+	}
+}
+
+// TestUtxoSerialization ensures serializing and deserializing unspent
+// trasaction output entries works as expected.
+func TestUtxoSerialization(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name       string
+		entry      *UTXOEntry
+		serialized []byte
+	}{
+		{
+			name: "blue score 1, coinbase",
+			entry: &UTXOEntry{
+				amount:         5000000000,
+				scriptPubKey:   hexToBytes("410496b538e853519c726a2c91e61ec11600ae1390813a627c66fb8be7947be63c52da7589379515d4e0a604f8141781e62294721166bf621e73a82cbf2342c858eeac"),
+				blockBlueScore: 1,
+				packedFlags:    tfCoinbase,
+			},
+			serialized: hexToBytes("03320496b538e853519c726a2c91e61ec11600ae1390813a627c66fb8be7947be63c52"),
+		},
+		{
+			name: "blue score 100001, not coinbase",
+			entry: &UTXOEntry{
+				amount:         1000000,
+				scriptPubKey:   hexToBytes("76a914ee8bd501094a7d5ca318da2506de35e1cb025ddc88ac"),
+				blockBlueScore: 100001,
+				packedFlags:    0,
+			},
+			serialized: hexToBytes("8b99420700ee8bd501094a7d5ca318da2506de35e1cb025ddc"),
+		},
+	}
+
+	for i, test := range tests {
+		// Ensure the utxo entry serializes to the expected value.
+		gotBytes := serializeUTXOEntry(test.entry)
+		if !bytes.Equal(gotBytes, test.serialized) {
+			t.Errorf("serializeUTXOEntry #%d (%s): mismatched "+
+				"bytes - got %x, want %x", i, test.name,
+				gotBytes, test.serialized)
+			continue
+		}
+
+		// Deserialize to a utxo entry.
+		utxoEntry, err := deserializeUTXOEntry(test.serialized)
+		if err != nil {
+			t.Errorf("deserializeUTXOEntry #%d (%s) unexpected "+
+				"error: %v", i, test.name, err)
+			continue
+		}
+
+		// Ensure the deserialized entry has the same properties as the
+		// ones in the test entry.
+		if utxoEntry.Amount() != test.entry.Amount() {
+			t.Errorf("deserializeUTXOEntry #%d (%s) mismatched "+
+				"amounts: got %d, want %d", i, test.name,
+				utxoEntry.Amount(), test.entry.Amount())
+			continue
+		}
+
+		if !bytes.Equal(utxoEntry.ScriptPubKey(), test.entry.ScriptPubKey()) {
+			t.Errorf("deserializeUTXOEntry #%d (%s) mismatched "+
+				"scripts: got %x, want %x", i, test.name,
+				utxoEntry.ScriptPubKey(), test.entry.ScriptPubKey())
+			continue
+		}
+		if utxoEntry.BlockBlueScore() != test.entry.BlockBlueScore() {
+			t.Errorf("deserializeUTXOEntry #%d (%s) mismatched "+
+				"block blue score: got %d, want %d", i, test.name,
+				utxoEntry.BlockBlueScore(), test.entry.BlockBlueScore())
+			continue
+		}
+		if utxoEntry.IsCoinbase() != test.entry.IsCoinbase() {
+			t.Errorf("deserializeUTXOEntry #%d (%s) mismatched "+
+				"coinbase flag: got %v, want %v", i, test.name,
+				utxoEntry.IsCoinbase(), test.entry.IsCoinbase())
+			continue
+		}
+	}
+}
+
+// TestUtxoEntryDeserializeErrors performs negative tests against deserializing
+// unspent transaction outputs to ensure error paths work as expected.
+func TestUtxoEntryDeserializeErrors(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name       string
+		serialized []byte
+		errType    error
+	}{
+		{
+			name:       "no data after header code",
+			serialized: hexToBytes("02"),
+			errType:    errDeserialize(""),
+		},
+		{
+			name:       "incomplete compressed txout",
+			serialized: hexToBytes("0232"),
+			errType:    errDeserialize(""),
+		},
+	}
+
+	for _, test := range tests {
+		// Ensure the expected error type is returned and the returned
+		// entry is nil.
+		entry, err := deserializeUTXOEntry(test.serialized)
+		if reflect.TypeOf(err) != reflect.TypeOf(test.errType) {
+			t.Errorf("deserializeUTXOEntry (%s): expected error "+
+				"type does not match - got %T, want %T",
+				test.name, err, test.errType)
+			continue
+		}
+		if entry != nil {
+			t.Errorf("deserializeUTXOEntry (%s): returned entry "+
+				"is not nil", test.name)
+			continue
+		}
+	}
+}
+
+// TestDAGStateSerialization ensures serializing and deserializing the
+// DAG state works as expected.
+func TestDAGStateSerialization(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name       string
+		state      *dagState
+		serialized []byte
+	}{
+		{
+			name: "genesis",
+			state: &dagState{
+				TipHashes:         []*daghash.Hash{newHashFromStr("000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f")},
+				LastFinalityPoint: newHashFromStr("000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f"),
+			},
+			serialized: []byte("{\"TipHashes\":[[111,226,140,10,182,241,179,114,193,166,162,70,174,99,247,79,147,30,131,101,225,90,8,156,104,214,25,0,0,0,0,0]],\"LastFinalityPoint\":[111,226,140,10,182,241,179,114,193,166,162,70,174,99,247,79,147,30,131,101,225,90,8,156,104,214,25,0,0,0,0,0]}"),
+		},
+		{
+			name: "block 1",
+			state: &dagState{
+				TipHashes:         []*daghash.Hash{newHashFromStr("00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048")},
+				LastFinalityPoint: newHashFromStr("000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f"),
+			},
+			serialized: []byte("{\"TipHashes\":[[72,96,235,24,191,27,22,32,227,126,148,144,252,138,66,117,20,65,111,215,81,89,171,134,104,142,154,131,0,0,0,0]],\"LastFinalityPoint\":[111,226,140,10,182,241,179,114,193,166,162,70,174,99,247,79,147,30,131,101,225,90,8,156,104,214,25,0,0,0,0,0]}"),
+		},
+	}
+
+	for i, test := range tests {
+		gotBytes, err := serializeDAGState(test.state)
+		if err != nil {
+			t.Errorf("serializeDAGState #%d (%s) "+
+				"unexpected error: %v", i, test.name, err)
+			continue
+		}
+
+		// Ensure the dagState serializes to the expected value.
+		if !bytes.Equal(gotBytes, test.serialized) {
+			t.Errorf("serializeDAGState #%d (%s): mismatched "+
+				"bytes - got %s, want %s", i, test.name,
+				string(gotBytes), string(test.serialized))
+			continue
+		}
+
+		// Ensure the serialized bytes are decoded back to the expected
+		// dagState.
+		state, err := deserializeDAGState(test.serialized)
+		if err != nil {
+			t.Errorf("deserializeDAGState #%d (%s) "+
+				"unexpected error: %v", i, test.name, err)
+			continue
+		}
+		if !reflect.DeepEqual(state, test.state) {
+			t.Errorf("deserializeDAGState #%d (%s) "+
+				"mismatched state - got %v, want %v", i,
+				test.name, state, test.state)
+			continue
+		}
+	}
+}
+
+// TestDAGStateDeserializeErrors performs negative tests against
+// deserializing the DAG state to ensure error paths work as expected.
+func TestDAGStateDeserializeErrors(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name       string
+		serialized []byte
+		errType    error
+	}{
+		{
+			name:       "nothing serialized",
+			serialized: hexToBytes(""),
+			errType:    database.Error{ErrorCode: database.ErrCorruption},
+		},
+		{
+			name:       "corrupted data",
+			serialized: []byte("[[111,226,140,10,182,241,179,114,193,166,162,70,174,99,247,7"),
+			errType:    database.Error{ErrorCode: database.ErrCorruption},
+		},
+	}
+
+	for _, test := range tests {
+		// Ensure the expected error type and code is returned.
+		_, err := deserializeDAGState(test.serialized)
+		if reflect.TypeOf(err) != reflect.TypeOf(test.errType) {
+			t.Errorf("deserializeDAGState (%s): expected "+
+				"error type does not match - got %T, want %T",
+				test.name, err, test.errType)
+			continue
+		}
+		if derr, ok := err.(database.Error); ok {
+			tderr := test.errType.(database.Error)
+			if derr.ErrorCode != tderr.ErrorCode {
+				t.Errorf("deserializeDAGState (%s): "+
+					"wrong error code got: %v, want: %v",
+					test.name, derr.ErrorCode,
+					tderr.ErrorCode)
+				continue
+			}
+		}
+	}
+}
+
+// newHashFromStr converts the passed big-endian hex string into a
+// daghash.Hash. It only differs from the one available in daghash in that
+// it panics in case of an error since it will only (and must only) be
+// called with hard-coded, and therefore known good, hashes.
+func newHashFromStr(hexStr string) *daghash.Hash {
+	hash, err := daghash.NewHashFromStr(hexStr)
+	if err != nil {
+		panic(err)
+	}
+	return hash
+}

blockdag/delayedblockheap.go

@@ -0,0 +1,73 @@
+package blockdag
+
+import (
+	"container/heap"
+)
+
+type baseDelayedBlocksHeap []*delayedBlock
+
+func (h baseDelayedBlocksHeap) Len() int {
+	return len(h)
+}
+func (h baseDelayedBlocksHeap) Swap(i, j int) {
+	h[i], h[j] = h[j], h[i]
+}
+
+func (h *baseDelayedBlocksHeap) Push(x interface{}) {
+	*h = append(*h, x.(*delayedBlock))
+}
+
+func (h *baseDelayedBlocksHeap) Pop() interface{} {
+	oldHeap := *h
+	oldLength := len(oldHeap)
+	popped := oldHeap[oldLength-1]
+	*h = oldHeap[0 : oldLength-1]
+	return popped
+}
+
+func (h baseDelayedBlocksHeap) peek() interface{} {
+	if h.Len() > 0 {
+		return h[h.Len()-1]
+	}
+	return nil
+}
+
+func (h baseDelayedBlocksHeap) Less(i, j int) bool {
+	return h[j].processTime.After(h[i].processTime)
+}
+
+type delayedBlocksHeap struct {
+	baseDelayedBlocksHeap *baseDelayedBlocksHeap
+	impl                  heap.Interface
+}
+
+// newDelayedBlocksHeap initializes and returns a new delayedBlocksHeap
+func newDelayedBlocksHeap() delayedBlocksHeap {
+	baseHeap := &baseDelayedBlocksHeap{}
+	h := delayedBlocksHeap{impl: baseHeap, baseDelayedBlocksHeap: baseHeap}
+	heap.Init(h.impl)
+	return h
+}
+
+// pop removes the block with lowest height from this heap and returns it
+func (dbh delayedBlocksHeap) pop() *delayedBlock {
+	return heap.Pop(dbh.impl).(*delayedBlock)
+}
+
+// Push pushes the block onto the heap
+func (dbh delayedBlocksHeap) Push(block *delayedBlock) {
+	heap.Push(dbh.impl, block)
+}
+
+// Len returns the length of this heap
+func (dbh delayedBlocksHeap) Len() int {
+	return dbh.impl.Len()
+}
+
+// peek returns the topmost element in the queue without poping it
+func (dbh delayedBlocksHeap) peek() *delayedBlock {
+	if dbh.baseDelayedBlocksHeap.peek() == nil {
+		return nil
+	}
+	return dbh.baseDelayedBlocksHeap.peek().(*delayedBlock)
+}

blockdag/difficulty.go

@@ -0,0 +1,52 @@
+// Copyright (c) 2013-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"math/big"
+	"time"
+
+	"github.com/kaspanet/kaspad/util"
+)
+
+// requiredDifficulty calculates the required difficulty for a
+// block given its bluest parent.
+func (dag *BlockDAG) requiredDifficulty(bluestParent *blockNode, newBlockTime time.Time) uint32 {
+	// Genesis block.
+	if bluestParent == nil || bluestParent.blueScore < dag.difficultyAdjustmentWindowSize+1 {
+		return dag.powMaxBits
+	}
+
+	// Fetch window of dag.difficultyAdjustmentWindowSize + 1 so we can have dag.difficultyAdjustmentWindowSize block intervals
+	timestampsWindow := blueBlockWindow(bluestParent, dag.difficultyAdjustmentWindowSize+1)
+	windowMinTimestamp, windowMaxTimeStamp := timestampsWindow.minMaxTimestamps()
+
+	// Remove the last block from the window so to calculate the average target of dag.difficultyAdjustmentWindowSize blocks
+	targetsWindow := timestampsWindow[:dag.difficultyAdjustmentWindowSize]
+
+	// Calculate new target difficulty as:
+	// averageWindowTarget * (windowMinTimestamp / (targetTimePerBlock * windowSize))
+	// The result uses integer division which means it will be slightly
+	// rounded down.
+	newTarget := targetsWindow.averageTarget()
+	newTarget.
+		Mul(newTarget, big.NewInt(windowMaxTimeStamp-windowMinTimestamp)).
+		Div(newTarget, big.NewInt(dag.targetTimePerBlock)).
+		Div(newTarget, big.NewInt(int64(dag.difficultyAdjustmentWindowSize)))
+	if newTarget.Cmp(dag.dagParams.PowMax) > 0 {
+		return dag.powMaxBits
+	}
+	newTargetBits := util.BigToCompact(newTarget)
+	return newTargetBits
+}
+
+// NextRequiredDifficulty calculates the required difficulty for a block that will
+// be built on top of the current tips.
+//
+// This function is safe for concurrent access.
+func (dag *BlockDAG) NextRequiredDifficulty(timestamp time.Time) uint32 {
+	difficulty := dag.requiredDifficulty(dag.virtual.parents.bluest(), timestamp)
+	return difficulty
+}

blockdag/difficulty_test.go

@@ -0,0 +1,208 @@
+// Copyright (c) 2014-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"github.com/kaspanet/kaspad/dagconfig"
+	"math/big"
+	"testing"
+	"time"
+
+	"github.com/kaspanet/kaspad/util"
+)
+
+// TestBigToCompact ensures BigToCompact converts big integers to the expected
+// compact representation.
+func TestBigToCompact(t *testing.T) {
+	tests := []struct {
+		in  int64
+		out uint32
+	}{
+		{0, 0},
+		{-1, 25231360},
+	}
+
+	for x, test := range tests {
+		n := big.NewInt(test.in)
+		r := util.BigToCompact(n)
+		if r != test.out {
+			t.Errorf("TestBigToCompact test #%d failed: got %d want %d\n",
+				x, r, test.out)
+			return
+		}
+	}
+}
+
+// TestCompactToBig ensures CompactToBig converts numbers using the compact
+// representation to the expected big intergers.
+func TestCompactToBig(t *testing.T) {
+	tests := []struct {
+		in  uint32
+		out int64
+	}{
+		{10000000, 0},
+	}
+
+	for x, test := range tests {
+		n := util.CompactToBig(test.in)
+		want := big.NewInt(test.out)
+		if n.Cmp(want) != 0 {
+			t.Errorf("TestCompactToBig test #%d failed: got %d want %d\n",
+				x, n.Int64(), want.Int64())
+			return
+		}
+	}
+}
+
+// TestCalcWork ensures CalcWork calculates the expected work value from values
+// in compact representation.
+func TestCalcWork(t *testing.T) {
+	tests := []struct {
+		in  uint32
+		out int64
+	}{
+		{10000000, 0},
+	}
+
+	for x, test := range tests {
+		bits := uint32(test.in)
+
+		r := util.CalcWork(bits)
+		if r.Int64() != test.out {
+			t.Errorf("TestCalcWork test #%d failed: got %v want %d\n",
+				x, r.Int64(), test.out)
+			return
+		}
+	}
+}
+
+func TestDifficulty(t *testing.T) {
+	params := dagconfig.SimnetParams
+	params.K = 1
+	params.DifficultyAdjustmentWindowSize = 264
+	dag, teardownFunc, err := DAGSetup("TestDifficulty", Config{
+		DAGParams: &params,
+	})
+	if err != nil {
+		t.Fatalf("Failed to setup DAG instance: %v", err)
+	}
+	defer teardownFunc()
+
+	zeroTime := time.Unix(0, 0)
+	addNode := func(parents blockSet, blockTime time.Time) *blockNode {
+		bluestParent := parents.bluest()
+		if blockTime == zeroTime {
+			blockTime = time.Unix(bluestParent.timestamp+1, 0)
+		}
+		block, err := PrepareBlockForTest(dag, parents.hashes(), nil)
+		if err != nil {
+			t.Fatalf("unexpected error in PrepareBlockForTest: %s", err)
+		}
+		block.Header.Timestamp = blockTime
+		block.Header.Bits = dag.requiredDifficulty(bluestParent, blockTime)
+
+		isOrphan, isDelayed, err := dag.ProcessBlock(util.NewBlock(block), BFNoPoWCheck)
+		if err != nil {
+			t.Fatalf("unexpected error in ProcessBlock: %s", err)
+		}
+		if isDelayed {
+			t.Fatalf("block is too far in the future")
+		}
+		if isOrphan {
+			t.Fatalf("block was unexpectedly orphan")
+		}
+		return dag.index.LookupNode(block.BlockHash())
+	}
+	tip := dag.genesis
+	for i := uint64(0); i < dag.difficultyAdjustmentWindowSize; i++ {
+		tip = addNode(setFromSlice(tip), zeroTime)
+		if tip.bits != dag.genesis.bits {
+			t.Fatalf("As long as the bluest parent's blue score is less then the difficulty adjustment window size, the difficulty should be the same as genesis'")
+		}
+	}
+	for i := uint64(0); i < dag.difficultyAdjustmentWindowSize+100; i++ {
+		tip = addNode(setFromSlice(tip), zeroTime)
+		if tip.bits != dag.genesis.bits {
+			t.Fatalf("As long as the block rate remains the same, the difficulty shouldn't change")
+		}
+	}
+	nodeInThePast := addNode(setFromSlice(tip), tip.PastMedianTime(dag))
+	if nodeInThePast.bits != tip.bits {
+		t.Fatalf("The difficulty should only change when nodeInThePast is in the past of a block bluest parent")
+	}
+	tip = nodeInThePast
+
+	tip = addNode(setFromSlice(tip), zeroTime)
+	if tip.bits != nodeInThePast.bits {
+		t.Fatalf("The difficulty should only change when nodeInThePast is in the past of a block bluest parent")
+	}
+	tip = addNode(setFromSlice(tip), zeroTime)
+	if compareBits(tip.bits, nodeInThePast.bits) >= 0 {
+		t.Fatalf("tip.bits should be smaller than nodeInThePast.bits because nodeInThePast increased the block rate, so the difficulty should increase as well")
+	}
+	expectedBits := uint32(0x207f83df)
+	if tip.bits != expectedBits {
+		t.Errorf("tip.bits was expected to be %x but got %x", expectedBits, tip.bits)
+	}
+
+	// Increase block rate to increase difficulty
+	for i := uint64(0); i < dag.difficultyAdjustmentWindowSize; i++ {
+		tip = addNode(setFromSlice(tip), tip.PastMedianTime(dag))
+		if compareBits(tip.bits, tip.parents.bluest().bits) > 0 {
+			t.Fatalf("Because we're increasing the block rate, the difficulty can't decrease")
+		}
+	}
+
+	// Add blocks until difficulty stabilizes
+	lastBits := tip.bits
+	sameBitsCount := uint64(0)
+	for sameBitsCount < dag.difficultyAdjustmentWindowSize+1 {
+		tip = addNode(setFromSlice(tip), zeroTime)
+		if tip.bits == lastBits {
+			sameBitsCount++
+		} else {
+			lastBits = tip.bits
+			sameBitsCount = 0
+		}
+	}
+	slowNode := addNode(setFromSlice(tip), time.Unix(tip.timestamp+2, 0))
+	if slowNode.bits != tip.bits {
+		t.Fatalf("The difficulty should only change when slowNode is in the past of a block bluest parent")
+	}
+
+	tip = slowNode
+
+	tip = addNode(setFromSlice(tip), zeroTime)
+	if tip.bits != slowNode.bits {
+		t.Fatalf("The difficulty should only change when slowNode is in the past of a block bluest parent")
+	}
+	tip = addNode(setFromSlice(tip), zeroTime)
+	if compareBits(tip.bits, slowNode.bits) <= 0 {
+		t.Fatalf("tip.bits should be smaller than slowNode.bits because slowNode decreased the block rate, so the difficulty should decrease as well")
+	}
+
+	splitNode := addNode(setFromSlice(tip), zeroTime)
+	tip = splitNode
+	for i := 0; i < 100; i++ {
+		tip = addNode(setFromSlice(tip), zeroTime)
+	}
+	blueTip := tip
+
+	redChainTip := splitNode
+	for i := 0; i < 10; i++ {
+		redChainTip = addNode(setFromSlice(redChainTip), redChainTip.PastMedianTime(dag))
+	}
+	tipWithRedPast := addNode(setFromSlice(redChainTip, blueTip), zeroTime)
+	tipWithoutRedPast := addNode(setFromSlice(blueTip), zeroTime)
+	if tipWithoutRedPast.bits != tipWithRedPast.bits {
+		t.Fatalf("tipWithoutRedPast.bits should be the same as tipWithRedPast.bits because red blocks shouldn't affect the difficulty")
+	}
+}
+
+func compareBits(a uint32, b uint32) int {
+	aTarget := util.CompactToBig(a)
+	bTarget := util.CompactToBig(b)
+	return aTarget.Cmp(bTarget)
+}

blockdag/doc.go

@@ -0,0 +1,57 @@
+/*
+Package blockdag implements kaspa block handling and DAG selection rules.
+
+The kaspa block handling and DAG selection rules are an integral, and quite
+likely the most important, part of kaspa. At its core, kaspa is a distributed
+consensus of which blocks are valid and which ones will comprise the DAG
+(public ledger) that ultimately determines accepted transactions, so it is
+extremely important that fully validating nodes agree on all rules.
+
+At a high level, this package provides support for inserting new blocks into
+the block DAG according to the aforementioned rules. It includes functionality
+such as rejecting duplicate blocks, ensuring blocks and transactions follow all
+rules, orphan handling, and DAG order along with reorganization.
+
+Since this package does not deal with other kaspa specifics such as network
+communication, it provides a notification system which gives the caller a high
+level of flexibility in how they want to react to certain events such as orphan
+blocks which need their parents requested and newly connected DAG blocks.
+
+Kaspa DAG Processing Overview
+
+Before a block is allowed into the block DAG, it must go through an intensive
+series of validation rules. The following list serves as a general outline of
+those rules to provide some intuition into what is going on under the hood, but
+is by no means exhaustive:
+
+ - Reject duplicate blocks
+ - Perform a series of sanity checks on the block and its transactions such as
+   verifying proof of work, timestamps, number and character of transactions,
+   transaction amounts, script complexity, and merkle root calculations
+ - Save the most recent orphan blocks for a limited time in case their parent
+   blocks become available
+ - Stop processing if the block is an orphan as the rest of the processing
+   depends on the block's position within the block DAG
+ - Perform a series of more thorough checks that depend on the block's position
+   within the block DAG such as verifying block difficulties adhere to
+   difficulty retarget rules, timestamps are after the median of the last
+   several blocks, all transactions are finalized, and
+   block versions are in line with the previous blocks
+ - When a block is being connected to the DAG, perform further checks on the
+   block's transactions such as verifying transaction duplicates, script
+   complexity for the combination of connected scripts, coinbase maturity,
+   double spends, and connected transaction values
+ - Run the transaction scripts to verify the spender is allowed to spend the
+   coins
+ - Insert the block into the block database
+
+Errors
+
+Errors returned by this package are either the raw errors provided by underlying
+calls or of type blockdag.RuleError. This allows the caller to differentiate
+between unexpected errors, such as database errors, versus errors due to rule
+violations through type assertions. In addition, callers can programmatically
+determine the specific rule violation by examining the ErrorCode field of the
+type asserted blockdag.RuleError.
+*/
+package blockdag

blockdag/error.go

@@ -0,0 +1,295 @@
+// Copyright (c) 2014-2016 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"fmt"
+)
+
+// DeploymentError identifies an error that indicates a deployment ID was
+// specified that does not exist.
+type DeploymentError uint32
+
+// Error returns the assertion error as a human-readable string and satisfies
+// the error interface.
+func (e DeploymentError) Error() string {
+	return fmt.Sprintf("deployment ID %d does not exist", uint32(e))
+}
+
+// AssertError identifies an error that indicates an internal code consistency
+// issue and should be treated as a critical and unrecoverable error.
+type AssertError string
+
+// Error returns the assertion error as a human-readable string and satisfies
+// the error interface.
+func (e AssertError) Error() string {
+	return "assertion failed: " + string(e)
+}
+
+// ErrorCode identifies a kind of error.
+type ErrorCode int
+
+// These constants are used to identify a specific RuleError.
+const (
+	// ErrDuplicateBlock indicates a block with the same hash already
+	// exists.
+	ErrDuplicateBlock ErrorCode = iota
+
+	// ErrBlockMassTooHigh indicates the mass of a block exceeds the maximum
+	// allowed limits.
+	ErrBlockMassTooHigh
+
+	// ErrBlockVersionTooOld indicates the block version is too old and is
+	// no longer accepted since the majority of the network has upgraded
+	// to a newer version.
+	ErrBlockVersionTooOld
+
+	// ErrInvalidTime indicates the time in the passed block has a precision
+	// that is more than one second. The DAG consensus rules require
+	// timestamps to have a maximum precision of one second.
+	ErrInvalidTime
+
+	// ErrTimeTooOld indicates the time is either before the median time of
+	// the last several blocks per the DAG consensus rules.
+	ErrTimeTooOld
+
+	// ErrTimeTooNew indicates the time is too far in the future as compared
+	// the current time.
+	ErrTimeTooNew
+
+	// ErrNoParents indicates that the block is missing parents
+	ErrNoParents
+
+	// ErrWrongParentsOrder indicates that the block's parents are not ordered by hash, as expected
+	ErrWrongParentsOrder
+
+	// ErrDifficultyTooLow indicates the difficulty for the block is lower
+	// than the difficulty required.
+	ErrDifficultyTooLow
+
+	// ErrUnexpectedDifficulty indicates specified bits do not align with
+	// the expected value either because it doesn't match the calculated
+	// valued based on difficulty regarted rules or it is out of the valid
+	// range.
+	ErrUnexpectedDifficulty
+
+	// ErrHighHash indicates the block does not hash to a value which is
+	// lower than the required target difficultly.
+	ErrHighHash
+
+	// ErrBadMerkleRoot indicates the calculated merkle root does not match
+	// the expected value.
+	ErrBadMerkleRoot
+
+	// ErrBadUTXOCommitment indicates the calculated UTXO commitment does not match
+	// the expected value.
+	ErrBadUTXOCommitment
+
+	// ErrFinalityPointTimeTooOld indicates a block has a timestamp before the
+	// last finality point.
+	ErrFinalityPointTimeTooOld
+
+	// ErrNoTransactions indicates the block does not have a least one
+	// transaction. A valid block must have at least the coinbase
+	// transaction.
+	ErrNoTransactions
+
+	// ErrNoTxInputs indicates a transaction does not have any inputs. A
+	// valid transaction must have at least one input.
+	ErrNoTxInputs
+
+	// ErrTxMassTooHigh indicates the mass of a transaction exceeds the maximum
+	// allowed limits.
+	ErrTxMassTooHigh
+
+	// ErrBadTxOutValue indicates an output value for a transaction is
+	// invalid in some way such as being out of range.
+	ErrBadTxOutValue
+
+	// ErrDuplicateTxInputs indicates a transaction references the same
+	// input more than once.
+	ErrDuplicateTxInputs
+
+	// ErrBadTxInput indicates a transaction input is invalid in some way
+	// such as referencing a previous transaction outpoint which is out of
+	// range or not referencing one at all.
+	ErrBadTxInput
+
+	// ErrMissingTxOut indicates a transaction output referenced by an input
+	// either does not exist or has already been spent.
+	ErrMissingTxOut
+
+	// ErrUnfinalizedTx indicates a transaction has not been finalized.
+	// A valid block may only contain finalized transactions.
+	ErrUnfinalizedTx
+
+	// ErrDuplicateTx indicates a block contains an identical transaction
+	// (or at least two transactions which hash to the same value). A
+	// valid block may only contain unique transactions.
+	ErrDuplicateTx
+
+	// ErrOverwriteTx indicates a block contains a transaction that has
+	// the same hash as a previous transaction which has not been fully
+	// spent.
+	ErrOverwriteTx
+
+	// ErrImmatureSpend indicates a transaction is attempting to spend a
+	// coinbase that has not yet reached the required maturity.
+	ErrImmatureSpend
+
+	// ErrSpendTooHigh indicates a transaction is attempting to spend more
+	// value than the sum of all of its inputs.
+	ErrSpendTooHigh
+
+	// ErrBadFees indicates the total fees for a block are invalid due to
+	// exceeding the maximum possible value.
+	ErrBadFees
+
+	// ErrTooManySigOps indicates the total number of signature operations
+	// for a transaction or block exceed the maximum allowed limits.
+	ErrTooManySigOps
+
+	// ErrFirstTxNotCoinbase indicates the first transaction in a block
+	// is not a coinbase transaction.
+	ErrFirstTxNotCoinbase
+
+	// ErrMultipleCoinbases indicates a block contains more than one
+	// coinbase transaction.
+	ErrMultipleCoinbases
+
+	// ErrBadCoinbasePayloadLen indicates the length of the payload
+	// for a coinbase transaction is too high.
+	ErrBadCoinbasePayloadLen
+
+	// ErrBadCoinbaseTransaction indicates that the block's coinbase transaction is not build as expected
+	ErrBadCoinbaseTransaction
+
+	// ErrScriptMalformed indicates a transaction script is malformed in
+	// some way. For example, it might be longer than the maximum allowed
+	// length or fail to parse.
+	ErrScriptMalformed
+
+	// ErrScriptValidation indicates the result of executing transaction
+	// script failed. The error covers any failure when executing scripts
+	// such signature verification failures and execution past the end of
+	// the stack.
+	ErrScriptValidation
+
+	// ErrParentBlockUnknown indicates that the parent block is not known.
+	ErrParentBlockUnknown
+
+	// ErrInvalidAncestorBlock indicates that an ancestor of this block has
+	// already failed validation.
+	ErrInvalidAncestorBlock
+
+	// ErrParentBlockNotCurrentTips indicates that the block's parents are not the
+	// current tips. This is not a block validation rule, but is required
+	// for block proposals submitted via getblocktemplate RPC.
+	ErrParentBlockNotCurrentTips
+
+	// ErrWithDiff indicates that there was an error with UTXOSet.WithDiff
+	ErrWithDiff
+
+	// ErrFinality indicates that a block doesn't adhere to the finality rules
+	ErrFinality
+
+	// ErrTransactionsNotSorted indicates that transactions in block are not
+	// sorted by subnetwork
+	ErrTransactionsNotSorted
+
+	// ErrInvalidGas transaction wants to use more GAS than allowed
+	// by subnetwork
+	ErrInvalidGas
+
+	// ErrInvalidPayload transaction includes a payload in a subnetwork that doesn't allow
+	// a Payload
+	ErrInvalidPayload
+
+	// ErrInvalidPayloadHash invalid hash of transaction's payload
+	ErrInvalidPayloadHash
+
+	// ErrSubnetwork indicates that a block doesn't adhere to the subnetwork
+	// registry rules
+	ErrSubnetworkRegistry
+
+	// ErrInvalidParentsRelation indicates that one of the parents of a block
+	// is also an ancestor of another parent
+	ErrInvalidParentsRelation
+)
+
+// Map of ErrorCode values back to their constant names for pretty printing.
+var errorCodeStrings = map[ErrorCode]string{
+	ErrDuplicateBlock:            "ErrDuplicateBlock",
+	ErrBlockMassTooHigh:          "ErrBlockMassTooHigh",
+	ErrBlockVersionTooOld:        "ErrBlockVersionTooOld",
+	ErrInvalidTime:               "ErrInvalidTime",
+	ErrTimeTooOld:                "ErrTimeTooOld",
+	ErrTimeTooNew:                "ErrTimeTooNew",
+	ErrNoParents:                 "ErrNoParents",
+	ErrWrongParentsOrder:         "ErrWrongParentsOrder",
+	ErrDifficultyTooLow:          "ErrDifficultyTooLow",
+	ErrUnexpectedDifficulty:      "ErrUnexpectedDifficulty",
+	ErrHighHash:                  "ErrHighHash",
+	ErrBadMerkleRoot:             "ErrBadMerkleRoot",
+	ErrFinalityPointTimeTooOld:   "ErrFinalityPointTimeTooOld",
+	ErrNoTransactions:            "ErrNoTransactions",
+	ErrNoTxInputs:                "ErrNoTxInputs",
+	ErrTxMassTooHigh:             "ErrTxMassTooHigh",
+	ErrBadTxOutValue:             "ErrBadTxOutValue",
+	ErrDuplicateTxInputs:         "ErrDuplicateTxInputs",
+	ErrBadTxInput:                "ErrBadTxInput",
+	ErrMissingTxOut:              "ErrMissingTxOut",
+	ErrUnfinalizedTx:             "ErrUnfinalizedTx",
+	ErrDuplicateTx:               "ErrDuplicateTx",
+	ErrOverwriteTx:               "ErrOverwriteTx",
+	ErrImmatureSpend:             "ErrImmatureSpend",
+	ErrSpendTooHigh:              "ErrSpendTooHigh",
+	ErrBadFees:                   "ErrBadFees",
+	ErrTooManySigOps:             "ErrTooManySigOps",
+	ErrFirstTxNotCoinbase:        "ErrFirstTxNotCoinbase",
+	ErrMultipleCoinbases:         "ErrMultipleCoinbases",
+	ErrBadCoinbasePayloadLen:     "ErrBadCoinbasePayloadLen",
+	ErrBadCoinbaseTransaction:    "ErrBadCoinbaseTransaction",
+	ErrScriptMalformed:           "ErrScriptMalformed",
+	ErrScriptValidation:          "ErrScriptValidation",
+	ErrParentBlockUnknown:        "ErrParentBlockUnknown",
+	ErrInvalidAncestorBlock:      "ErrInvalidAncestorBlock",
+	ErrParentBlockNotCurrentTips: "ErrParentBlockNotCurrentTips",
+	ErrWithDiff:                  "ErrWithDiff",
+	ErrFinality:                  "ErrFinality",
+	ErrTransactionsNotSorted:     "ErrTransactionsNotSorted",
+	ErrInvalidGas:                "ErrInvalidGas",
+	ErrInvalidPayload:            "ErrInvalidPayload",
+	ErrInvalidPayloadHash:        "ErrInvalidPayloadHash",
+	ErrInvalidParentsRelation:    "ErrInvalidParentsRelation",
+}
+
+// String returns the ErrorCode as a human-readable name.
+func (e ErrorCode) String() string {
+	if s := errorCodeStrings[e]; s != "" {
+		return s
+	}
+	return fmt.Sprintf("Unknown ErrorCode (%d)", int(e))
+}
+
+// RuleError identifies a rule violation. It is used to indicate that
+// processing of a block or transaction failed due to one of the many validation
+// rules. The caller can use type assertions to determine if a failure was
+// specifically due to a rule violation and access the ErrorCode field to
+// ascertain the specific reason for the rule violation.
+type RuleError struct {
+	ErrorCode   ErrorCode // Describes the kind of error
+	Description string    // Human readable description of the issue
+}
+
+// Error satisfies the error interface and prints human-readable errors.
+func (e RuleError) Error() string {
+	return e.Description
+}
+
+// ruleError creates an RuleError given a set of arguments.
+func ruleError(c ErrorCode, desc string) RuleError {
+	return RuleError{ErrorCode: c, Description: desc}
+}

blockdag/error_test.go

@@ -0,0 +1,143 @@
+// Copyright (c) 2014-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"fmt"
+	"testing"
+)
+
+// TestErrorCodeStringer tests the stringized output for the ErrorCode type.
+func TestErrorCodeStringer(t *testing.T) {
+	tests := []struct {
+		in   ErrorCode
+		want string
+	}{
+		{ErrDuplicateBlock, "ErrDuplicateBlock"},
+		{ErrBlockMassTooHigh, "ErrBlockMassTooHigh"},
+		{ErrBlockVersionTooOld, "ErrBlockVersionTooOld"},
+		{ErrInvalidTime, "ErrInvalidTime"},
+		{ErrTimeTooOld, "ErrTimeTooOld"},
+		{ErrTimeTooNew, "ErrTimeTooNew"},
+		{ErrNoParents, "ErrNoParents"},
+		{ErrWrongParentsOrder, "ErrWrongParentsOrder"},
+		{ErrDifficultyTooLow, "ErrDifficultyTooLow"},
+		{ErrUnexpectedDifficulty, "ErrUnexpectedDifficulty"},
+		{ErrHighHash, "ErrHighHash"},
+		{ErrBadMerkleRoot, "ErrBadMerkleRoot"},
+		{ErrFinalityPointTimeTooOld, "ErrFinalityPointTimeTooOld"},
+		{ErrNoTransactions, "ErrNoTransactions"},
+		{ErrNoTxInputs, "ErrNoTxInputs"},
+		{ErrTxMassTooHigh, "ErrTxMassTooHigh"},
+		{ErrBadTxOutValue, "ErrBadTxOutValue"},
+		{ErrDuplicateTxInputs, "ErrDuplicateTxInputs"},
+		{ErrBadTxInput, "ErrBadTxInput"},
+		{ErrMissingTxOut, "ErrMissingTxOut"},
+		{ErrUnfinalizedTx, "ErrUnfinalizedTx"},
+		{ErrDuplicateTx, "ErrDuplicateTx"},
+		{ErrOverwriteTx, "ErrOverwriteTx"},
+		{ErrImmatureSpend, "ErrImmatureSpend"},
+		{ErrSpendTooHigh, "ErrSpendTooHigh"},
+		{ErrBadFees, "ErrBadFees"},
+		{ErrTooManySigOps, "ErrTooManySigOps"},
+		{ErrFirstTxNotCoinbase, "ErrFirstTxNotCoinbase"},
+		{ErrMultipleCoinbases, "ErrMultipleCoinbases"},
+		{ErrBadCoinbasePayloadLen, "ErrBadCoinbasePayloadLen"},
+		{ErrBadCoinbaseTransaction, "ErrBadCoinbaseTransaction"},
+		{ErrScriptMalformed, "ErrScriptMalformed"},
+		{ErrScriptValidation, "ErrScriptValidation"},
+		{ErrParentBlockUnknown, "ErrParentBlockUnknown"},
+		{ErrInvalidAncestorBlock, "ErrInvalidAncestorBlock"},
+		{ErrParentBlockNotCurrentTips, "ErrParentBlockNotCurrentTips"},
+		{ErrWithDiff, "ErrWithDiff"},
+		{ErrFinality, "ErrFinality"},
+		{ErrTransactionsNotSorted, "ErrTransactionsNotSorted"},
+		{ErrInvalidGas, "ErrInvalidGas"},
+		{ErrInvalidPayload, "ErrInvalidPayload"},
+		{ErrInvalidPayloadHash, "ErrInvalidPayloadHash"},
+		{ErrInvalidParentsRelation, "ErrInvalidParentsRelation"},
+		{0xffff, "Unknown ErrorCode (65535)"},
+	}
+
+	t.Logf("Running %d tests", len(tests))
+	for i, test := range tests {
+		result := test.in.String()
+		if result != test.want {
+			t.Errorf("String #%d\n got: %s want: %s", i, result,
+				test.want)
+			continue
+		}
+	}
+}
+
+// TestRuleError tests the error output for the RuleError type.
+func TestRuleError(t *testing.T) {
+	tests := []struct {
+		in   RuleError
+		want string
+	}{
+		{
+			RuleError{Description: "duplicate block"},
+			"duplicate block",
+		},
+		{
+			RuleError{Description: "human-readable error"},
+			"human-readable error",
+		},
+	}
+
+	t.Logf("Running %d tests", len(tests))
+	for i, test := range tests {
+		result := test.in.Error()
+		if result != test.want {
+			t.Errorf("Error #%d\n got: %s want: %s", i, result,
+				test.want)
+			continue
+		}
+	}
+}
+
+// TestDeploymentError tests the stringized output for the DeploymentError type.
+func TestDeploymentError(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		in   DeploymentError
+		want string
+	}{
+		{
+			DeploymentError(0),
+			"deployment ID 0 does not exist",
+		},
+		{
+			DeploymentError(10),
+			"deployment ID 10 does not exist",
+		},
+		{
+			DeploymentError(123),
+			"deployment ID 123 does not exist",
+		},
+	}
+
+	t.Logf("Running %d tests", len(tests))
+	for i, test := range tests {
+		result := test.in.Error()
+		if result != test.want {
+			t.Errorf("Error #%d\n got: %s want: %s", i, result,
+				test.want)
+			continue
+		}
+	}
+}
+
+func TestAssertError(t *testing.T) {
+	message := "abc 123"
+	err := AssertError(message)
+	expectedMessage := fmt.Sprintf("assertion failed: %s", message)
+	if expectedMessage != err.Error() {
+		t.Errorf("Unexpected AssertError message. "+
+			"Got: %s, want: %s", err.Error(), expectedMessage)
+	}
+}

blockdag/external_dag_test.go

@@ -0,0 +1,566 @@
+package blockdag_test
+
+import (
+	"fmt"
+	"github.com/pkg/errors"
+	"math"
+	"testing"
+
+	"github.com/kaspanet/kaspad/util/subnetworkid"
+
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/util/testtools"
+
+	"github.com/kaspanet/kaspad/blockdag"
+	"github.com/kaspanet/kaspad/dagconfig"
+	"github.com/kaspanet/kaspad/mining"
+	"github.com/kaspanet/kaspad/txscript"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/kaspanet/kaspad/wire"
+)
+
+// TestFinality checks that the finality mechanism works as expected.
+// This is how the flow goes:
+// 1) We build a chain of params.FinalityInterval blocks and call its tip altChainTip.
+// 2) We build another chain (let's call it mainChain) of 2 * params.FinalityInterval
+// blocks, which points to genesis, and then we check that the block in that
+// chain with height of params.FinalityInterval is marked as finality point (This is
+// very predictable, because the blue score of each new block in a chain is the
+// parents plus one).
+// 3) We make a new child to block with height (2 * params.FinalityInterval - 1)
+// in mainChain, and we check that connecting it to the DAG
+// doesn't affect the last finality point.
+// 4) We make a block that points to genesis, and check that it
+// gets rejected because its blue score is lower then the last finality
+// point.
+// 5) We make a block that points to altChainTip, and check that it
+// gets rejected because it doesn't have the last finality point in
+// its selected parent chain.
+func TestFinality(t *testing.T) {
+	params := dagconfig.SimnetParams
+	params.K = 1
+	params.FinalityInterval = 100
+	dag, teardownFunc, err := blockdag.DAGSetup("TestFinality", blockdag.Config{
+		DAGParams: &params,
+	})
+	if err != nil {
+		t.Fatalf("Failed to setup DAG instance: %v", err)
+	}
+	defer teardownFunc()
+	buildNodeToDag := func(parentHashes []*daghash.Hash) (*util.Block, error) {
+		msgBlock, err := mining.PrepareBlockForTest(dag, &params, parentHashes, nil, false)
+		if err != nil {
+			return nil, err
+		}
+		block := util.NewBlock(msgBlock)
+
+		isOrphan, isDelayed, err := dag.ProcessBlock(block, blockdag.BFNoPoWCheck)
+		if err != nil {
+			return nil, err
+		}
+		if isDelayed {
+			return nil, errors.Errorf("ProcessBlock: block " +
+				"is too far in the future")
+		}
+		if isOrphan {
+			return nil, errors.Errorf("ProcessBlock: unexpected returned orphan block")
+		}
+
+		return block, nil
+	}
+
+	genesis := util.NewBlock(params.GenesisBlock)
+	currentNode := genesis
+
+	// First we build a chain of params.FinalityInterval blocks for future use
+	for i := uint64(0); i < params.FinalityInterval; i++ {
+		currentNode, err = buildNodeToDag([]*daghash.Hash{currentNode.Hash()})
+		if err != nil {
+			t.Fatalf("TestFinality: buildNodeToDag unexpectedly returned an error: %v", err)
+		}
+	}
+
+	altChainTip := currentNode
+
+	// Now we build a new chain of 2 * params.FinalityInterval blocks, pointed to genesis, and
+	// we expect the block with height 1 * params.FinalityInterval to be the last finality point
+	currentNode = genesis
+	for i := uint64(0); i < params.FinalityInterval; i++ {
+		currentNode, err = buildNodeToDag([]*daghash.Hash{currentNode.Hash()})
+		if err != nil {
+			t.Fatalf("TestFinality: buildNodeToDag unexpectedly returned an error: %v", err)
+		}
+	}
+
+	expectedFinalityPoint := currentNode
+
+	for i := uint64(0); i < params.FinalityInterval; i++ {
+		currentNode, err = buildNodeToDag([]*daghash.Hash{currentNode.Hash()})
+		if err != nil {
+			t.Fatalf("TestFinality: buildNodeToDag unexpectedly returned an error: %v", err)
+		}
+	}
+
+	if !dag.LastFinalityPointHash().IsEqual(expectedFinalityPoint.Hash()) {
+		t.Errorf("TestFinality: dag.lastFinalityPoint expected to be %v but got %v", expectedFinalityPoint, dag.LastFinalityPointHash())
+	}
+
+	// Here we check that even if we create a parallel tip (a new tip with
+	// the same parents as the current one) with the same blue score as the
+	// current tip, it still won't affect the last finality point.
+	_, err = buildNodeToDag(currentNode.MsgBlock().Header.ParentHashes)
+	if err != nil {
+		t.Fatalf("TestFinality: buildNodeToDag unexpectedly returned an error: %v", err)
+	}
+	if !dag.LastFinalityPointHash().IsEqual(expectedFinalityPoint.Hash()) {
+		t.Errorf("TestFinality: dag.lastFinalityPoint was unexpectly changed")
+	}
+
+	// Here we check that a block with lower blue score than the last finality
+	// point will get rejected
+	fakeCoinbaseTx, err := dag.NextBlockCoinbaseTransaction(nil, nil)
+	if err != nil {
+		t.Errorf("NextBlockCoinbaseTransaction: %s", err)
+	}
+	merkleRoot := blockdag.BuildHashMerkleTreeStore([]*util.Tx{fakeCoinbaseTx}).Root()
+	beforeFinalityBlock := wire.NewMsgBlock(&wire.BlockHeader{
+		Version:              0x10000000,
+		ParentHashes:         []*daghash.Hash{genesis.Hash()},
+		HashMerkleRoot:       merkleRoot,
+		AcceptedIDMerkleRoot: &daghash.ZeroHash,
+		UTXOCommitment:       &daghash.ZeroHash,
+		Timestamp:            dag.SelectedTipHeader().Timestamp,
+		Bits:                 genesis.MsgBlock().Header.Bits,
+	})
+	beforeFinalityBlock.AddTransaction(fakeCoinbaseTx.MsgTx())
+	_, _, err = dag.ProcessBlock(util.NewBlock(beforeFinalityBlock), blockdag.BFNoPoWCheck)
+	if err == nil {
+		t.Errorf("TestFinality: buildNodeToDag expected an error but got <nil>")
+	}
+	rErr, ok := err.(blockdag.RuleError)
+	if ok {
+		if rErr.ErrorCode != blockdag.ErrFinality {
+			t.Errorf("TestFinality: buildNodeToDag expected an error with code %v but instead got %v", blockdag.ErrFinality, rErr.ErrorCode)
+		}
+	} else {
+		t.Errorf("TestFinality: buildNodeToDag got unexpected error: %v", err)
+	}
+
+	// Here we check that a block that doesn't have the last finality point in
+	// its selected parent chain will get rejected
+	_, err = buildNodeToDag([]*daghash.Hash{altChainTip.Hash()})
+	if err == nil {
+		t.Errorf("TestFinality: buildNodeToDag expected an error but got <nil>")
+	}
+	rErr, ok = err.(blockdag.RuleError)
+	if ok {
+		if rErr.ErrorCode != blockdag.ErrFinality {
+			t.Errorf("TestFinality: buildNodeToDag expected an error with code %v but instead got %v", blockdag.ErrFinality, rErr.ErrorCode)
+		}
+	} else {
+		t.Errorf("TestFinality: buildNodeToDag got unexpected error: %v", rErr)
+	}
+}
+
+// TestFinalityInterval tests that the finality interval is
+// smaller then wire.MaxInvPerMsg, so when a peer receives
+// a getblocks message it should always be able to send
+// all the necessary invs.
+func TestFinalityInterval(t *testing.T) {
+	netParams := []*dagconfig.Params{
+		&dagconfig.MainnetParams,
+		&dagconfig.TestnetParams,
+		&dagconfig.DevnetParams,
+		&dagconfig.RegressionNetParams,
+		&dagconfig.SimnetParams,
+	}
+	for _, params := range netParams {
+		if params.FinalityInterval > wire.MaxInvPerMsg {
+			t.Errorf("FinalityInterval in %s should be lower or equal to wire.MaxInvPerMsg", params.Name)
+		}
+	}
+}
+
+// TestSubnetworkRegistry tests the full subnetwork registry flow
+func TestSubnetworkRegistry(t *testing.T) {
+	params := dagconfig.SimnetParams
+	params.K = 1
+	params.BlockCoinbaseMaturity = 0
+	dag, teardownFunc, err := blockdag.DAGSetup("TestSubnetworkRegistry", blockdag.Config{
+		DAGParams: &params,
+	})
+	if err != nil {
+		t.Fatalf("Failed to setup DAG instance: %v", err)
+	}
+	defer teardownFunc()
+
+	gasLimit := uint64(12345)
+	subnetworkID, err := testtools.RegisterSubnetworkForTest(dag, &params, gasLimit)
+	if err != nil {
+		t.Fatalf("could not register network: %s", err)
+	}
+	limit, err := dag.SubnetworkStore.GasLimit(subnetworkID)
+	if err != nil {
+		t.Fatalf("could not retrieve gas limit: %s", err)
+	}
+	if limit != gasLimit {
+		t.Fatalf("unexpected gas limit. want: %d, got: %d", gasLimit, limit)
+	}
+}
+
+func TestChainedTransactions(t *testing.T) {
+	params := dagconfig.SimnetParams
+	params.BlockCoinbaseMaturity = 0
+	// Create a new database and dag instance to run tests against.
+	dag, teardownFunc, err := blockdag.DAGSetup("TestChainedTransactions", blockdag.Config{
+		DAGParams: &params,
+	})
+	if err != nil {
+		t.Fatalf("Failed to setup dag instance: %v", err)
+	}
+	defer teardownFunc()
+
+	block1, err := mining.PrepareBlockForTest(dag, &params, []*daghash.Hash{params.GenesisHash}, nil, false)
+	if err != nil {
+		t.Fatalf("PrepareBlockForTest: %v", err)
+	}
+	isOrphan, isDelayed, err := dag.ProcessBlock(util.NewBlock(block1), blockdag.BFNoPoWCheck)
+	if err != nil {
+		t.Fatalf("ProcessBlock: %v", err)
+	}
+	if isDelayed {
+		t.Fatalf("ProcessBlock: block1 " +
+			"is too far in the future")
+	}
+	if isOrphan {
+		t.Fatalf("ProcessBlock: block1 got unexpectedly orphaned")
+	}
+	cbTx := block1.Transactions[0]
+
+	signatureScript, err := txscript.PayToScriptHashSignatureScript(blockdag.OpTrueScript, nil)
+	if err != nil {
+		t.Fatalf("Failed to build signature script: %s", err)
+	}
+	txIn := &wire.TxIn{
+		PreviousOutpoint: wire.Outpoint{TxID: *cbTx.TxID(), Index: 0},
+		SignatureScript:  signatureScript,
+		Sequence:         wire.MaxTxInSequenceNum,
+	}
+	txOut := &wire.TxOut{
+		ScriptPubKey: blockdag.OpTrueScript,
+		Value:        uint64(1),
+	}
+	tx := wire.NewNativeMsgTx(wire.TxVersion, []*wire.TxIn{txIn}, []*wire.TxOut{txOut})
+
+	chainedTxIn := &wire.TxIn{
+		PreviousOutpoint: wire.Outpoint{TxID: *tx.TxID(), Index: 0},
+		SignatureScript:  signatureScript,
+		Sequence:         wire.MaxTxInSequenceNum,
+	}
+
+	scriptPubKey, err := txscript.PayToScriptHashScript(blockdag.OpTrueScript)
+	if err != nil {
+		t.Fatalf("Failed to build public key script: %s", err)
+	}
+	chainedTxOut := &wire.TxOut{
+		ScriptPubKey: scriptPubKey,
+		Value:        uint64(1),
+	}
+	chainedTx := wire.NewNativeMsgTx(wire.TxVersion, []*wire.TxIn{chainedTxIn}, []*wire.TxOut{chainedTxOut})
+
+	block2, err := mining.PrepareBlockForTest(dag, &params, []*daghash.Hash{block1.BlockHash()}, []*wire.MsgTx{tx, chainedTx}, true)
+	if err != nil {
+		t.Fatalf("PrepareBlockForTest: %v", err)
+	}
+
+	//Checks that dag.ProcessBlock fails because we don't allow a transaction to spend another transaction from the same block
+	isOrphan, isDelayed, err = dag.ProcessBlock(util.NewBlock(block2), blockdag.BFNoPoWCheck)
+	if err == nil {
+		t.Errorf("ProcessBlock expected an error")
+	} else if rErr, ok := err.(blockdag.RuleError); ok {
+		if rErr.ErrorCode != blockdag.ErrMissingTxOut {
+			t.Errorf("ProcessBlock expected an %v error code but got %v", blockdag.ErrMissingTxOut, rErr.ErrorCode)
+		}
+	} else {
+		t.Errorf("ProcessBlock expected a blockdag.RuleError but got %v", err)
+	}
+	if isDelayed {
+		t.Fatalf("ProcessBlock: block2 " +
+			"is too far in the future")
+	}
+	if isOrphan {
+		t.Errorf("ProcessBlock: block2 got unexpectedly orphaned")
+	}
+
+	nonChainedTxIn := &wire.TxIn{
+		PreviousOutpoint: wire.Outpoint{TxID: *cbTx.TxID(), Index: 0},
+		SignatureScript:  signatureScript,
+		Sequence:         wire.MaxTxInSequenceNum,
+	}
+	nonChainedTxOut := &wire.TxOut{
+		ScriptPubKey: scriptPubKey,
+		Value:        uint64(1),
+	}
+	nonChainedTx := wire.NewNativeMsgTx(wire.TxVersion, []*wire.TxIn{nonChainedTxIn}, []*wire.TxOut{nonChainedTxOut})
+
+	block3, err := mining.PrepareBlockForTest(dag, &params, []*daghash.Hash{block1.BlockHash()}, []*wire.MsgTx{nonChainedTx}, false)
+	if err != nil {
+		t.Fatalf("PrepareBlockForTest: %v", err)
+	}
+
+	//Checks that dag.ProcessBlock doesn't fail because all of its transaction are dependant on transactions from previous blocks
+	isOrphan, isDelayed, err = dag.ProcessBlock(util.NewBlock(block3), blockdag.BFNoPoWCheck)
+	if err != nil {
+		t.Errorf("ProcessBlock: %v", err)
+	}
+	if isDelayed {
+		t.Fatalf("ProcessBlock: block3 " +
+			"is too far in the future")
+	}
+	if isOrphan {
+		t.Errorf("ProcessBlock: block3 got unexpectedly orphaned")
+	}
+}
+
+// TestOrderInDiffFromAcceptanceData makes sure that the order of transactions in
+// dag.diffFromAcceptanceData is such that if txA is spent by txB then txA is processed
+// before txB.
+func TestOrderInDiffFromAcceptanceData(t *testing.T) {
+	// Create a new database and DAG instance to run tests against.
+	params := dagconfig.SimnetParams
+	params.K = math.MaxUint8
+	dag, teardownFunc, err := blockdag.DAGSetup("TestOrderInDiffFromAcceptanceData", blockdag.Config{
+		DAGParams: &params,
+	})
+	if err != nil {
+		t.Fatalf("Failed to setup DAG instance: %v", err)
+	}
+	defer teardownFunc()
+	dag.TestSetCoinbaseMaturity(0)
+
+	createBlock := func(previousBlock *util.Block) *util.Block {
+		// Prepare a transaction that spends the previous block's coinbase transaction
+		var txs []*wire.MsgTx
+		if !previousBlock.IsGenesis() {
+			previousCoinbaseTx := previousBlock.MsgBlock().Transactions[0]
+			signatureScript, err := txscript.PayToScriptHashSignatureScript(blockdag.OpTrueScript, nil)
+			if err != nil {
+				t.Fatalf("TestOrderInDiffFromAcceptanceData: Failed to build signature script: %s", err)
+			}
+			txIn := &wire.TxIn{
+				PreviousOutpoint: wire.Outpoint{TxID: *previousCoinbaseTx.TxID(), Index: 0},
+				SignatureScript:  signatureScript,
+				Sequence:         wire.MaxTxInSequenceNum,
+			}
+			txOut := &wire.TxOut{
+				ScriptPubKey: blockdag.OpTrueScript,
+				Value:        uint64(1),
+			}
+			txs = append(txs, wire.NewNativeMsgTx(wire.TxVersion, []*wire.TxIn{txIn}, []*wire.TxOut{txOut}))
+		}
+
+		// Create the block
+		msgBlock, err := mining.PrepareBlockForTest(dag, &params, []*daghash.Hash{previousBlock.Hash()}, txs, false)
+		if err != nil {
+			t.Fatalf("TestOrderInDiffFromAcceptanceData: Failed to prepare block: %s", err)
+		}
+
+		// Add the block to the DAG
+		newBlock := util.NewBlock(msgBlock)
+		isOrphan, isDelayed, err := dag.ProcessBlock(newBlock, blockdag.BFNoPoWCheck)
+		if err != nil {
+			t.Errorf("TestOrderInDiffFromAcceptanceData: %s", err)
+		}
+		if isDelayed {
+			t.Fatalf("TestOrderInDiffFromAcceptanceData: block is too far in the future")
+		}
+		if isOrphan {
+			t.Fatalf("TestOrderInDiffFromAcceptanceData: block got unexpectedly orphaned")
+		}
+		return newBlock
+	}
+
+	// Create two block chains starting from the genesis block. Every time a block is added
+	// one of the chains is selected as the selected parent chain while all the blocks in
+	// the other chain (and their transactions) get accepted by the new virtual. If the
+	// transactions in the non-selected parent chain get processed in the wrong order then
+	// diffFromAcceptanceData panics.
+	blockAmountPerChain := 100
+	chainATip := util.NewBlock(params.GenesisBlock)
+	chainBTip := chainATip
+	for i := 0; i < blockAmountPerChain; i++ {
+		chainATip = createBlock(chainATip)
+		chainBTip = createBlock(chainBTip)
+	}
+}
+
+// TestGasLimit tests the gas limit rules
+func TestGasLimit(t *testing.T) {
+	params := dagconfig.SimnetParams
+	params.K = 1
+	params.BlockCoinbaseMaturity = 0
+	dag, teardownFunc, err := blockdag.DAGSetup("TestSubnetworkRegistry", blockdag.Config{
+		DAGParams: &params,
+	})
+	if err != nil {
+		t.Fatalf("Failed to setup DAG instance: %v", err)
+	}
+	defer teardownFunc()
+
+	// First we prepare a subnetwork and a block with coinbase outputs to fund our tests
+	gasLimit := uint64(12345)
+	subnetworkID, err := testtools.RegisterSubnetworkForTest(dag, &params, gasLimit)
+	if err != nil {
+		t.Fatalf("could not register network: %s", err)
+	}
+
+	cbTxs := []*wire.MsgTx{}
+	for i := 0; i < 4; i++ {
+		fundsBlock, err := mining.PrepareBlockForTest(dag, &params, dag.TipHashes(), nil, false)
+		if err != nil {
+			t.Fatalf("PrepareBlockForTest: %v", err)
+		}
+		isOrphan, isDelayed, err := dag.ProcessBlock(util.NewBlock(fundsBlock), blockdag.BFNoPoWCheck)
+		if err != nil {
+			t.Fatalf("ProcessBlock: %v", err)
+		}
+		if isDelayed {
+			t.Fatalf("ProcessBlock: the funds block " +
+				"is too far in the future")
+		}
+		if isOrphan {
+			t.Fatalf("ProcessBlock: fundsBlock got unexpectedly orphan")
+		}
+
+		cbTxs = append(cbTxs, fundsBlock.Transactions[util.CoinbaseTransactionIndex])
+	}
+
+	signatureScript, err := txscript.PayToScriptHashSignatureScript(blockdag.OpTrueScript, nil)
+	if err != nil {
+		t.Fatalf("Failed to build signature script: %s", err)
+	}
+
+	scriptPubKey, err := txscript.PayToScriptHashScript(blockdag.OpTrueScript)
+	if err != nil {
+		t.Fatalf("Failed to build public key script: %s", err)
+	}
+
+	tx1In := &wire.TxIn{
+		PreviousOutpoint: *wire.NewOutpoint(cbTxs[0].TxID(), 0),
+		Sequence:         wire.MaxTxInSequenceNum,
+		SignatureScript:  signatureScript,
+	}
+	tx1Out := &wire.TxOut{
+		Value:        cbTxs[0].TxOut[0].Value,
+		ScriptPubKey: scriptPubKey,
+	}
+	tx1 := wire.NewSubnetworkMsgTx(wire.TxVersion, []*wire.TxIn{tx1In}, []*wire.TxOut{tx1Out}, subnetworkID, 10000, []byte{})
+
+	tx2In := &wire.TxIn{
+		PreviousOutpoint: *wire.NewOutpoint(cbTxs[1].TxID(), 0),
+		Sequence:         wire.MaxTxInSequenceNum,
+		SignatureScript:  signatureScript,
+	}
+	tx2Out := &wire.TxOut{
+		Value:        cbTxs[1].TxOut[0].Value,
+		ScriptPubKey: scriptPubKey,
+	}
+	tx2 := wire.NewSubnetworkMsgTx(wire.TxVersion, []*wire.TxIn{tx2In}, []*wire.TxOut{tx2Out}, subnetworkID, 10000, []byte{})
+
+	// Here we check that we can't process a block that has transactions that exceed the gas limit
+	overLimitBlock, err := mining.PrepareBlockForTest(dag, &params, dag.TipHashes(), []*wire.MsgTx{tx1, tx2}, true)
+	if err != nil {
+		t.Fatalf("PrepareBlockForTest: %v", err)
+	}
+	isOrphan, isDelayed, err := dag.ProcessBlock(util.NewBlock(overLimitBlock), blockdag.BFNoPoWCheck)
+	if err == nil {
+		t.Fatalf("ProcessBlock expected to have an error in block that exceeds gas limit")
+	}
+	rErr, ok := err.(blockdag.RuleError)
+	if !ok {
+		t.Fatalf("ProcessBlock expected a RuleError, but got %v", err)
+	} else if rErr.ErrorCode != blockdag.ErrInvalidGas {
+		t.Fatalf("ProcessBlock expected error code %s but got %s", blockdag.ErrInvalidGas, rErr.ErrorCode)
+	}
+	if isDelayed {
+		t.Fatalf("ProcessBlock: overLimitBlock " +
+			"is too far in the future")
+	}
+	if isOrphan {
+		t.Fatalf("ProcessBlock: overLimitBlock got unexpectedly orphan")
+	}
+
+	overflowGasTxIn := &wire.TxIn{
+		PreviousOutpoint: *wire.NewOutpoint(cbTxs[2].TxID(), 0),
+		Sequence:         wire.MaxTxInSequenceNum,
+		SignatureScript:  signatureScript,
+	}
+	overflowGasTxOut := &wire.TxOut{
+		Value:        cbTxs[2].TxOut[0].Value,
+		ScriptPubKey: scriptPubKey,
+	}
+	overflowGasTx := wire.NewSubnetworkMsgTx(wire.TxVersion, []*wire.TxIn{overflowGasTxIn}, []*wire.TxOut{overflowGasTxOut},
+		subnetworkID, math.MaxUint64, []byte{})
+
+	// Here we check that we can't process a block that its transactions' gas overflows uint64
+	overflowGasBlock, err := mining.PrepareBlockForTest(dag, &params, dag.TipHashes(), []*wire.MsgTx{tx1, overflowGasTx}, true)
+	if err != nil {
+		t.Fatalf("PrepareBlockForTest: %v", err)
+	}
+	isOrphan, isDelayed, err = dag.ProcessBlock(util.NewBlock(overflowGasBlock), blockdag.BFNoPoWCheck)
+	if err == nil {
+		t.Fatalf("ProcessBlock expected to have an error")
+	}
+	rErr, ok = err.(blockdag.RuleError)
+	if !ok {
+		t.Fatalf("ProcessBlock expected a RuleError, but got %v", err)
+	} else if rErr.ErrorCode != blockdag.ErrInvalidGas {
+		t.Fatalf("ProcessBlock expected error code %s but got %s", blockdag.ErrInvalidGas, rErr.ErrorCode)
+	}
+	if isOrphan {
+		t.Fatalf("ProcessBlock: overLimitBlock got unexpectedly orphan")
+	}
+
+	nonExistentSubnetwork := &subnetworkid.SubnetworkID{123}
+	nonExistentSubnetworkTxIn := &wire.TxIn{
+		PreviousOutpoint: *wire.NewOutpoint(cbTxs[3].TxID(), 0),
+		Sequence:         wire.MaxTxInSequenceNum,
+		SignatureScript:  signatureScript,
+	}
+	nonExistentSubnetworkTxOut := &wire.TxOut{
+		Value:        cbTxs[3].TxOut[0].Value,
+		ScriptPubKey: scriptPubKey,
+	}
+	nonExistentSubnetworkTx := wire.NewSubnetworkMsgTx(wire.TxVersion, []*wire.TxIn{nonExistentSubnetworkTxIn},
+		[]*wire.TxOut{nonExistentSubnetworkTxOut}, nonExistentSubnetwork, 1, []byte{})
+
+	nonExistentSubnetworkBlock, err := mining.PrepareBlockForTest(dag, &params, dag.TipHashes(), []*wire.MsgTx{nonExistentSubnetworkTx, overflowGasTx}, true)
+	if err != nil {
+		t.Fatalf("PrepareBlockForTest: %v", err)
+	}
+
+	// Here we check that we can't process a block with a transaction from a non-existent subnetwork
+	isOrphan, isDelayed, err = dag.ProcessBlock(util.NewBlock(nonExistentSubnetworkBlock), blockdag.BFNoPoWCheck)
+	expectedErrStr := fmt.Sprintf("Error getting gas limit for subnetworkID '%s': subnetwork '%s' not found",
+		nonExistentSubnetwork, nonExistentSubnetwork)
+	if err.Error() != expectedErrStr {
+		t.Fatalf("ProcessBlock expected error \"%v\" but got \"%v\"", expectedErrStr, err)
+	}
+
+	// Here we check that we can process a block with a transaction that doesn't exceed the gas limit
+	validBlock, err := mining.PrepareBlockForTest(dag, &params, dag.TipHashes(), []*wire.MsgTx{tx1}, true)
+	if err != nil {
+		t.Fatalf("PrepareBlockForTest: %v", err)
+	}
+	isOrphan, isDelayed, err = dag.ProcessBlock(util.NewBlock(validBlock), blockdag.BFNoPoWCheck)
+	if err != nil {
+		t.Fatalf("ProcessBlock: %v", err)
+	}
+	if isDelayed {
+		t.Fatalf("ProcessBlock: overLimitBlock " +
+			"is too far in the future")
+	}
+	if isOrphan {
+		t.Fatalf("ProcessBlock: overLimitBlock got unexpectedly orphan")
+	}
+}

blockdag/ghostdag.go

@@ -0,0 +1,176 @@
+package blockdag
+
+import (
+	"github.com/kaspanet/kaspad/dagconfig"
+	"github.com/pkg/errors"
+	"sort"
+)
+
+// ghostdag runs the GHOSTDAG protocol and updates newNode.blues,
+// newNode.selectedParent and newNode.bluesAnticoneSizes accordingly.
+// The function updates newNode.blues by iterating over the blocks in
+// the anticone of newNode.selectedParent (which is the parent with the
+// highest blue score) and adds any block to newNode.blues if by adding
+// it to newNode.blues these conditions will not be violated:
+//
+// 1) |anticone-of-candidate-block ∩ blue-set-of-newNode| ≤ K
+//
+// 2) For every blue block in blue-set-of-newNode:
+//    |(anticone-of-blue-block ∩ blue-set-newNode) ∪ {candidate-block}| ≤ K.
+//    We validate this condition by maintaining a map bluesAnticoneSizes for
+//    each block which holds all the blue anticone sizes that were affected by
+//    the new added blue blocks.
+//    So to find out what is |anticone-of-blue ∩ blue-set-of-newNode| we just iterate in
+//    the selected parent chain of newNode until we find an existing entry in
+//    bluesAnticoneSizes.
+//
+// For further details see the article https://eprint.iacr.org/2018/104.pdf
+func (dag *BlockDAG) ghostdag(newNode *blockNode) (selectedParentAnticone []*blockNode, err error) {
+	newNode.selectedParent = newNode.parents.bluest()
+	newNode.bluesAnticoneSizes[*newNode.selectedParent.hash] = 0
+	newNode.blues = []*blockNode{newNode.selectedParent}
+	selectedParentAnticone, err = dag.selectedParentAnticone(newNode)
+	if err != nil {
+		return nil, err
+	}
+
+	sort.Slice(selectedParentAnticone, func(i, j int) bool {
+		return selectedParentAnticone[i].less(selectedParentAnticone[j])
+	})
+
+	for _, blueCandidate := range selectedParentAnticone {
+		candidateBluesAnticoneSizes := make(map[*blockNode]dagconfig.KType)
+		var candidateAnticoneSize dagconfig.KType
+		possiblyBlue := true
+
+		// Iterate over all blocks in the blue set of newNode that are not in the past
+		// of blueCandidate, and check for each one of them if blueCandidate potentially
+		// enlarges their blue anticone to be over K, or that they enlarge the blue anticone
+		// of blueCandidate to be over K.
+		for chainBlock := newNode; possiblyBlue; chainBlock = chainBlock.selectedParent {
+			// If blueCandidate is in the future of chainBlock, it means
+			// that all remaining blues are in the past of chainBlock and thus
+			// in the past of blueCandidate. In this case we know for sure that
+			// the anticone of blueCandidate will not exceed K, and we can mark
+			// it as blue.
+			//
+			// newNode is always in the future of blueCandidate, so there's
+			// no point in checking it.
+			if chainBlock != newNode {
+				if isAncestorOfBlueCandidate, err := dag.isAncestorOf(chainBlock, blueCandidate); err != nil {
+					return nil, err
+				} else if isAncestorOfBlueCandidate {
+					break
+				}
+			}
+
+			for _, block := range chainBlock.blues {
+				// Skip blocks that exist in the past of blueCandidate.
+				if isAncestorOfBlueCandidate, err := dag.isAncestorOf(block, blueCandidate); err != nil {
+					return nil, err
+				} else if isAncestorOfBlueCandidate {
+					continue
+				}
+
+				candidateBluesAnticoneSizes[block], err = dag.blueAnticoneSize(block, newNode)
+				if err != nil {
+					return nil, err
+				}
+				candidateAnticoneSize++
+
+				if candidateAnticoneSize > dag.dagParams.K {
+					// k-cluster violation: The candidate's blue anticone exceeded k
+					possiblyBlue = false
+					break
+				}
+
+				if candidateBluesAnticoneSizes[block] == dag.dagParams.K {
+					// k-cluster violation: A block in candidate's blue anticone already
+					// has k blue blocks in its own anticone
+					possiblyBlue = false
+					break
+				}
+
+				// This is a sanity check that validates that a blue
+				// block's blue anticone is not already larger than K.
+				if candidateBluesAnticoneSizes[block] > dag.dagParams.K {
+					return nil, errors.New("found blue anticone size larger than k")
+				}
+			}
+		}
+
+		if possiblyBlue {
+			// No k-cluster violation found, we can now set the candidate block as blue
+			newNode.blues = append(newNode.blues, blueCandidate)
+			newNode.bluesAnticoneSizes[*blueCandidate.hash] = candidateAnticoneSize
+			for blue, blueAnticoneSize := range candidateBluesAnticoneSizes {
+				newNode.bluesAnticoneSizes[*blue.hash] = blueAnticoneSize + 1
+			}
+
+			// The maximum length of node.blues can be K+1 because
+			// it contains the selected parent.
+			if dagconfig.KType(len(newNode.blues)) == dag.dagParams.K+1 {
+				break
+			}
+		}
+	}
+
+	newNode.blueScore = newNode.selectedParent.blueScore + uint64(len(newNode.blues))
+	return selectedParentAnticone, nil
+}
+
+// selectedParentAnticone returns the blocks in the anticone of the selected parent of the given node.
+// The function work as follows.
+// We start by adding all parents of the node (other than the selected parent) to a process queue.
+// For each node in the queue:
+//   we check whether it is in the past of the selected parent.
+//   If not, we add the node to the resulting anticone-set and queue it for processing.
+func (dag *BlockDAG) selectedParentAnticone(node *blockNode) ([]*blockNode, error) {
+	anticoneSet := newSet()
+	var anticoneSlice []*blockNode
+	selectedParentPast := newSet()
+	var queue []*blockNode
+	// Queueing all parents (other than the selected parent itself) for processing.
+	for parent := range node.parents {
+		if parent == node.selectedParent {
+			continue
+		}
+		anticoneSet.add(parent)
+		anticoneSlice = append(anticoneSlice, parent)
+		queue = append(queue, parent)
+	}
+	for len(queue) > 0 {
+		var current *blockNode
+		current, queue = queue[0], queue[1:]
+		// For each parent of a the current node we check whether it is in the past of the selected parent. If not,
+		// we add the it to the resulting anticone-set and queue it for further processing.
+		for parent := range current.parents {
+			if anticoneSet.contains(parent) || selectedParentPast.contains(parent) {
+				continue
+			}
+			isAncestorOfSelectedParent, err := dag.isAncestorOf(parent, node.selectedParent)
+			if err != nil {
+				return nil, err
+			}
+			if isAncestorOfSelectedParent {
+				selectedParentPast.add(parent)
+				continue
+			}
+			anticoneSet.add(parent)
+			anticoneSlice = append(anticoneSlice, parent)
+			queue = append(queue, parent)
+		}
+	}
+	return anticoneSlice, nil
+}
+
+// blueAnticoneSize returns the blue anticone size of 'block' from the worldview of 'context'.
+// Expects 'block' to be in the blue set of 'context'
+func (dag *BlockDAG) blueAnticoneSize(block, context *blockNode) (dagconfig.KType, error) {
+	for current := context; current != nil; current = current.selectedParent {
+		if blueAnticoneSize, ok := current.bluesAnticoneSizes[*block.hash]; ok {
+			return blueAnticoneSize, nil
+		}
+	}
+	return 0, errors.Errorf("block %s is not in blue set of %s", block.hash, context.hash)
+}

blockdag/ghostdag_test.go

@@ -0,0 +1,278 @@
+package blockdag
+
+import (
+	"fmt"
+	"github.com/kaspanet/kaspad/dagconfig"
+	"github.com/kaspanet/kaspad/util"
+	"reflect"
+	"sort"
+	"testing"
+)
+
+type testBlockData struct {
+	parents                []string
+	id                     string // id is a virtual entity that is used only for tests so we can define relations between blocks without knowing their hash
+	expectedScore          uint64
+	expectedSelectedParent string
+	expectedBlues          []string
+}
+
+// TestGHOSTDAG iterates over several dag simulations, and checks
+// that the blue score, blue set and selected parent of each
+// block are calculated as expected.
+func TestGHOSTDAG(t *testing.T) {
+	dagParams := dagconfig.SimnetParams
+
+	tests := []struct {
+		k            dagconfig.KType
+		expectedReds []string
+		dagData      []*testBlockData
+	}{
+		{
+			k:            3,
+			expectedReds: []string{"F", "G", "H", "I", "O", "P"},
+			dagData: []*testBlockData{
+				{
+					parents:                []string{"A"},
+					id:                     "B",
+					expectedScore:          1,
+					expectedSelectedParent: "A",
+					expectedBlues:          []string{"A"},
+				},
+				{
+					parents:                []string{"B"},
+					id:                     "C",
+					expectedScore:          2,
+					expectedSelectedParent: "B",
+					expectedBlues:          []string{"B"},
+				},
+				{
+					parents:                []string{"A"},
+					id:                     "D",
+					expectedScore:          1,
+					expectedSelectedParent: "A",
+					expectedBlues:          []string{"A"},
+				},
+				{
+					parents:                []string{"C", "D"},
+					id:                     "E",
+					expectedScore:          4,
+					expectedSelectedParent: "C",
+					expectedBlues:          []string{"C", "D"},
+				},
+				{
+					parents:                []string{"A"},
+					id:                     "F",
+					expectedScore:          1,
+					expectedSelectedParent: "A",
+					expectedBlues:          []string{"A"},
+				},
+				{
+					parents:                []string{"F"},
+					id:                     "G",
+					expectedScore:          2,
+					expectedSelectedParent: "F",
+					expectedBlues:          []string{"F"},
+				},
+				{
+					parents:                []string{"A"},
+					id:                     "H",
+					expectedScore:          1,
+					expectedSelectedParent: "A",
+					expectedBlues:          []string{"A"},
+				},
+				{
+					parents:                []string{"A"},
+					id:                     "I",
+					expectedScore:          1,
+					expectedSelectedParent: "A",
+					expectedBlues:          []string{"A"},
+				},
+				{
+					parents:                []string{"E", "G"},
+					id:                     "J",
+					expectedScore:          5,
+					expectedSelectedParent: "E",
+					expectedBlues:          []string{"E"},
+				},
+				{
+					parents:                []string{"J"},
+					id:                     "K",
+					expectedScore:          6,
+					expectedSelectedParent: "J",
+					expectedBlues:          []string{"J"},
+				},
+				{
+					parents:                []string{"I", "K"},
+					id:                     "L",
+					expectedScore:          7,
+					expectedSelectedParent: "K",
+					expectedBlues:          []string{"K"},
+				},
+				{
+					parents:                []string{"L"},
+					id:                     "M",
+					expectedScore:          8,
+					expectedSelectedParent: "L",
+					expectedBlues:          []string{"L"},
+				},
+				{
+					parents:                []string{"M"},
+					id:                     "N",
+					expectedScore:          9,
+					expectedSelectedParent: "M",
+					expectedBlues:          []string{"M"},
+				},
+				{
+					parents:                []string{"M"},
+					id:                     "O",
+					expectedScore:          9,
+					expectedSelectedParent: "M",
+					expectedBlues:          []string{"M"},
+				},
+				{
+					parents:                []string{"M"},
+					id:                     "P",
+					expectedScore:          9,
+					expectedSelectedParent: "M",
+					expectedBlues:          []string{"M"},
+				},
+				{
+					parents:                []string{"M"},
+					id:                     "Q",
+					expectedScore:          9,
+					expectedSelectedParent: "M",
+					expectedBlues:          []string{"M"},
+				},
+				{
+					parents:                []string{"M"},
+					id:                     "R",
+					expectedScore:          9,
+					expectedSelectedParent: "M",
+					expectedBlues:          []string{"M"},
+				},
+				{
+					parents:                []string{"R"},
+					id:                     "S",
+					expectedScore:          10,
+					expectedSelectedParent: "R",
+					expectedBlues:          []string{"R"},
+				},
+				{
+					parents:                []string{"N", "O", "P", "Q", "S"},
+					id:                     "T",
+					expectedScore:          13,
+					expectedSelectedParent: "S",
+					expectedBlues:          []string{"S", "N", "Q"},
+				},
+			},
+		},
+	}
+
+	for i, test := range tests {
+		func() {
+			resetExtraNonceForTest()
+			dagParams.K = test.k
+			dag, teardownFunc, err := DAGSetup(fmt.Sprintf("TestGHOSTDAG%d", i), Config{
+				DAGParams: &dagParams,
+			})
+			if err != nil {
+				t.Fatalf("Failed to setup dag instance: %v", err)
+			}
+			defer teardownFunc()
+
+			genesisNode := dag.genesis
+			blockByIDMap := make(map[string]*blockNode)
+			idByBlockMap := make(map[*blockNode]string)
+			blockByIDMap["A"] = genesisNode
+			idByBlockMap[genesisNode] = "A"
+
+			for _, blockData := range test.dagData {
+				parents := blockSet{}
+				for _, parentID := range blockData.parents {
+					parent := blockByIDMap[parentID]
+					parents.add(parent)
+				}
+
+				block, err := PrepareBlockForTest(dag, parents.hashes(), nil)
+				if err != nil {
+					t.Fatalf("TestGHOSTDAG: block %v got unexpected error from PrepareBlockForTest: %v", blockData.id, err)
+				}
+
+				utilBlock := util.NewBlock(block)
+				isOrphan, isDelayed, err := dag.ProcessBlock(utilBlock, BFNoPoWCheck)
+				if err != nil {
+					t.Fatalf("TestGHOSTDAG: dag.ProcessBlock got unexpected error for block %v: %v", blockData.id, err)
+				}
+				if isDelayed {
+					t.Fatalf("TestGHOSTDAG: block %s "+
+						"is too far in the future", blockData.id)
+				}
+				if isOrphan {
+					t.Fatalf("TestGHOSTDAG: block %v was unexpectedly orphan", blockData.id)
+				}
+
+				node := dag.index.LookupNode(utilBlock.Hash())
+
+				blockByIDMap[blockData.id] = node
+				idByBlockMap[node] = blockData.id
+
+				bluesIDs := make([]string, 0, len(node.blues))
+				for _, blue := range node.blues {
+					bluesIDs = append(bluesIDs, idByBlockMap[blue])
+				}
+				selectedParentID := idByBlockMap[node.selectedParent]
+				fullDataStr := fmt.Sprintf("blues: %v, selectedParent: %v, score: %v",
+					bluesIDs, selectedParentID, node.blueScore)
+				if blockData.expectedScore != node.blueScore {
+					t.Errorf("Test %d: Block %v expected to have score %v but got %v (fulldata: %v)",
+						i, blockData.id, blockData.expectedScore, node.blueScore, fullDataStr)
+				}
+				if blockData.expectedSelectedParent != selectedParentID {
+					t.Errorf("Test %d: Block %v expected to have selected parent %v but got %v (fulldata: %v)",
+						i, blockData.id, blockData.expectedSelectedParent, selectedParentID, fullDataStr)
+				}
+				if !reflect.DeepEqual(blockData.expectedBlues, bluesIDs) {
+					t.Errorf("Test %d: Block %v expected to have blues %v but got %v (fulldata: %v)",
+						i, blockData.id, blockData.expectedBlues, bluesIDs, fullDataStr)
+				}
+			}
+
+			reds := make(map[string]bool)
+
+			for id := range blockByIDMap {
+				reds[id] = true
+			}
+
+			for tip := &dag.virtual.blockNode; tip.selectedParent != nil; tip = tip.selectedParent {
+				tipID := idByBlockMap[tip]
+				delete(reds, tipID)
+				for _, blue := range tip.blues {
+					blueID := idByBlockMap[blue]
+					delete(reds, blueID)
+				}
+			}
+			if !checkReds(test.expectedReds, reds) {
+				redsIDs := make([]string, 0, len(reds))
+				for id := range reds {
+					redsIDs = append(redsIDs, id)
+				}
+				sort.Strings(redsIDs)
+				sort.Strings(test.expectedReds)
+				t.Errorf("Test %d: Expected reds %v but got %v", i, test.expectedReds, redsIDs)
+			}
+		}()
+	}
+}
+
+func checkReds(expectedReds []string, reds map[string]bool) bool {
+	if len(expectedReds) != len(reds) {
+		return false
+	}
+	for _, redID := range expectedReds {
+		if !reds[redID] {
+			return false
+		}
+	}
+	return true
+}

blockdag/indexers/README.md

@@ -0,0 +1,24 @@
+indexers
+========
+
+[![ISC License](http://img.shields.io/badge/license-ISC-blue.svg)](http://copyfree.org)
+[![GoDoc](https://godoc.org/github.com/kaspanet/kaspad/blockdag/indexers?status.png)](http://godoc.org/github.com/kaspanet/kaspad/blockdag/indexers)
+
+Package indexers implements optional block chain indexes.
+
+These indexes are typically used to enhance the amount of information available
+via an RPC interface.
+
+## Supported Indexers
+
+- Transaction-by-hash (txindex) Index
+  - Creates a mapping from the hash of each transaction to the block that
+    contains it along with its offset and length within the serialized block
+- Transaction-by-address (addrindex) Index
+  - Creates a mapping from every address to all transactions which either credit
+    or debit the address
+  - Requires the transaction-by-hash index
+- AcceptanceData-by-block Index
+  - Creates a mapping from the hash of each block to the list of transaction this block
+    accepts from it's .Blues
+

blockdag/indexers/acceptanceindex.go

@@ -0,0 +1,234 @@
+package indexers
+
+import (
+	"bytes"
+	"encoding/gob"
+	"github.com/kaspanet/kaspad/blockdag"
+	"github.com/kaspanet/kaspad/database"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/wire"
+	"github.com/pkg/errors"
+)
+
+const (
+	// acceptanceIndexName is the human-readable name for the index.
+	acceptanceIndexName = "acceptance index"
+)
+
+var (
+	// acceptanceIndexKey is the key of the acceptance index and the db bucket used
+	// to house it.
+	acceptanceIndexKey = []byte("acceptanceidx")
+)
+
+// AcceptanceIndex implements a txAcceptanceData by block hash index. That is to say,
+// it stores a mapping between a block's hash and the set of transactions that the
+// block accepts among its blue blocks.
+type AcceptanceIndex struct {
+	db  database.DB
+	dag *blockdag.BlockDAG
+}
+
+// Ensure the AcceptanceIndex type implements the Indexer interface.
+var _ Indexer = (*AcceptanceIndex)(nil)
+
+// NewAcceptanceIndex returns a new instance of an indexer that is used to create a
+// mapping between block hashes and their txAcceptanceData.
+//
+// It implements the Indexer interface which plugs into the IndexManager that in
+// turn is used by the blockdag package. This allows the index to be
+// seamlessly maintained along with the DAG.
+func NewAcceptanceIndex() *AcceptanceIndex {
+	return &AcceptanceIndex{}
+}
+
+// DropAcceptanceIndex drops the acceptance index from the provided database if it
+// exists.
+func DropAcceptanceIndex(db database.DB, interrupt <-chan struct{}) error {
+	return dropIndex(db, acceptanceIndexKey, acceptanceIndexName, interrupt)
+}
+
+// Key returns the database key to use for the index as a byte slice.
+//
+// This is part of the Indexer interface.
+func (idx *AcceptanceIndex) Key() []byte {
+	return acceptanceIndexKey
+}
+
+// Name returns the human-readable name of the index.
+//
+// This is part of the Indexer interface.
+func (idx *AcceptanceIndex) Name() string {
+	return acceptanceIndexName
+}
+
+// Create is invoked when the indexer manager determines the index needs
+// to be created for the first time. It creates the bucket for the
+// acceptance index.
+//
+// This is part of the Indexer interface.
+func (idx *AcceptanceIndex) Create(dbTx database.Tx) error {
+	_, err := dbTx.Metadata().CreateBucket(acceptanceIndexKey)
+	return err
+}
+
+// Init initializes the hash-based acceptance index.
+//
+// This is part of the Indexer interface.
+func (idx *AcceptanceIndex) Init(db database.DB, dag *blockdag.BlockDAG) error {
+	idx.db = db
+	idx.dag = dag
+	return nil
+}
+
+// ConnectBlock is invoked by the index manager when a new block has been
+// connected to the DAG.
+//
+// This is part of the Indexer interface.
+func (idx *AcceptanceIndex) ConnectBlock(dbTx database.Tx, _ *util.Block, blockID uint64, _ *blockdag.BlockDAG,
+	txsAcceptanceData blockdag.MultiBlockTxsAcceptanceData, _ blockdag.MultiBlockTxsAcceptanceData) error {
+	return dbPutTxsAcceptanceData(dbTx, blockID, txsAcceptanceData)
+}
+
+// TxsAcceptanceData returns the acceptance data of all the transactions that
+// were accepted by the block with hash blockHash.
+func (idx *AcceptanceIndex) TxsAcceptanceData(blockHash *daghash.Hash) (blockdag.MultiBlockTxsAcceptanceData, error) {
+	var txsAcceptanceData blockdag.MultiBlockTxsAcceptanceData
+	err := idx.db.View(func(dbTx database.Tx) error {
+		var err error
+		txsAcceptanceData, err = dbFetchTxsAcceptanceDataByHash(dbTx, blockHash)
+		return err
+	})
+	if err != nil {
+		return nil, err
+	}
+	return txsAcceptanceData, nil
+}
+
+// Recover is invoked when the indexer wasn't turned on for several blocks
+// and the indexer needs to close the gaps.
+//
+// This is part of the Indexer interface.
+func (idx *AcceptanceIndex) Recover(dbTx database.Tx, currentBlockID, lastKnownBlockID uint64) error {
+	for blockID := currentBlockID + 1; blockID <= lastKnownBlockID; blockID++ {
+		hash, err := blockdag.DBFetchBlockHashByID(dbTx, currentBlockID)
+		if err != nil {
+			return err
+		}
+		txAcceptanceData, err := idx.dag.TxsAcceptedByBlockHash(hash)
+		if err != nil {
+			return err
+		}
+		err = idx.ConnectBlock(dbTx, nil, blockID, nil, txAcceptanceData, nil)
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func dbPutTxsAcceptanceData(dbTx database.Tx, blockID uint64,
+	txsAcceptanceData blockdag.MultiBlockTxsAcceptanceData) error {
+	serializedTxsAcceptanceData, err := serializeMultiBlockTxsAcceptanceData(txsAcceptanceData)
+	if err != nil {
+		return err
+	}
+
+	bucket := dbTx.Metadata().Bucket(acceptanceIndexKey)
+	return bucket.Put(blockdag.SerializeBlockID(blockID), serializedTxsAcceptanceData)
+}
+
+func dbFetchTxsAcceptanceDataByHash(dbTx database.Tx,
+	hash *daghash.Hash) (blockdag.MultiBlockTxsAcceptanceData, error) {
+
+	blockID, err := blockdag.DBFetchBlockIDByHash(dbTx, hash)
+	if err != nil {
+		return nil, err
+	}
+
+	return dbFetchTxsAcceptanceDataByID(dbTx, blockID)
+}
+
+func dbFetchTxsAcceptanceDataByID(dbTx database.Tx,
+	blockID uint64) (blockdag.MultiBlockTxsAcceptanceData, error) {
+	serializedBlockID := blockdag.SerializeBlockID(blockID)
+	bucket := dbTx.Metadata().Bucket(acceptanceIndexKey)
+	serializedTxsAcceptanceData := bucket.Get(serializedBlockID)
+	if serializedTxsAcceptanceData == nil {
+		return nil, errors.Errorf("no entry in the accpetance index for block id %d", blockID)
+	}
+
+	return deserializeMultiBlockTxsAcceptanceData(serializedTxsAcceptanceData)
+}
+
+type serializableTxAcceptanceData struct {
+	MsgTx      wire.MsgTx
+	IsAccepted bool
+}
+
+type serializableBlockTxsAcceptanceData struct {
+	BlockHash        daghash.Hash
+	TxAcceptanceData []serializableTxAcceptanceData
+}
+
+type serializableMultiBlockTxsAcceptanceData []serializableBlockTxsAcceptanceData
+
+func serializeMultiBlockTxsAcceptanceData(
+	multiBlockTxsAcceptanceData blockdag.MultiBlockTxsAcceptanceData) ([]byte, error) {
+	// Convert MultiBlockTxsAcceptanceData to a serializable format
+	serializableData := make(serializableMultiBlockTxsAcceptanceData, len(multiBlockTxsAcceptanceData))
+	for i, blockTxsAcceptanceData := range multiBlockTxsAcceptanceData {
+		serializableBlockData := serializableBlockTxsAcceptanceData{
+			BlockHash:        blockTxsAcceptanceData.BlockHash,
+			TxAcceptanceData: make([]serializableTxAcceptanceData, len(blockTxsAcceptanceData.TxAcceptanceData)),
+		}
+		for i, txAcceptanceData := range blockTxsAcceptanceData.TxAcceptanceData {
+			serializableBlockData.TxAcceptanceData[i] = serializableTxAcceptanceData{
+				MsgTx:      *txAcceptanceData.Tx.MsgTx(),
+				IsAccepted: txAcceptanceData.IsAccepted,
+			}
+		}
+		serializableData[i] = serializableBlockData
+	}
+
+	// Serialize
+	var buffer bytes.Buffer
+	encoder := gob.NewEncoder(&buffer)
+	err := encoder.Encode(serializableData)
+	if err != nil {
+		return nil, err
+	}
+	return buffer.Bytes(), nil
+}
+
+func deserializeMultiBlockTxsAcceptanceData(
+	serializedTxsAcceptanceData []byte) (blockdag.MultiBlockTxsAcceptanceData, error) {
+	// Deserialize
+	buffer := bytes.NewBuffer(serializedTxsAcceptanceData)
+	decoder := gob.NewDecoder(buffer)
+	var serializedData serializableMultiBlockTxsAcceptanceData
+	err := decoder.Decode(&serializedData)
+	if err != nil {
+		return nil, err
+	}
+
+	// Convert serializable format to MultiBlockTxsAcceptanceData
+	multiBlockTxsAcceptanceData := make(blockdag.MultiBlockTxsAcceptanceData, len(serializedData))
+	for i, serializableBlockData := range serializedData {
+		blockTxsAcceptanceData := blockdag.BlockTxsAcceptanceData{
+			BlockHash:        serializableBlockData.BlockHash,
+			TxAcceptanceData: make([]blockdag.TxAcceptanceData, len(serializableBlockData.TxAcceptanceData)),
+		}
+		for i, txData := range serializableBlockData.TxAcceptanceData {
+			msgTx := txData.MsgTx
+			blockTxsAcceptanceData.TxAcceptanceData[i] = blockdag.TxAcceptanceData{
+				Tx:         util.NewTx(&msgTx),
+				IsAccepted: txData.IsAccepted,
+			}
+		}
+		multiBlockTxsAcceptanceData[i] = blockTxsAcceptanceData
+	}
+
+	return multiBlockTxsAcceptanceData, nil
+}

blockdag/indexers/acceptanceindex_test.go

@@ -0,0 +1,340 @@
+package indexers
+
+import (
+	"github.com/kaspanet/kaspad/blockdag"
+	"github.com/kaspanet/kaspad/dagconfig"
+	"github.com/kaspanet/kaspad/database"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/wire"
+	"github.com/pkg/errors"
+	"io"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"reflect"
+	"syscall"
+	"testing"
+)
+
+func TestAcceptanceIndexSerializationAndDeserialization(t *testing.T) {
+	// Create test data
+	hash, _ := daghash.NewHashFromStr("1111111111111111111111111111111111111111111111111111111111111111")
+	txIn1 := &wire.TxIn{SignatureScript: []byte{1}, PreviousOutpoint: wire.Outpoint{Index: 1}, Sequence: 0}
+	txIn2 := &wire.TxIn{SignatureScript: []byte{2}, PreviousOutpoint: wire.Outpoint{Index: 2}, Sequence: 0}
+	txOut1 := &wire.TxOut{ScriptPubKey: []byte{1}, Value: 10}
+	txOut2 := &wire.TxOut{ScriptPubKey: []byte{2}, Value: 20}
+	blockTxsAcceptanceData := blockdag.BlockTxsAcceptanceData{
+		BlockHash: *hash,
+		TxAcceptanceData: []blockdag.TxAcceptanceData{
+			{
+				Tx:         util.NewTx(wire.NewNativeMsgTx(wire.TxVersion, []*wire.TxIn{txIn1}, []*wire.TxOut{txOut1})),
+				IsAccepted: true,
+			},
+			{
+				Tx:         util.NewTx(wire.NewNativeMsgTx(wire.TxVersion, []*wire.TxIn{txIn2}, []*wire.TxOut{txOut2})),
+				IsAccepted: false,
+			},
+		},
+	}
+	multiBlockTxsAcceptanceData := blockdag.MultiBlockTxsAcceptanceData{blockTxsAcceptanceData}
+
+	// Serialize
+	serializedTxsAcceptanceData, err := serializeMultiBlockTxsAcceptanceData(multiBlockTxsAcceptanceData)
+	if err != nil {
+		t.Fatalf("TestAcceptanceIndexSerializationAndDeserialization: serialization failed: %s", err)
+	}
+
+	// Deserialize
+	deserializedTxsAcceptanceData, err := deserializeMultiBlockTxsAcceptanceData(serializedTxsAcceptanceData)
+	if err != nil {
+		t.Fatalf("TestAcceptanceIndexSerializationAndDeserialization: deserialization failed: %s", err)
+	}
+
+	// Check that they're the same
+	if !reflect.DeepEqual(multiBlockTxsAcceptanceData, deserializedTxsAcceptanceData) {
+		t.Fatalf("TestAcceptanceIndexSerializationAndDeserialization: original data and deseralize data aren't equal")
+	}
+}
+
+// TestAcceptanceIndexRecover tests the recoverability of the
+// acceptance index.
+// It does it by following these steps:
+// * It creates a DAG with enabled acceptance index (let's call it dag1) and
+//   make it process some blocks.
+// * It creates a copy of dag1 (let's call it dag2), and disables the acceptance
+//   index in it.
+// * It processes two more blocks in both dag1 and dag2.
+// * A copy of dag2 is created (let's call it dag3) with enabled
+//   acceptance index
+// * It checks that the two missing blocks are added to dag3 acceptance index by
+//   comparing dag1's last block acceptance data and dag3's last block acceptance
+//   data.
+func TestAcceptanceIndexRecover(t *testing.T) {
+	params := &dagconfig.SimnetParams
+	params.BlockCoinbaseMaturity = 0
+
+	testFiles := []string{
+		"blk_0_to_4.dat",
+		"blk_3B.dat",
+	}
+
+	var blocks []*util.Block
+	for _, file := range testFiles {
+		blockTmp, err := blockdag.LoadBlocks(filepath.Join("../testdata/", file))
+		if err != nil {
+			t.Fatalf("Error loading file: %v\n", err)
+		}
+		blocks = append(blocks, blockTmp...)
+	}
+
+	db1AcceptanceIndex := NewAcceptanceIndex()
+	db1IndexManager := NewManager([]Indexer{db1AcceptanceIndex})
+	db1Path, err := ioutil.TempDir("", "TestAcceptanceIndexRecover1")
+	if err != nil {
+		t.Fatalf("Error creating temporary directory: %s", err)
+	}
+	defer os.RemoveAll(db1Path)
+
+	db1, err := database.Create("ffldb", db1Path, params.Net)
+	if err != nil {
+		t.Fatalf("error creating db: %s", err)
+	}
+
+	db1Config := blockdag.Config{
+		IndexManager: db1IndexManager,
+		DAGParams:    params,
+		DB:           db1,
+	}
+
+	db1DAG, teardown, err := blockdag.DAGSetup("", db1Config)
+	if err != nil {
+		t.Fatalf("TestAcceptanceIndexRecover: Failed to setup DAG instance: %v", err)
+	}
+	if teardown != nil {
+		defer teardown()
+	}
+
+	for i := 1; i < len(blocks)-2; i++ {
+		isOrphan, isDelayed, err := db1DAG.ProcessBlock(blocks[i], blockdag.BFNone)
+		if err != nil {
+			t.Fatalf("ProcessBlock fail on block %v: %v\n", i, err)
+		}
+		if isDelayed {
+			t.Fatalf("ProcessBlock: block %d "+
+				"is too far in the future", i)
+		}
+		if isOrphan {
+			t.Fatalf("ProcessBlock incorrectly returned block %v "+
+				"is an orphan\n", i)
+		}
+	}
+
+	err = db1.FlushCache()
+	if err != nil {
+		t.Fatalf("Error flushing database to disk: %s", err)
+	}
+
+	db2Path, err := ioutil.TempDir("", "TestAcceptanceIndexRecover2")
+	if err != nil {
+		t.Fatalf("Error creating temporary directory: %s", err)
+	}
+	defer os.RemoveAll(db2Path)
+
+	err = copyDirectory(db1Path, db2Path)
+	if err != nil {
+		t.Fatalf("copyDirectory: %s", err)
+	}
+
+	for i := len(blocks) - 2; i < len(blocks); i++ {
+		isOrphan, isDelayed, err := db1DAG.ProcessBlock(blocks[i], blockdag.BFNone)
+		if err != nil {
+			t.Fatalf("ProcessBlock fail on block %v: %v\n", i, err)
+		}
+		if isDelayed {
+			t.Fatalf("ProcessBlock: block %d "+
+				"is too far in the future", i)
+		}
+		if isOrphan {
+			t.Fatalf("ProcessBlock incorrectly returned block %v "+
+				"is an orphan\n", i)
+		}
+	}
+
+	db1LastBlockAcceptanceData, err := db1AcceptanceIndex.TxsAcceptanceData(blocks[len(blocks)-1].Hash())
+	if err != nil {
+		t.Fatalf("Error fetching acceptance data: %s", err)
+	}
+
+	db2, err := database.Open("ffldb", db2Path, params.Net)
+	if err != nil {
+		t.Fatalf("Error opening database: %s", err)
+	}
+
+	db2Config := blockdag.Config{
+		DAGParams: params,
+		DB:        db2,
+	}
+
+	db2DAG, teardown, err := blockdag.DAGSetup("", db2Config)
+	if err != nil {
+		t.Fatalf("TestAcceptanceIndexRecover: Failed to setup DAG instance: %v", err)
+	}
+	if teardown != nil {
+		defer teardown()
+	}
+
+	for i := len(blocks) - 2; i < len(blocks); i++ {
+		isOrphan, isDelayed, err := db2DAG.ProcessBlock(blocks[i], blockdag.BFNone)
+		if err != nil {
+			t.Fatalf("ProcessBlock fail on block %v: %v\n", i, err)
+		}
+		if isDelayed {
+			t.Fatalf("ProcessBlock: block %d "+
+				"is too far in the future", i)
+		}
+		if isOrphan {
+			t.Fatalf("ProcessBlock incorrectly returned block %v "+
+				"is an orphan\n", i)
+		}
+	}
+
+	err = db2.FlushCache()
+	if err != nil {
+		t.Fatalf("Error flushing database to disk: %s", err)
+	}
+	db3Path, err := ioutil.TempDir("", "TestAcceptanceIndexRecover3")
+	if err != nil {
+		t.Fatalf("Error creating temporary directory: %s", err)
+	}
+	defer os.RemoveAll(db3Path)
+	err = copyDirectory(db2Path, db3Path)
+	if err != nil {
+		t.Fatalf("copyDirectory: %s", err)
+	}
+
+	db3, err := database.Open("ffldb", db3Path, params.Net)
+	if err != nil {
+		t.Fatalf("Error opening database: %s", err)
+	}
+
+	db3AcceptanceIndex := NewAcceptanceIndex()
+	db3IndexManager := NewManager([]Indexer{db3AcceptanceIndex})
+	db3Config := blockdag.Config{
+		IndexManager: db3IndexManager,
+		DAGParams:    params,
+		DB:           db3,
+	}
+
+	_, teardown, err = blockdag.DAGSetup("", db3Config)
+	if err != nil {
+		t.Fatalf("TestAcceptanceIndexRecover: Failed to setup DAG instance: %v", err)
+	}
+	if teardown != nil {
+		defer teardown()
+	}
+
+	db3LastBlockAcceptanceData, err := db3AcceptanceIndex.TxsAcceptanceData(blocks[len(blocks)-1].Hash())
+	if err != nil {
+		t.Fatalf("Error fetching acceptance data: %s", err)
+	}
+	if !reflect.DeepEqual(db1LastBlockAcceptanceData, db3LastBlockAcceptanceData) {
+		t.Fatalf("recovery failed")
+	}
+}
+
+// This function is copied and modified from this stackoverflow answer: https://stackoverflow.com/a/56314145/2413761
+func copyDirectory(scrDir, dest string) error {
+	entries, err := ioutil.ReadDir(scrDir)
+	if err != nil {
+		return err
+	}
+	for _, entry := range entries {
+		sourcePath := filepath.Join(scrDir, entry.Name())
+		destPath := filepath.Join(dest, entry.Name())
+
+		fileInfo, err := os.Stat(sourcePath)
+		if err != nil {
+			return err
+		}
+
+		stat, ok := fileInfo.Sys().(*syscall.Stat_t)
+		if !ok {
+			return errors.Errorf("failed to get raw syscall.Stat_t data for '%s'", sourcePath)
+		}
+
+		switch fileInfo.Mode() & os.ModeType {
+		case os.ModeDir:
+			if err := createIfNotExists(destPath, 0755); err != nil {
+				return err
+			}
+			if err := copyDirectory(sourcePath, destPath); err != nil {
+				return err
+			}
+		case os.ModeSymlink:
+			if err := copySymLink(sourcePath, destPath); err != nil {
+				return err
+			}
+		default:
+			if err := copyFile(sourcePath, destPath); err != nil {
+				return err
+			}
+		}
+
+		if err := os.Lchown(destPath, int(stat.Uid), int(stat.Gid)); err != nil {
+			return err
+		}
+
+		isSymlink := entry.Mode()&os.ModeSymlink != 0
+		if !isSymlink {
+			if err := os.Chmod(destPath, entry.Mode()); err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
+// This function is copied and modified from this stackoverflow answer: https://stackoverflow.com/a/56314145/2413761
+func copyFile(srcFile, dstFile string) error {
+	out, err := os.Create(dstFile)
+	defer out.Close()
+	if err != nil {
+		return err
+	}
+
+	in, err := os.Open(srcFile)
+	defer in.Close()
+	if err != nil {
+		return err
+	}
+
+	_, err = io.Copy(out, in)
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+// This function is copied and modified from this stackoverflow answer: https://stackoverflow.com/a/56314145/2413761
+func createIfNotExists(dir string, perm os.FileMode) error {
+	if blockdag.FileExists(dir) {
+		return nil
+	}
+
+	if err := os.MkdirAll(dir, perm); err != nil {
+		return errors.Errorf("failed to create directory: '%s', error: '%s'", dir, err.Error())
+	}
+
+	return nil
+}
+
+// This function is copied and modified from this stackoverflow answer: https://stackoverflow.com/a/56314145/2413761
+func copySymLink(source, dest string) error {
+	link, err := os.Readlink(source)
+	if err != nil {
+		return err
+	}
+	return os.Symlink(link, dest)
+}

blockdag/indexers/addrindex.go

@@ -0,0 +1,902 @@
+// Copyright (c) 2016 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package indexers
+
+import (
+	"fmt"
+	"github.com/pkg/errors"
+	"sync"
+
+	"github.com/kaspanet/kaspad/blockdag"
+	"github.com/kaspanet/kaspad/dagconfig"
+	"github.com/kaspanet/kaspad/database"
+	"github.com/kaspanet/kaspad/txscript"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/wire"
+)
+
+const (
+	// addrIndexName is the human-readable name for the index.
+	addrIndexName = "address index"
+
+	// level0MaxEntries is the maximum number of transactions that are
+	// stored in level 0 of an address index entry. Subsequent levels store
+	// 2^n * level0MaxEntries entries, or in words, double the maximum of
+	// the previous level.
+	level0MaxEntries = 8
+
+	// addrKeySize is the number of bytes an address key consumes in the
+	// index. It consists of 1 byte address type + 20 bytes hash160.
+	addrKeySize = 1 + 20
+
+	// levelKeySize is the number of bytes a level key in the address index
+	// consumes. It consists of the address key + 1 byte for the level.
+	levelKeySize = addrKeySize + 1
+
+	// levelOffset is the offset in the level key which identifes the level.
+	levelOffset = levelKeySize - 1
+
+	// addrKeyTypePubKeyHash is the address type in an address key which
+	// represents both a pay-to-pubkey-hash and a pay-to-pubkey address.
+	// This is done because both are identical for the purposes of the
+	// address index.
+	addrKeyTypePubKeyHash = 0
+
+	// addrKeyTypeScriptHash is the address type in an address key which
+	// represents a pay-to-script-hash address. This is necessary because
+	// the hash of a pubkey address might be the same as that of a script
+	// hash.
+	addrKeyTypeScriptHash = 1
+
+	// Size of a transaction entry. It consists of 8 bytes block id + 4
+	// bytes offset + 4 bytes length.
+	txEntrySize = 8 + 4 + 4
+)
+
+var (
+	// addrIndexKey is the key of the address index and the db bucket used
+	// to house it.
+	addrIndexKey = []byte("txbyaddridx")
+
+	// errUnsupportedAddressType is an error that is used to signal an
+	// unsupported address type has been used.
+	errUnsupportedAddressType = errors.New("address type is not supported " +
+		"by the address index")
+)
+
+// -----------------------------------------------------------------------------
+// The address index maps addresses referenced in the blockDAG to a list of
+// all the transactions involving that address. Transactions are stored
+// according to their order of appearance in the blockDAG. That is to say
+// first by block height and then by offset inside the block. It is also
+// important to note that this implementation requires the transaction index
+// since it is needed in order to catch up old blocks due to the fact the spent
+// outputs will already be pruned from the utxo set.
+//
+// The approach used to store the index is similar to a log-structured merge
+// tree (LSM tree) and is thus similar to how leveldb works internally.
+//
+// Every address consists of one or more entries identified by a level starting
+// from 0 where each level holds a maximum number of entries such that each
+// subsequent level holds double the maximum of the previous one. In equation
+// form, the number of entries each level holds is 2^n * firstLevelMaxSize.
+//
+// New transactions are appended to level 0 until it becomes full at which point
+// the entire level 0 entry is appended to the level 1 entry and level 0 is
+// cleared. This process continues until level 1 becomes full at which point it
+// will be appended to level 2 and cleared and so on.
+//
+// The result of this is the lower levels contain newer transactions and the
+// transactions within each level are ordered from oldest to newest.
+//
+// The intent of this approach is to provide a balance between space efficiency
+// and indexing cost. Storing one entry per transaction would have the lowest
+// indexing cost, but would waste a lot of space because the same address hash
+// would be duplicated for every transaction key. On the other hand, storing a
+// single entry with all transactions would be the most space efficient, but
+// would cause indexing cost to grow quadratically with the number of
+// transactions involving the same address. The approach used here provides
+// logarithmic insertion and retrieval.
+//
+// The serialized key format is:
+//
+//   <addr type><addr hash><level>
+//
+//   Field           Type      Size
+//   addr type       uint8     1 byte
+//   addr hash       hash160   20 bytes
+//   level           uint8     1 byte
+//   -----
+//   Total: 22 bytes
+//
+// The serialized value format is:
+//
+//   [<block id><start offset><tx length>,...]
+//
+//   Field           Type      Size
+//   block id        uint64    8 bytes
+//   start offset    uint32    4 bytes
+//   tx length       uint32    4 bytes
+//   -----
+//   Total: 16 bytes per indexed tx
+// -----------------------------------------------------------------------------
+
+// fetchBlockHashFunc defines a callback function to use in order to convert a
+// serialized block ID to an associated block hash.
+type fetchBlockHashFunc func(serializedID []byte) (*daghash.Hash, error)
+
+// serializeAddrIndexEntry serializes the provided block id and transaction
+// location according to the format described in detail above.
+func serializeAddrIndexEntry(blockID uint64, txLoc wire.TxLoc) []byte {
+	// Serialize the entry.
+	serialized := make([]byte, 16)
+	byteOrder.PutUint64(serialized, blockID)
+	byteOrder.PutUint32(serialized[8:], uint32(txLoc.TxStart))
+	byteOrder.PutUint32(serialized[12:], uint32(txLoc.TxLen))
+	return serialized
+}
+
+// deserializeAddrIndexEntry decodes the passed serialized byte slice into the
+// provided region struct according to the format described in detail above and
+// uses the passed block hash fetching function in order to conver the block ID
+// to the associated block hash.
+func deserializeAddrIndexEntry(serialized []byte, region *database.BlockRegion, fetchBlockHash fetchBlockHashFunc) error {
+	// Ensure there are enough bytes to decode.
+	if len(serialized) < txEntrySize {
+		return errDeserialize("unexpected end of data")
+	}
+
+	hash, err := fetchBlockHash(serialized[0:8])
+	if err != nil {
+		return err
+	}
+	region.Hash = hash
+	region.Offset = byteOrder.Uint32(serialized[8:12])
+	region.Len = byteOrder.Uint32(serialized[12:16])
+	return nil
+}
+
+// keyForLevel returns the key for a specific address and level in the address
+// index entry.
+func keyForLevel(addrKey [addrKeySize]byte, level uint8) [levelKeySize]byte {
+	var key [levelKeySize]byte
+	copy(key[:], addrKey[:])
+	key[levelOffset] = level
+	return key
+}
+
+// dbPutAddrIndexEntry updates the address index to include the provided entry
+// according to the level-based scheme described in detail above.
+func dbPutAddrIndexEntry(bucket internalBucket, addrKey [addrKeySize]byte, blockID uint64, txLoc wire.TxLoc) error {
+	// Start with level 0 and its initial max number of entries.
+	curLevel := uint8(0)
+	maxLevelBytes := level0MaxEntries * txEntrySize
+
+	// Simply append the new entry to level 0 and return now when it will
+	// fit. This is the most common path.
+	newData := serializeAddrIndexEntry(blockID, txLoc)
+	level0Key := keyForLevel(addrKey, 0)
+	level0Data := bucket.Get(level0Key[:])
+	if len(level0Data)+len(newData) <= maxLevelBytes {
+		mergedData := newData
+		if len(level0Data) > 0 {
+			mergedData = make([]byte, len(level0Data)+len(newData))
+			copy(mergedData, level0Data)
+			copy(mergedData[len(level0Data):], newData)
+		}
+		return bucket.Put(level0Key[:], mergedData)
+	}
+
+	// At this point, level 0 is full, so merge each level into higher
+	// levels as many times as needed to free up level 0.
+	prevLevelData := level0Data
+	for {
+		// Each new level holds twice as much as the previous one.
+		curLevel++
+		maxLevelBytes *= 2
+
+		// Move to the next level as long as the current level is full.
+		curLevelKey := keyForLevel(addrKey, curLevel)
+		curLevelData := bucket.Get(curLevelKey[:])
+		if len(curLevelData) == maxLevelBytes {
+			prevLevelData = curLevelData
+			continue
+		}
+
+		// The current level has room for the data in the previous one,
+		// so merge the data from previous level into it.
+		mergedData := prevLevelData
+		if len(curLevelData) > 0 {
+			mergedData = make([]byte, len(curLevelData)+
+				len(prevLevelData))
+			copy(mergedData, curLevelData)
+			copy(mergedData[len(curLevelData):], prevLevelData)
+		}
+		err := bucket.Put(curLevelKey[:], mergedData)
+		if err != nil {
+			return err
+		}
+
+		// Move all of the levels before the previous one up a level.
+		for mergeLevel := curLevel - 1; mergeLevel > 0; mergeLevel-- {
+			mergeLevelKey := keyForLevel(addrKey, mergeLevel)
+			prevLevelKey := keyForLevel(addrKey, mergeLevel-1)
+			prevData := bucket.Get(prevLevelKey[:])
+			err := bucket.Put(mergeLevelKey[:], prevData)
+			if err != nil {
+				return err
+			}
+		}
+		break
+	}
+
+	// Finally, insert the new entry into level 0 now that it is empty.
+	return bucket.Put(level0Key[:], newData)
+}
+
+// dbFetchAddrIndexEntries returns block regions for transactions referenced by
+// the given address key and the number of entries skipped since it could have
+// been less in the case where there are less total entries than the requested
+// number of entries to skip.
+func dbFetchAddrIndexEntries(bucket internalBucket, addrKey [addrKeySize]byte, numToSkip, numRequested uint32, reverse bool, fetchBlockHash fetchBlockHashFunc) ([]database.BlockRegion, uint32, error) {
+	// When the reverse flag is not set, all levels need to be fetched
+	// because numToSkip and numRequested are counted from the oldest
+	// transactions (highest level) and thus the total count is needed.
+	// However, when the reverse flag is set, only enough records to satisfy
+	// the requested amount are needed.
+	var level uint8
+	var serialized []byte
+	for !reverse || len(serialized) < int(numToSkip+numRequested)*txEntrySize {
+		curLevelKey := keyForLevel(addrKey, level)
+		levelData := bucket.Get(curLevelKey[:])
+		if levelData == nil {
+			// Stop when there are no more levels.
+			break
+		}
+
+		// Higher levels contain older transactions, so prepend them.
+		prepended := make([]byte, len(serialized)+len(levelData))
+		copy(prepended, levelData)
+		copy(prepended[len(levelData):], serialized)
+		serialized = prepended
+		level++
+	}
+
+	// When the requested number of entries to skip is larger than the
+	// number available, skip them all and return now with the actual number
+	// skipped.
+	numEntries := uint32(len(serialized) / txEntrySize)
+	if numToSkip >= numEntries {
+		return nil, numEntries, nil
+	}
+
+	// Nothing more to do when there are no requested entries.
+	if numRequested == 0 {
+		return nil, numToSkip, nil
+	}
+
+	// Limit the number to load based on the number of available entries,
+	// the number to skip, and the number requested.
+	numToLoad := numEntries - numToSkip
+	if numToLoad > numRequested {
+		numToLoad = numRequested
+	}
+
+	// Start the offset after all skipped entries and load the calculated
+	// number.
+	results := make([]database.BlockRegion, numToLoad)
+	for i := uint32(0); i < numToLoad; i++ {
+		// Calculate the read offset according to the reverse flag.
+		var offset uint32
+		if reverse {
+			offset = (numEntries - numToSkip - i - 1) * txEntrySize
+		} else {
+			offset = (numToSkip + i) * txEntrySize
+		}
+
+		// Deserialize and populate the result.
+		err := deserializeAddrIndexEntry(serialized[offset:],
+			&results[i], fetchBlockHash)
+		if err != nil {
+			// Ensure any deserialization errors are returned as
+			// database corruption errors.
+			if isDeserializeErr(err) {
+				err = database.Error{
+					ErrorCode: database.ErrCorruption,
+					Description: fmt.Sprintf("failed to "+
+						"deserialized address index "+
+						"for key %x: %s", addrKey, err),
+				}
+			}
+
+			return nil, 0, err
+		}
+	}
+
+	return results, numToSkip, nil
+}
+
+// minEntriesToReachLevel returns the minimum number of entries that are
+// required to reach the given address index level.
+func minEntriesToReachLevel(level uint8) int {
+	maxEntriesForLevel := level0MaxEntries
+	minRequired := 1
+	for l := uint8(1); l <= level; l++ {
+		minRequired += maxEntriesForLevel
+		maxEntriesForLevel *= 2
+	}
+	return minRequired
+}
+
+// maxEntriesForLevel returns the maximum number of entries allowed for the
+// given address index level.
+func maxEntriesForLevel(level uint8) int {
+	numEntries := level0MaxEntries
+	for l := level; l > 0; l-- {
+		numEntries *= 2
+	}
+	return numEntries
+}
+
+// dbRemoveAddrIndexEntries removes the specified number of entries from from
+// the address index for the provided key. An assertion error will be returned
+// if the count exceeds the total number of entries in the index.
+func dbRemoveAddrIndexEntries(bucket internalBucket, addrKey [addrKeySize]byte, count int) error {
+	// Nothing to do if no entries are being deleted.
+	if count <= 0 {
+		return nil
+	}
+
+	// Make use of a local map to track pending updates and define a closure
+	// to apply it to the database. This is done in order to reduce the
+	// number of database reads and because there is more than one exit
+	// path that needs to apply the updates.
+	pendingUpdates := make(map[uint8][]byte)
+	applyPending := func() error {
+		for level, data := range pendingUpdates {
+			curLevelKey := keyForLevel(addrKey, level)
+			if len(data) == 0 {
+				err := bucket.Delete(curLevelKey[:])
+				if err != nil {
+					return err
+				}
+				continue
+			}
+			err := bucket.Put(curLevelKey[:], data)
+			if err != nil {
+				return err
+			}
+		}
+		return nil
+	}
+
+	// Loop forwards through the levels while removing entries until the
+	// specified number has been removed. This will potentially result in
+	// entirely empty lower levels which will be backfilled below.
+	var highestLoadedLevel uint8
+	numRemaining := count
+	for level := uint8(0); numRemaining > 0; level++ {
+		// Load the data for the level from the database.
+		curLevelKey := keyForLevel(addrKey, level)
+		curLevelData := bucket.Get(curLevelKey[:])
+		if len(curLevelData) == 0 && numRemaining > 0 {
+			return AssertError(fmt.Sprintf("dbRemoveAddrIndexEntries "+
+				"not enough entries for address key %x to "+
+				"delete %d entries", addrKey, count))
+		}
+		pendingUpdates[level] = curLevelData
+		highestLoadedLevel = level
+
+		// Delete the entire level as needed.
+		numEntries := len(curLevelData) / txEntrySize
+		if numRemaining >= numEntries {
+			pendingUpdates[level] = nil
+			numRemaining -= numEntries
+			continue
+		}
+
+		// Remove remaining entries to delete from the level.
+		offsetEnd := len(curLevelData) - (numRemaining * txEntrySize)
+		pendingUpdates[level] = curLevelData[:offsetEnd]
+		break
+	}
+
+	// When all elements in level 0 were not removed there is nothing left
+	// to do other than updating the database.
+	if len(pendingUpdates[0]) != 0 {
+		return applyPending()
+	}
+
+	// At this point there are one or more empty levels before the current
+	// level which need to be backfilled and the current level might have
+	// had some entries deleted from it as well. Since all levels after
+	// level 0 are required to either be empty, half full, or completely
+	// full, the current level must be adjusted accordingly by backfilling
+	// each previous levels in a way which satisfies the requirements. Any
+	// entries that are left are assigned to level 0 after the loop as they
+	// are guaranteed to fit by the logic in the loop. In other words, this
+	// effectively squashes all remaining entries in the current level into
+	// the lowest possible levels while following the level rules.
+	//
+	// Note that the level after the current level might also have entries
+	// and gaps are not allowed, so this also keeps track of the lowest
+	// empty level so the code below knows how far to backfill in case it is
+	// required.
+	lowestEmptyLevel := uint8(255)
+	curLevelData := pendingUpdates[highestLoadedLevel]
+	curLevelMaxEntries := maxEntriesForLevel(highestLoadedLevel)
+	for level := highestLoadedLevel; level > 0; level-- {
+		// When there are not enough entries left in the current level
+		// for the number that would be required to reach it, clear the
+		// the current level which effectively moves them all up to the
+		// previous level on the next iteration. Otherwise, there are
+		// are sufficient entries, so update the current level to
+		// contain as many entries as possible while still leaving
+		// enough remaining entries required to reach the level.
+		numEntries := len(curLevelData) / txEntrySize
+		prevLevelMaxEntries := curLevelMaxEntries / 2
+		minPrevRequired := minEntriesToReachLevel(level - 1)
+		if numEntries < prevLevelMaxEntries+minPrevRequired {
+			lowestEmptyLevel = level
+			pendingUpdates[level] = nil
+		} else {
+			// This level can only be completely full or half full,
+			// so choose the appropriate offset to ensure enough
+			// entries remain to reach the level.
+			var offset int
+			if numEntries-curLevelMaxEntries >= minPrevRequired {
+				offset = curLevelMaxEntries * txEntrySize
+			} else {
+				offset = prevLevelMaxEntries * txEntrySize
+			}
+			pendingUpdates[level] = curLevelData[:offset]
+			curLevelData = curLevelData[offset:]
+		}
+
+		curLevelMaxEntries = prevLevelMaxEntries
+	}
+	pendingUpdates[0] = curLevelData
+	if len(curLevelData) == 0 {
+		lowestEmptyLevel = 0
+	}
+
+	// When the highest loaded level is empty, it's possible the level after
+	// it still has data and thus that data needs to be backfilled as well.
+	for len(pendingUpdates[highestLoadedLevel]) == 0 {
+		// When the next level is empty too, the is no data left to
+		// continue backfilling, so there is nothing left to do.
+		// Otherwise, populate the pending updates map with the newly
+		// loaded data and update the highest loaded level accordingly.
+		level := highestLoadedLevel + 1
+		curLevelKey := keyForLevel(addrKey, level)
+		levelData := bucket.Get(curLevelKey[:])
+		if len(levelData) == 0 {
+			break
+		}
+		pendingUpdates[level] = levelData
+		highestLoadedLevel = level
+
+		// At this point the highest level is not empty, but it might
+		// be half full. When that is the case, move it up a level to
+		// simplify the code below which backfills all lower levels that
+		// are still empty. This also means the current level will be
+		// empty, so the loop will perform another another iteration to
+		// potentially backfill this level with data from the next one.
+		curLevelMaxEntries := maxEntriesForLevel(level)
+		if len(levelData)/txEntrySize != curLevelMaxEntries {
+			pendingUpdates[level] = nil
+			pendingUpdates[level-1] = levelData
+			level--
+			curLevelMaxEntries /= 2
+		}
+
+		// Backfill all lower levels that are still empty by iteratively
+		// halfing the data until the lowest empty level is filled.
+		for level > lowestEmptyLevel {
+			offset := (curLevelMaxEntries / 2) * txEntrySize
+			pendingUpdates[level] = levelData[:offset]
+			levelData = levelData[offset:]
+			pendingUpdates[level-1] = levelData
+			level--
+			curLevelMaxEntries /= 2
+		}
+
+		// The lowest possible empty level is now the highest loaded
+		// level.
+		lowestEmptyLevel = highestLoadedLevel
+	}
+
+	// Apply the pending updates.
+	return applyPending()
+}
+
+// addrToKey converts known address types to an addrindex key. An error is
+// returned for unsupported types.
+func addrToKey(addr util.Address) ([addrKeySize]byte, error) {
+	switch addr := addr.(type) {
+	case *util.AddressPubKeyHash:
+		var result [addrKeySize]byte
+		result[0] = addrKeyTypePubKeyHash
+		copy(result[1:], addr.Hash160()[:])
+		return result, nil
+
+	case *util.AddressScriptHash:
+		var result [addrKeySize]byte
+		result[0] = addrKeyTypeScriptHash
+		copy(result[1:], addr.Hash160()[:])
+		return result, nil
+	}
+
+	return [addrKeySize]byte{}, errUnsupportedAddressType
+}
+
+// AddrIndex implements a transaction by address index. That is to say, it
+// supports querying all transactions that reference a given address because
+// they are either crediting or debiting the address. The returned transactions
+// are ordered according to their order of appearance in the blockDAG. In
+// other words, first by block height and then by offset inside the block.
+//
+// In addition, support is provided for a memory-only index of unconfirmed
+// transactions such as those which are kept in the memory pool before inclusion
+// in a block.
+type AddrIndex struct {
+	// The following fields are set when the instance is created and can't
+	// be changed afterwards, so there is no need to protect them with a
+	// separate mutex.
+	db        database.DB
+	dagParams *dagconfig.Params
+
+	// The following fields are used to quickly link transactions and
+	// addresses that have not been included into a block yet when an
+	// address index is being maintained. The are protected by the
+	// unconfirmedLock field.
+	//
+	// The txnsByAddr field is used to keep an index of all transactions
+	// which either create an output to a given address or spend from a
+	// previous output to it keyed by the address.
+	//
+	// The addrsByTx field is essentially the reverse and is used to
+	// keep an index of all addresses which a given transaction involves.
+	// This allows fairly efficient updates when transactions are removed
+	// once they are included into a block.
+	unconfirmedLock sync.RWMutex
+	txnsByAddr      map[[addrKeySize]byte]map[daghash.TxID]*util.Tx
+	addrsByTx       map[daghash.TxID]map[[addrKeySize]byte]struct{}
+}
+
+// Ensure the AddrIndex type implements the Indexer interface.
+var _ Indexer = (*AddrIndex)(nil)
+
+// Ensure the AddrIndex type implements the NeedsInputser interface.
+var _ NeedsInputser = (*AddrIndex)(nil)
+
+// NeedsInputs signals that the index requires the referenced inputs in order
+// to properly create the index.
+//
+// This implements the NeedsInputser interface.
+func (idx *AddrIndex) NeedsInputs() bool {
+	return true
+}
+
+// Init is only provided to satisfy the Indexer interface as there is nothing to
+// initialize for this index.
+//
+// This is part of the Indexer interface.
+func (idx *AddrIndex) Init(db database.DB, _ *blockdag.BlockDAG) error {
+	idx.db = db
+	return nil
+}
+
+// Key returns the database key to use for the index as a byte slice.
+//
+// This is part of the Indexer interface.
+func (idx *AddrIndex) Key() []byte {
+	return addrIndexKey
+}
+
+// Name returns the human-readable name of the index.
+//
+// This is part of the Indexer interface.
+func (idx *AddrIndex) Name() string {
+	return addrIndexName
+}
+
+// Create is invoked when the indexer manager determines the index needs
+// to be created for the first time. It creates the bucket for the address
+// index.
+//
+// This is part of the Indexer interface.
+func (idx *AddrIndex) Create(dbTx database.Tx) error {
+	_, err := dbTx.Metadata().CreateBucket(addrIndexKey)
+	return err
+}
+
+// writeIndexData represents the address index data to be written for one block.
+// It consists of the address mapped to an ordered list of the transactions
+// that involve the address in block. It is ordered so the transactions can be
+// stored in the order they appear in the block.
+type writeIndexData map[[addrKeySize]byte][]int
+
+// indexScriptPubKey extracts all standard addresses from the passed public key
+// script and maps each of them to the associated transaction using the passed
+// map.
+func (idx *AddrIndex) indexScriptPubKey(data writeIndexData, scriptPubKey []byte, txIdx int) {
+	// Nothing to index if the script is non-standard or otherwise doesn't
+	// contain any addresses.
+	_, addr, err := txscript.ExtractScriptPubKeyAddress(scriptPubKey,
+		idx.dagParams)
+	if err != nil || addr == nil {
+		return
+	}
+
+	addrKey, err := addrToKey(addr)
+	if err != nil {
+		// Ignore unsupported address types.
+		return
+	}
+
+	// Avoid inserting the transaction more than once. Since the
+	// transactions are indexed serially any duplicates will be
+	// indexed in a row, so checking the most recent entry for the
+	// address is enough to detect duplicates.
+	indexedTxns := data[addrKey]
+	numTxns := len(indexedTxns)
+	if numTxns > 0 && indexedTxns[numTxns-1] == txIdx {
+		return
+	}
+	indexedTxns = append(indexedTxns, txIdx)
+	data[addrKey] = indexedTxns
+}
+
+// indexBlock extract all of the standard addresses from all of the transactions
+// in the passed block and maps each of them to the associated transaction using
+// the passed map.
+func (idx *AddrIndex) indexBlock(data writeIndexData, block *util.Block, dag *blockdag.BlockDAG) {
+	for txIdx, tx := range block.Transactions() {
+		// Coinbases do not reference any inputs. Since the block is
+		// required to have already gone through full validation, it has
+		// already been proven on the first transaction in the block is
+		// a coinbase.
+		if txIdx > util.CoinbaseTransactionIndex {
+			for _, txIn := range tx.MsgTx().TxIn {
+				// The UTXO should always have the input since
+				// the index contract requires it, however, be
+				// safe and simply ignore any missing entries.
+				entry, ok := dag.GetUTXOEntry(txIn.PreviousOutpoint)
+				if !ok {
+					continue
+				}
+
+				idx.indexScriptPubKey(data, entry.ScriptPubKey(), txIdx)
+			}
+		}
+
+		for _, txOut := range tx.MsgTx().TxOut {
+			idx.indexScriptPubKey(data, txOut.ScriptPubKey, txIdx)
+		}
+	}
+}
+
+// ConnectBlock is invoked by the index manager when a new block has been
+// connected to the DAG. This indexer adds a mapping for each address
+// the transactions in the block involve.
+//
+// This is part of the Indexer interface.
+func (idx *AddrIndex) ConnectBlock(dbTx database.Tx, block *util.Block, blockID uint64, dag *blockdag.BlockDAG,
+	_ blockdag.MultiBlockTxsAcceptanceData, _ blockdag.MultiBlockTxsAcceptanceData) error {
+
+	// The offset and length of the transactions within the serialized
+	// block.
+	txLocs, err := block.TxLoc()
+	if err != nil {
+		return err
+	}
+
+	// Build all of the address to transaction mappings in a local map.
+	addrsToTxns := make(writeIndexData)
+	idx.indexBlock(addrsToTxns, block, dag)
+
+	// Add all of the index entries for each address.
+	addrIdxBucket := dbTx.Metadata().Bucket(addrIndexKey)
+	for addrKey, txIdxs := range addrsToTxns {
+		for _, txIdx := range txIdxs {
+			err := dbPutAddrIndexEntry(addrIdxBucket, addrKey,
+				blockID, txLocs[txIdx])
+			if err != nil {
+				return err
+			}
+		}
+	}
+
+	return nil
+}
+
+// TxRegionsForAddress returns a slice of block regions which identify each
+// transaction that involves the passed address according to the specified
+// number to skip, number requested, and whether or not the results should be
+// reversed. It also returns the number actually skipped since it could be less
+// in the case where there are not enough entries.
+//
+// NOTE: These results only include transactions confirmed in blocks. See the
+// UnconfirmedTxnsForAddress method for obtaining unconfirmed transactions
+// that involve a given address.
+//
+// This function is safe for concurrent access.
+func (idx *AddrIndex) TxRegionsForAddress(dbTx database.Tx, addr util.Address, numToSkip, numRequested uint32, reverse bool) ([]database.BlockRegion, uint32, error) {
+	addrKey, err := addrToKey(addr)
+	if err != nil {
+		return nil, 0, err
+	}
+
+	var regions []database.BlockRegion
+	var skipped uint32
+	err = idx.db.View(func(dbTx database.Tx) error {
+		// Create closure to lookup the block hash given the ID using
+		// the database transaction.
+		fetchBlockHash := func(id []byte) (*daghash.Hash, error) {
+			// Deserialize and populate the result.
+			return blockdag.DBFetchBlockHashBySerializedID(dbTx, id)
+		}
+
+		var err error
+		addrIdxBucket := dbTx.Metadata().Bucket(addrIndexKey)
+		regions, skipped, err = dbFetchAddrIndexEntries(addrIdxBucket,
+			addrKey, numToSkip, numRequested, reverse,
+			fetchBlockHash)
+		return err
+	})
+
+	return regions, skipped, err
+}
+
+// indexUnconfirmedAddresses modifies the unconfirmed (memory-only) address
+// index to include mappings for the addresses encoded by the passed public key
+// script to the transaction.
+//
+// This function is safe for concurrent access.
+func (idx *AddrIndex) indexUnconfirmedAddresses(scriptPubKey []byte, tx *util.Tx) {
+	// The error is ignored here since the only reason it can fail is if the
+	// script fails to parse and it was already validated before being
+	// admitted to the mempool.
+	_, addr, _ := txscript.ExtractScriptPubKeyAddress(scriptPubKey,
+		idx.dagParams)
+	// Ignore unsupported address types.
+	addrKey, err := addrToKey(addr)
+	if err != nil {
+		return
+	}
+
+	// Add a mapping from the address to the transaction.
+	idx.unconfirmedLock.Lock()
+	addrIndexEntry := idx.txnsByAddr[addrKey]
+	if addrIndexEntry == nil {
+		addrIndexEntry = make(map[daghash.TxID]*util.Tx)
+		idx.txnsByAddr[addrKey] = addrIndexEntry
+	}
+	addrIndexEntry[*tx.ID()] = tx
+
+	// Add a mapping from the transaction to the address.
+	addrsByTxEntry := idx.addrsByTx[*tx.ID()]
+	if addrsByTxEntry == nil {
+		addrsByTxEntry = make(map[[addrKeySize]byte]struct{})
+		idx.addrsByTx[*tx.ID()] = addrsByTxEntry
+	}
+	addrsByTxEntry[addrKey] = struct{}{}
+	idx.unconfirmedLock.Unlock()
+}
+
+// AddUnconfirmedTx adds all addresses related to the transaction to the
+// unconfirmed (memory-only) address index.
+//
+// NOTE: This transaction MUST have already been validated by the memory pool
+// before calling this function with it and have all of the inputs available in
+// the provided utxo view. Failure to do so could result in some or all
+// addresses not being indexed.
+//
+// This function is safe for concurrent access.
+func (idx *AddrIndex) AddUnconfirmedTx(tx *util.Tx, utxoSet blockdag.UTXOSet) {
+	// Index addresses of all referenced previous transaction outputs.
+	//
+	// The existence checks are elided since this is only called after the
+	// transaction has already been validated and thus all inputs are
+	// already known to exist.
+	for _, txIn := range tx.MsgTx().TxIn {
+		entry, ok := utxoSet.Get(txIn.PreviousOutpoint)
+		if !ok {
+			// Ignore missing entries. This should never happen
+			// in practice since the function comments specifically
+			// call out all inputs must be available.
+			continue
+		}
+		idx.indexUnconfirmedAddresses(entry.ScriptPubKey(), tx)
+	}
+
+	// Index addresses of all created outputs.
+	for _, txOut := range tx.MsgTx().TxOut {
+		idx.indexUnconfirmedAddresses(txOut.ScriptPubKey, tx)
+	}
+}
+
+// RemoveUnconfirmedTx removes the passed transaction from the unconfirmed
+// (memory-only) address index.
+//
+// This function is safe for concurrent access.
+func (idx *AddrIndex) RemoveUnconfirmedTx(txID *daghash.TxID) {
+	idx.unconfirmedLock.Lock()
+	defer idx.unconfirmedLock.Unlock()
+
+	// Remove all address references to the transaction from the address
+	// index and remove the entry for the address altogether if it no longer
+	// references any transactions.
+	for addrKey := range idx.addrsByTx[*txID] {
+		delete(idx.txnsByAddr[addrKey], *txID)
+		if len(idx.txnsByAddr[addrKey]) == 0 {
+			delete(idx.txnsByAddr, addrKey)
+		}
+	}
+
+	// Remove the entry from the transaction to address lookup map as well.
+	delete(idx.addrsByTx, *txID)
+}
+
+// UnconfirmedTxnsForAddress returns all transactions currently in the
+// unconfirmed (memory-only) address index that involve the passed address.
+// Unsupported address types are ignored and will result in no results.
+//
+// This function is safe for concurrent access.
+func (idx *AddrIndex) UnconfirmedTxnsForAddress(addr util.Address) []*util.Tx {
+	// Ignore unsupported address types.
+	addrKey, err := addrToKey(addr)
+	if err != nil {
+		return nil
+	}
+
+	// Protect concurrent access.
+	idx.unconfirmedLock.RLock()
+	defer idx.unconfirmedLock.RUnlock()
+
+	// Return a new slice with the results if there are any. This ensures
+	// safe concurrency.
+	if txns, exists := idx.txnsByAddr[addrKey]; exists {
+		addressTxns := make([]*util.Tx, 0, len(txns))
+		for _, tx := range txns {
+			addressTxns = append(addressTxns, tx)
+		}
+		return addressTxns
+	}
+
+	return nil
+}
+
+// Recover is invoked when the indexer wasn't turned on for several blocks
+// and the indexer needs to close the gaps.
+//
+// This is part of the Indexer interface.
+func (idx *AddrIndex) Recover(dbTx database.Tx, currentBlockID, lastKnownBlockID uint64) error {
+	return errors.Errorf("addrindex was turned off for %d blocks and can't be recovered."+
+		" To resume working drop the addrindex with --dropaddrindex", lastKnownBlockID-currentBlockID)
+}
+
+// NewAddrIndex returns a new instance of an indexer that is used to create a
+// mapping of all addresses in the blockDAG to the respective transactions
+// that involve them.
+//
+// It implements the Indexer interface which plugs into the IndexManager that in
+// turn is used by the blockDAG package. This allows the index to be
+// seamlessly maintained along with the DAG.
+func NewAddrIndex(dagParams *dagconfig.Params) *AddrIndex {
+	return &AddrIndex{
+		dagParams:  dagParams,
+		txnsByAddr: make(map[[addrKeySize]byte]map[daghash.TxID]*util.Tx),
+		addrsByTx:  make(map[daghash.TxID]map[[addrKeySize]byte]struct{}),
+	}
+}
+
+// DropAddrIndex drops the address index from the provided database if it
+// exists.
+func DropAddrIndex(db database.DB, interrupt <-chan struct{}) error {
+	return dropIndex(db, addrIndexKey, addrIndexName, interrupt)
+}

blockdag/indexers/addrindex_test.go

@@ -0,0 +1,277 @@
+// Copyright (c) 2016 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package indexers
+
+import (
+	"bytes"
+	"fmt"
+	"github.com/pkg/errors"
+	"testing"
+
+	"github.com/kaspanet/kaspad/wire"
+)
+
+// addrIndexBucket provides a mock address index database bucket by implementing
+// the internalBucket interface.
+type addrIndexBucket struct {
+	levels map[[levelKeySize]byte][]byte
+}
+
+// Clone returns a deep copy of the mock address index bucket.
+func (b *addrIndexBucket) Clone() *addrIndexBucket {
+	levels := make(map[[levelKeySize]byte][]byte)
+	for k, v := range b.levels {
+		vCopy := make([]byte, len(v))
+		copy(vCopy, v)
+		levels[k] = vCopy
+	}
+	return &addrIndexBucket{levels: levels}
+}
+
+// Get returns the value associated with the key from the mock address index
+// bucket.
+//
+// This is part of the internalBucket interface.
+func (b *addrIndexBucket) Get(key []byte) []byte {
+	var levelKey [levelKeySize]byte
+	copy(levelKey[:], key)
+	return b.levels[levelKey]
+}
+
+// Put stores the provided key/value pair to the mock address index bucket.
+//
+// This is part of the internalBucket interface.
+func (b *addrIndexBucket) Put(key []byte, value []byte) error {
+	var levelKey [levelKeySize]byte
+	copy(levelKey[:], key)
+	b.levels[levelKey] = value
+	return nil
+}
+
+// Delete removes the provided key from the mock address index bucket.
+//
+// This is part of the internalBucket interface.
+func (b *addrIndexBucket) Delete(key []byte) error {
+	var levelKey [levelKeySize]byte
+	copy(levelKey[:], key)
+	delete(b.levels, levelKey)
+	return nil
+}
+
+// printLevels returns a string with a visual representation of the provided
+// address key taking into account the max size of each level. It is useful
+// when creating and debugging test cases.
+func (b *addrIndexBucket) printLevels(addrKey [addrKeySize]byte) string {
+	highestLevel := uint8(0)
+	for k := range b.levels {
+		if !bytes.Equal(k[:levelOffset], addrKey[:]) {
+			continue
+		}
+		level := uint8(k[levelOffset])
+		if level > highestLevel {
+			highestLevel = level
+		}
+	}
+
+	var levelBuf bytes.Buffer
+	_, _ = levelBuf.WriteString("\n")
+	maxEntries := level0MaxEntries
+	for level := uint8(0); level <= highestLevel; level++ {
+		data := b.levels[keyForLevel(addrKey, level)]
+		numEntries := len(data) / txEntrySize
+		for i := 0; i < numEntries; i++ {
+			start := i * txEntrySize
+			num := byteOrder.Uint32(data[start:])
+			_, _ = levelBuf.WriteString(fmt.Sprintf("%02d ", num))
+		}
+		for i := numEntries; i < maxEntries; i++ {
+			_, _ = levelBuf.WriteString("_  ")
+		}
+		_, _ = levelBuf.WriteString("\n")
+		maxEntries *= 2
+	}
+
+	return levelBuf.String()
+}
+
+// sanityCheck ensures that all data stored in the bucket for the given address
+// adheres to the level-based rules described by the address index
+// documentation.
+func (b *addrIndexBucket) sanityCheck(addrKey [addrKeySize]byte, expectedTotal int) error {
+	// Find the highest level for the key.
+	highestLevel := uint8(0)
+	for k := range b.levels {
+		if !bytes.Equal(k[:levelOffset], addrKey[:]) {
+			continue
+		}
+		level := uint8(k[levelOffset])
+		if level > highestLevel {
+			highestLevel = level
+		}
+	}
+
+	// Ensure the expected total number of entries are present and that
+	// all levels adhere to the rules described in the address index
+	// documentation.
+	var totalEntries int
+	maxEntries := level0MaxEntries
+	for level := uint8(0); level <= highestLevel; level++ {
+		// Level 0 can'have more entries than the max allowed if the
+		// levels after it have data and it can't be empty. All other
+		// levels must either be half full or full.
+		data := b.levels[keyForLevel(addrKey, level)]
+		numEntries := len(data) / txEntrySize
+		totalEntries += numEntries
+		if level == 0 {
+			if (highestLevel != 0 && numEntries == 0) ||
+				numEntries > maxEntries {
+
+				return errors.Errorf("level %d has %d entries",
+					level, numEntries)
+			}
+		} else if numEntries != maxEntries && numEntries != maxEntries/2 {
+			return errors.Errorf("level %d has %d entries", level,
+				numEntries)
+		}
+		maxEntries *= 2
+	}
+	if totalEntries != expectedTotal {
+		return errors.Errorf("expected %d entries - got %d", expectedTotal,
+			totalEntries)
+	}
+
+	// Ensure all of the numbers are in order starting from the highest
+	// level moving to the lowest level.
+	expectedNum := uint32(0)
+	for level := highestLevel + 1; level > 0; level-- {
+		data := b.levels[keyForLevel(addrKey, level)]
+		numEntries := len(data) / txEntrySize
+		for i := 0; i < numEntries; i++ {
+			start := i * txEntrySize
+			num := byteOrder.Uint32(data[start:])
+			if num != expectedNum {
+				return errors.Errorf("level %d offset %d does "+
+					"not contain the expected number of "+
+					"%d - got %d", level, i, num,
+					expectedNum)
+			}
+			expectedNum++
+		}
+	}
+
+	return nil
+}
+
+// TestAddrIndexLevels ensures that adding and deleting entries to the address
+// index creates multiple levels as described by the address index
+// documentation.
+func TestAddrIndexLevels(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name        string
+		key         [addrKeySize]byte
+		numInsert   int
+		printLevels bool // Set to help debug a specific test.
+	}{
+		{
+			name:      "level 0 not full",
+			numInsert: level0MaxEntries - 1,
+		},
+		{
+			name:      "level 1 half",
+			numInsert: level0MaxEntries + 1,
+		},
+		{
+			name:      "level 1 full",
+			numInsert: level0MaxEntries*2 + 1,
+		},
+		{
+			name:      "level 2 half, level 1 half",
+			numInsert: level0MaxEntries*3 + 1,
+		},
+		{
+			name:      "level 2 half, level 1 full",
+			numInsert: level0MaxEntries*4 + 1,
+		},
+		{
+			name:      "level 2 full, level 1 half",
+			numInsert: level0MaxEntries*5 + 1,
+		},
+		{
+			name:      "level 2 full, level 1 full",
+			numInsert: level0MaxEntries*6 + 1,
+		},
+		{
+			name:      "level 3 half, level 2 half, level 1 half",
+			numInsert: level0MaxEntries*7 + 1,
+		},
+		{
+			name:      "level 3 full, level 2 half, level 1 full",
+			numInsert: level0MaxEntries*12 + 1,
+		},
+	}
+
+nextTest:
+	for testNum, test := range tests {
+		// Insert entries in order.
+		populatedBucket := &addrIndexBucket{
+			levels: make(map[[levelKeySize]byte][]byte),
+		}
+		for i := 0; i < test.numInsert; i++ {
+			txLoc := wire.TxLoc{TxStart: i * 2}
+			err := dbPutAddrIndexEntry(populatedBucket, test.key,
+				uint64(i), txLoc)
+			if err != nil {
+				t.Errorf("dbPutAddrIndexEntry #%d (%s) - "+
+					"unexpected error: %v", testNum,
+					test.name, err)
+				continue nextTest
+			}
+		}
+		if test.printLevels {
+			t.Log(populatedBucket.printLevels(test.key))
+		}
+
+		// Delete entries from the populated bucket until all entries
+		// have been deleted. The bucket is reset to the fully
+		// populated bucket on each iteration so every combination is
+		// tested. Notice the upper limit purposes exceeds the number
+		// of entries to ensure attempting to delete more entries than
+		// there are works correctly.
+		for numDelete := 0; numDelete <= test.numInsert+1; numDelete++ {
+			// Clone populated bucket to run each delete against.
+			bucket := populatedBucket.Clone()
+
+			// Remove the number of entries for this iteration.
+			err := dbRemoveAddrIndexEntries(bucket, test.key,
+				numDelete)
+			if err != nil {
+				if numDelete <= test.numInsert {
+					t.Errorf("dbRemoveAddrIndexEntries (%s) "+
+						" delete %d - unexpected error: "+
+						"%v", test.name, numDelete, err)
+					continue nextTest
+				}
+			}
+			if test.printLevels {
+				t.Log(bucket.printLevels(test.key))
+			}
+
+			// Sanity check the levels to ensure the adhere to all
+			// rules.
+			numExpected := test.numInsert
+			if numDelete <= test.numInsert {
+				numExpected -= numDelete
+			}
+			err = bucket.sanityCheck(test.key, numExpected)
+			if err != nil {
+				t.Errorf("sanity check fail (%s) delete %d: %v",
+					test.name, numDelete, err)
+				continue nextTest
+			}
+		}
+	}
+}

blockdag/indexers/common.go

@@ -0,0 +1,112 @@
+// Copyright (c) 2016 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+/*
+Package indexers implements optional block DAG indexes.
+*/
+package indexers
+
+import (
+	"encoding/binary"
+	"github.com/kaspanet/kaspad/blockdag"
+	"github.com/kaspanet/kaspad/database"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/pkg/errors"
+)
+
+var (
+	// byteOrder is the preferred byte order used for serializing numeric
+	// fields for storage in the database.
+	byteOrder = binary.LittleEndian
+
+	// errInterruptRequested indicates that an operation was cancelled due
+	// to a user-requested interrupt.
+	errInterruptRequested = errors.New("interrupt requested")
+)
+
+// NeedsInputser provides a generic interface for an indexer to specify the it
+// requires the ability to look up inputs for a transaction.
+type NeedsInputser interface {
+	NeedsInputs() bool
+}
+
+// Indexer provides a generic interface for an indexer that is managed by an
+// index manager such as the Manager type provided by this package.
+type Indexer interface {
+	// Key returns the key of the index as a byte slice.
+	Key() []byte
+
+	// Name returns the human-readable name of the index.
+	Name() string
+
+	// Create is invoked when the indexer manager determines the index needs
+	// to be created for the first time.
+	Create(dbTx database.Tx) error
+
+	// Init is invoked when the index manager is first initializing the
+	// index. This differs from the Create method in that it is called on
+	// every load, including the case the index was just created.
+	Init(db database.DB, dag *blockdag.BlockDAG) error
+
+	// ConnectBlock is invoked when the index manager is notified that a new
+	// block has been connected to the DAG.
+	ConnectBlock(dbTx database.Tx,
+		block *util.Block,
+		blockID uint64,
+		dag *blockdag.BlockDAG,
+		acceptedTxsData blockdag.MultiBlockTxsAcceptanceData,
+		virtualTxsAcceptanceData blockdag.MultiBlockTxsAcceptanceData) error
+
+	// Recover is invoked when the indexer wasn't turned on for several blocks
+	// and the indexer needs to close the gaps.
+	Recover(dbTx database.Tx, currentBlockID, lastKnownBlockID uint64) error
+}
+
+// AssertError identifies an error that indicates an internal code consistency
+// issue and should be treated as a critical and unrecoverable error.
+type AssertError string
+
+// Error returns the assertion error as a huma-readable string and satisfies
+// the error interface.
+func (e AssertError) Error() string {
+	return "assertion failed: " + string(e)
+}
+
+// errDeserialize signifies that a problem was encountered when deserializing
+// data.
+type errDeserialize string
+
+// Error implements the error interface.
+func (e errDeserialize) Error() string {
+	return string(e)
+}
+
+// isDeserializeErr returns whether or not the passed error is an errDeserialize
+// error.
+func isDeserializeErr(err error) bool {
+	_, ok := err.(errDeserialize)
+	return ok
+}
+
+// internalBucket is an abstraction over a database bucket. It is used to make
+// the code easier to test since it allows mock objects in the tests to only
+// implement these functions instead of everything a database.Bucket supports.
+type internalBucket interface {
+	Get(key []byte) []byte
+	Put(key []byte, value []byte) error
+	Delete(key []byte) error
+}
+
+// interruptRequested returns true when the provided channel has been closed.
+// This simplifies early shutdown slightly since the caller can just use an if
+// statement instead of a select.
+func interruptRequested(interrupted <-chan struct{}) bool {
+	select {
+	case <-interrupted:
+		return true
+	default:
+	}
+
+	return false
+}

blockdag/indexers/log.go

@@ -0,0 +1,13 @@
+// Copyright (c) 2016 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package indexers
+
+import (
+	"github.com/kaspanet/kaspad/logger"
+	"github.com/kaspanet/kaspad/util/panics"
+)
+
+var log, _ = logger.Get(logger.SubsystemTags.INDX)
+var spawn = panics.GoroutineWrapperFunc(log)

blockdag/indexers/manager.go

@@ -0,0 +1,389 @@
+// Copyright (c) 2016 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package indexers
+
+import (
+	"github.com/kaspanet/kaspad/blockdag"
+	"github.com/kaspanet/kaspad/database"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/kaspanet/kaspad/util/daghash"
+)
+
+var (
+	// indexTipsBucketName is the name of the db bucket used to house the
+	// current tip of each index.
+	indexTipsBucketName = []byte("idxtips")
+
+	indexCurrentBlockIDBucketName = []byte("idxcurrentblockid")
+)
+
+// Manager defines an index manager that manages multiple optional indexes and
+// implements the blockdag.IndexManager interface so it can be seamlessly
+// plugged into normal DAG processing.
+type Manager struct {
+	db             database.DB
+	enabledIndexes []Indexer
+}
+
+// Ensure the Manager type implements the blockdag.IndexManager interface.
+var _ blockdag.IndexManager = (*Manager)(nil)
+
+// indexDropKey returns the key for an index which indicates it is in the
+// process of being dropped.
+func indexDropKey(idxKey []byte) []byte {
+	dropKey := make([]byte, len(idxKey)+1)
+	dropKey[0] = 'd'
+	copy(dropKey[1:], idxKey)
+	return dropKey
+}
+
+// maybeFinishDrops determines if each of the enabled indexes are in the middle
+// of being dropped and finishes dropping them when the are. This is necessary
+// because dropping and index has to be done in several atomic steps rather than
+// one big atomic step due to the massive number of entries.
+func (m *Manager) maybeFinishDrops(interrupt <-chan struct{}) error {
+	indexNeedsDrop := make([]bool, len(m.enabledIndexes))
+	err := m.db.View(func(dbTx database.Tx) error {
+		// None of the indexes needs to be dropped if the index tips
+		// bucket hasn't been created yet.
+		indexesBucket := dbTx.Metadata().Bucket(indexTipsBucketName)
+		if indexesBucket == nil {
+			return nil
+		}
+
+		// Mark the indexer as requiring a drop if one is already in
+		// progress.
+		for i, indexer := range m.enabledIndexes {
+			dropKey := indexDropKey(indexer.Key())
+			if indexesBucket.Get(dropKey) != nil {
+				indexNeedsDrop[i] = true
+			}
+		}
+
+		return nil
+	})
+	if err != nil {
+		return err
+	}
+
+	if interruptRequested(interrupt) {
+		return errInterruptRequested
+	}
+
+	// Finish dropping any of the enabled indexes that are already in the
+	// middle of being dropped.
+	for i, indexer := range m.enabledIndexes {
+		if !indexNeedsDrop[i] {
+			continue
+		}
+
+		log.Infof("Resuming %s drop", indexer.Name())
+		err := dropIndex(m.db, indexer.Key(), indexer.Name(), interrupt)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+// maybeCreateIndexes determines if each of the enabled indexes have already
+// been created and creates them if not.
+func (m *Manager) maybeCreateIndexes(dbTx database.Tx) error {
+	indexesBucket := dbTx.Metadata().Bucket(indexTipsBucketName)
+	for _, indexer := range m.enabledIndexes {
+		// Nothing to do if the index tip already exists.
+		idxKey := indexer.Key()
+		if indexesBucket.Get(idxKey) != nil {
+			continue
+		}
+
+		// The tip for the index does not exist, so create it and
+		// invoke the create callback for the index so it can perform
+		// any one-time initialization it requires.
+		if err := indexer.Create(dbTx); err != nil {
+			return err
+		}
+
+		// TODO (Mike): this is temporary solution to prevent node from not starting
+		// because it thinks indexers are not initialized.
+		// Indexers, however, do not work properly, and a general solution to their work operation is required
+		indexesBucket.Put(idxKey, []byte{0})
+	}
+
+	return nil
+}
+
+// Init initializes the enabled indexes. This is called during DAG
+// initialization and primarily consists of catching up all indexes to the
+// current tips. This is necessary since each index can be disabled
+// and re-enabled at any time and attempting to catch-up indexes at the same
+// time new blocks are being downloaded would lead to an overall longer time to
+// catch up due to the I/O contention.
+//
+// This is part of the blockdag.IndexManager interface.
+func (m *Manager) Init(db database.DB, blockDAG *blockdag.BlockDAG, interrupt <-chan struct{}) error {
+	// Nothing to do when no indexes are enabled.
+	if len(m.enabledIndexes) == 0 {
+		return nil
+	}
+
+	if interruptRequested(interrupt) {
+		return errInterruptRequested
+	}
+
+	m.db = db
+
+	// Finish and drops that were previously interrupted.
+	if err := m.maybeFinishDrops(interrupt); err != nil {
+		return err
+	}
+
+	// Create the initial state for the indexes as needed.
+	err := m.db.Update(func(dbTx database.Tx) error {
+		// Create the bucket for the current tips as needed.
+		meta := dbTx.Metadata()
+		_, err := meta.CreateBucketIfNotExists(indexTipsBucketName)
+		if err != nil {
+			return err
+		}
+		if _, err := meta.CreateBucketIfNotExists(indexCurrentBlockIDBucketName); err != nil {
+			return err
+		}
+
+		return m.maybeCreateIndexes(dbTx)
+	})
+	if err != nil {
+		return err
+	}
+
+	// Initialize each of the enabled indexes.
+	for _, indexer := range m.enabledIndexes {
+		if err := indexer.Init(db, blockDAG); err != nil {
+			return err
+		}
+	}
+
+	return m.recoverIfNeeded()
+}
+
+// recoverIfNeeded checks if the node worked for some time
+// without one of the current enabled indexes, and if it's
+// the case, recovers the missing blocks from the index.
+func (m *Manager) recoverIfNeeded() error {
+	return m.db.Update(func(dbTx database.Tx) error {
+		lastKnownBlockID := blockdag.DBFetchCurrentBlockID(dbTx)
+		for _, indexer := range m.enabledIndexes {
+			serializedCurrentIdxBlockID := dbTx.Metadata().Bucket(indexCurrentBlockIDBucketName).Get(indexer.Key())
+			currentIdxBlockID := uint64(0)
+			if serializedCurrentIdxBlockID != nil {
+				currentIdxBlockID = blockdag.DeserializeBlockID(serializedCurrentIdxBlockID)
+			}
+			if lastKnownBlockID > currentIdxBlockID {
+				err := indexer.Recover(dbTx, currentIdxBlockID, lastKnownBlockID)
+				if err != nil {
+					return err
+				}
+			}
+		}
+		return nil
+	})
+}
+
+// ConnectBlock must be invoked when a block is added to the DAG. It
+// keeps track of the state of each index it is managing, performs some sanity
+// checks, and invokes each indexer.
+//
+// This is part of the blockdag.IndexManager interface.
+func (m *Manager) ConnectBlock(dbTx database.Tx, block *util.Block, blockID uint64, dag *blockdag.BlockDAG,
+	txsAcceptanceData blockdag.MultiBlockTxsAcceptanceData, virtualTxsAcceptanceData blockdag.MultiBlockTxsAcceptanceData) error {
+
+	// Call each of the currently active optional indexes with the block
+	// being connected so they can update accordingly.
+	for _, index := range m.enabledIndexes {
+		// Notify the indexer with the connected block so it can index it.
+		if err := index.ConnectBlock(dbTx, block, blockID, dag, txsAcceptanceData, virtualTxsAcceptanceData); err != nil {
+			return err
+		}
+	}
+
+	// Add the new block ID index entry for the block being connected and
+	// update the current internal block ID accordingly.
+	err := m.updateIndexersWithCurrentBlockID(dbTx, block.Hash(), blockID)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+func (m *Manager) updateIndexersWithCurrentBlockID(dbTx database.Tx, blockHash *daghash.Hash, blockID uint64) error {
+	serializedBlockID := blockdag.SerializeBlockID(blockID)
+	for _, index := range m.enabledIndexes {
+		err := dbTx.Metadata().Bucket(indexCurrentBlockIDBucketName).Put(index.Key(), serializedBlockID)
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// NewManager returns a new index manager with the provided indexes enabled.
+//
+// The manager returned satisfies the blockdag.IndexManager interface and thus
+// cleanly plugs into the normal blockdag processing path.
+func NewManager(enabledIndexes []Indexer) *Manager {
+	return &Manager{
+		enabledIndexes: enabledIndexes,
+	}
+}
+
+// dropIndex drops the passed index from the database. Since indexes can be
+// massive, it deletes the index in multiple database transactions in order to
+// keep memory usage to reasonable levels. It also marks the drop in progress
+// so the drop can be resumed if it is stopped before it is done before the
+// index can be used again.
+func dropIndex(db database.DB, idxKey []byte, idxName string, interrupt <-chan struct{}) error {
+	// Nothing to do if the index doesn't already exist.
+	var needsDelete bool
+	err := db.View(func(dbTx database.Tx) error {
+		indexesBucket := dbTx.Metadata().Bucket(indexTipsBucketName)
+		if indexesBucket != nil && indexesBucket.Get(idxKey) != nil {
+			needsDelete = true
+		}
+		return nil
+	})
+	if err != nil {
+		return err
+	}
+	if !needsDelete {
+		log.Infof("Not dropping %s because it does not exist", idxName)
+		return nil
+	}
+
+	// Mark that the index is in the process of being dropped so that it
+	// can be resumed on the next start if interrupted before the process is
+	// complete.
+	log.Infof("Dropping all %s entries. This might take a while...",
+		idxName)
+	err = db.Update(func(dbTx database.Tx) error {
+		indexesBucket := dbTx.Metadata().Bucket(indexTipsBucketName)
+		return indexesBucket.Put(indexDropKey(idxKey), idxKey)
+	})
+	if err != nil {
+		return err
+	}
+
+	// Since the indexes can be so large, attempting to simply delete
+	// the bucket in a single database transaction would result in massive
+	// memory usage and likely crash many systems due to ulimits. In order
+	// to avoid this, use a cursor to delete a maximum number of entries out
+	// of the bucket at a time. Recurse buckets depth-first to delete any
+	// sub-buckets.
+	const maxDeletions = 2000000
+	var totalDeleted uint64
+
+	// Recurse through all buckets in the index, cataloging each for
+	// later deletion.
+	var subBuckets [][][]byte
+	var subBucketClosure func(database.Tx, []byte, [][]byte) error
+	subBucketClosure = func(dbTx database.Tx,
+		subBucket []byte, tlBucket [][]byte) error {
+		// Get full bucket name and append to subBuckets for later
+		// deletion.
+		var bucketName [][]byte
+		if (tlBucket == nil) || (len(tlBucket) == 0) {
+			bucketName = append(bucketName, subBucket)
+		} else {
+			bucketName = append(tlBucket, subBucket)
+		}
+		subBuckets = append(subBuckets, bucketName)
+		// Recurse sub-buckets to append to subBuckets slice.
+		bucket := dbTx.Metadata()
+		for _, subBucketName := range bucketName {
+			bucket = bucket.Bucket(subBucketName)
+		}
+		return bucket.ForEachBucket(func(k []byte) error {
+			return subBucketClosure(dbTx, k, bucketName)
+		})
+	}
+
+	// Call subBucketClosure with top-level bucket.
+	err = db.View(func(dbTx database.Tx) error {
+		return subBucketClosure(dbTx, idxKey, nil)
+	})
+	if err != nil {
+		return nil
+	}
+
+	// Iterate through each sub-bucket in reverse, deepest-first, deleting
+	// all keys inside them and then dropping the buckets themselves.
+	for i := range subBuckets {
+		bucketName := subBuckets[len(subBuckets)-1-i]
+		// Delete maxDeletions key/value pairs at a time.
+		for numDeleted := maxDeletions; numDeleted == maxDeletions; {
+			numDeleted = 0
+			err := db.Update(func(dbTx database.Tx) error {
+				subBucket := dbTx.Metadata()
+				for _, subBucketName := range bucketName {
+					subBucket = subBucket.Bucket(subBucketName)
+				}
+				cursor := subBucket.Cursor()
+				for ok := cursor.First(); ok; ok = cursor.Next() &&
+					numDeleted < maxDeletions {
+
+					if err := cursor.Delete(); err != nil {
+						return err
+					}
+					numDeleted++
+				}
+				return nil
+			})
+			if err != nil {
+				return err
+			}
+
+			if numDeleted > 0 {
+				totalDeleted += uint64(numDeleted)
+				log.Infof("Deleted %d keys (%d total) from %s",
+					numDeleted, totalDeleted, idxName)
+			}
+		}
+
+		if interruptRequested(interrupt) {
+			return errInterruptRequested
+		}
+
+		// Drop the bucket itself.
+		err = db.Update(func(dbTx database.Tx) error {
+			bucket := dbTx.Metadata()
+			for j := 0; j < len(bucketName)-1; j++ {
+				bucket = bucket.Bucket(bucketName[j])
+			}
+			return bucket.DeleteBucket(bucketName[len(bucketName)-1])
+		})
+	}
+
+	// Remove the index tip, index bucket, and in-progress drop flag now
+	// that all index entries have been removed.
+	err = db.Update(func(dbTx database.Tx) error {
+		meta := dbTx.Metadata()
+		indexesBucket := meta.Bucket(indexTipsBucketName)
+		if err := indexesBucket.Delete(idxKey); err != nil {
+			return err
+		}
+
+		if err := meta.Bucket(indexCurrentBlockIDBucketName).Delete(idxKey); err != nil {
+			return err
+		}
+
+		return indexesBucket.Delete(indexDropKey(idxKey))
+	})
+	if err != nil {
+		return err
+	}
+
+	log.Infof("Dropped %s", idxName)
+	return nil
+}

blockdag/indexers/txindex.go

@@ -0,0 +1,438 @@
+// Copyright (c) 2016 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package indexers
+
+import (
+	"fmt"
+	"github.com/kaspanet/kaspad/blockdag"
+	"github.com/kaspanet/kaspad/database"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/wire"
+	"github.com/pkg/errors"
+)
+
+const (
+	// txIndexName is the human-readable name for the index.
+	txIndexName = "transaction index"
+
+	includingBlocksIndexKeyEntrySize = 8 // 4 bytes for offset + 4 bytes for transaction length
+)
+
+var (
+	includingBlocksIndexKey = []byte("includingblocksidx")
+
+	acceptingBlocksIndexKey = []byte("acceptingblocksidx")
+)
+
+// txsAcceptedByVirtual is the in-memory index of txIDs that were accepted
+// by the current virtual
+var txsAcceptedByVirtual map[daghash.TxID]bool
+
+// -----------------------------------------------------------------------------
+// The transaction index consists of an entry for every transaction in the DAG.
+//
+// There are two buckets used in total. The first bucket maps the hash of
+// each transaction to its location in each block it's included in. The second bucket
+// contains all of the blocks that from their viewpoint the transaction has been
+// accepted (i.e. the transaction is found in their blue set without double spends),
+// and their blue block (or themselves) that included the transaction.
+//
+// NOTE: Although it is technically possible for multiple transactions to have
+// the same hash as long as the previous transaction with the same hash is fully
+// spent, this code only stores the most recent one because doing otherwise
+// would add a non-trivial amount of space and overhead for something that will
+// realistically never happen per the probability and even if it did, the old
+// one must be fully spent and so the most likely transaction a caller would
+// want for a given hash is the most recent one anyways.
+//
+// The including blocks index contains a sub bucket for each transaction hash (32 byte each), that its serialized format is:
+//
+//   <block id> = <start offset><tx length>
+//
+//   Field           Type              Size
+//   block id        uint64          8 bytes
+//   start offset    uint32          4 bytes
+//   tx length       uint32          4 bytes
+//   -----
+//   Total: 16 bytes
+//
+// The accepting blocks index contains a sub bucket for each transaction hash (32 byte each), that its serialized format is:
+//
+//   <accepting block id> = <including block id>
+//
+//   Field           Type              Size
+//   accepting block id        uint64          8 bytes
+//   including block id        uint64          8 bytes
+//   -----
+//   Total: 16 bytes
+//
+// -----------------------------------------------------------------------------
+
+func putIncludingBlocksEntry(target []byte, txLoc wire.TxLoc) {
+	byteOrder.PutUint32(target, uint32(txLoc.TxStart))
+	byteOrder.PutUint32(target[4:], uint32(txLoc.TxLen))
+}
+
+func dbPutIncludingBlocksEntry(dbTx database.Tx, txID *daghash.TxID, blockID uint64, serializedData []byte) error {
+	bucket, err := dbTx.Metadata().Bucket(includingBlocksIndexKey).CreateBucketIfNotExists(txID[:])
+	if err != nil {
+		return err
+	}
+	return bucket.Put(blockdag.SerializeBlockID(blockID), serializedData)
+}
+
+func dbPutAcceptingBlocksEntry(dbTx database.Tx, txID *daghash.TxID, blockID uint64, serializedData []byte) error {
+	bucket, err := dbTx.Metadata().Bucket(acceptingBlocksIndexKey).CreateBucketIfNotExists(txID[:])
+	if err != nil {
+		return err
+	}
+	return bucket.Put(blockdag.SerializeBlockID(blockID), serializedData)
+}
+
+// dbFetchFirstTxRegion uses an existing database transaction to fetch the block
+// region for the provided transaction hash from the transaction index. When
+// there is no entry for the provided hash, nil will be returned for the both
+// the region and the error.
+//
+// P.S Because the transaction can be found in multiple blocks, this function arbitarily
+// returns the first block region that is stored in the txindex.
+func dbFetchFirstTxRegion(dbTx database.Tx, txID *daghash.TxID) (*database.BlockRegion, error) {
+	// Load the record from the database and return now if it doesn't exist.
+	txBucket := dbTx.Metadata().Bucket(includingBlocksIndexKey).Bucket(txID[:])
+	if txBucket == nil {
+		return nil, database.Error{
+			ErrorCode: database.ErrCorruption,
+			Description: fmt.Sprintf("No block region "+
+				"was found for %s", txID),
+		}
+	}
+	cursor := txBucket.Cursor()
+	if ok := cursor.First(); !ok {
+		return nil, database.Error{
+			ErrorCode: database.ErrCorruption,
+			Description: fmt.Sprintf("No block region "+
+				"was found for %s", txID),
+		}
+	}
+	serializedBlockID := cursor.Key()
+	serializedData := cursor.Value()
+	if len(serializedData) == 0 {
+		return nil, nil
+	}
+
+	// Ensure the serialized data has enough bytes to properly deserialize.
+	if len(serializedData) < includingBlocksIndexKeyEntrySize {
+		return nil, database.Error{
+			ErrorCode: database.ErrCorruption,
+			Description: fmt.Sprintf("corrupt transaction index "+
+				"entry for %s", txID),
+		}
+	}
+
+	// Load the block hash associated with the block ID.
+	hash, err := blockdag.DBFetchBlockHashBySerializedID(dbTx, serializedBlockID)
+	if err != nil {
+		return nil, database.Error{
+			ErrorCode: database.ErrCorruption,
+			Description: fmt.Sprintf("corrupt transaction index "+
+				"entry for %s: %s", txID, err),
+		}
+	}
+
+	// Deserialize the final entry.
+	region := database.BlockRegion{Hash: &daghash.Hash{}}
+	copy(region.Hash[:], hash[:])
+	region.Offset = byteOrder.Uint32(serializedData[:4])
+	region.Len = byteOrder.Uint32(serializedData[4:])
+
+	return &region, nil
+}
+
+// dbAddTxIndexEntries uses an existing database transaction to add a
+// transaction index entry for every transaction in the passed block.
+func dbAddTxIndexEntries(dbTx database.Tx, block *util.Block, blockID uint64, multiBlockTxsAcceptanceData blockdag.MultiBlockTxsAcceptanceData) error {
+	// The offset and length of the transactions within the serialized
+	// block.
+	txLocs, err := block.TxLoc()
+	if err != nil {
+		return err
+	}
+
+	// As an optimization, allocate a single slice big enough to hold all
+	// of the serialized transaction index entries for the block and
+	// serialize them directly into the slice. Then, pass the appropriate
+	// subslice to the database to be written. This approach significantly
+	// cuts down on the number of required allocations.
+	includingBlocksOffset := 0
+	serializedIncludingBlocksValues := make([]byte, len(block.Transactions())*includingBlocksIndexKeyEntrySize)
+	for i, tx := range block.Transactions() {
+		putIncludingBlocksEntry(serializedIncludingBlocksValues[includingBlocksOffset:], txLocs[i])
+		endOffset := includingBlocksOffset + includingBlocksIndexKeyEntrySize
+		err := dbPutIncludingBlocksEntry(dbTx, tx.ID(), blockID,
+			serializedIncludingBlocksValues[includingBlocksOffset:endOffset:endOffset])
+		if err != nil {
+			return err
+		}
+		includingBlocksOffset += includingBlocksIndexKeyEntrySize
+	}
+
+	for _, blockTxsAcceptanceData := range multiBlockTxsAcceptanceData {
+		var includingBlockID uint64
+		if blockTxsAcceptanceData.BlockHash.IsEqual(block.Hash()) {
+			includingBlockID = blockID
+		} else {
+			includingBlockID, err = blockdag.DBFetchBlockIDByHash(dbTx, &blockTxsAcceptanceData.BlockHash)
+			if err != nil {
+				return err
+			}
+		}
+
+		serializedIncludingBlockID := blockdag.SerializeBlockID(includingBlockID)
+
+		for _, txAcceptanceData := range blockTxsAcceptanceData.TxAcceptanceData {
+			if !txAcceptanceData.IsAccepted {
+				continue
+			}
+			err = dbPutAcceptingBlocksEntry(dbTx, txAcceptanceData.Tx.ID(), blockID, serializedIncludingBlockID)
+			if err != nil {
+				return err
+			}
+		}
+	}
+
+	return nil
+}
+
+func updateTxsAcceptedByVirtual(virtualTxsAcceptanceData blockdag.MultiBlockTxsAcceptanceData) error {
+	// Initialize a new txsAcceptedByVirtual
+	entries := 0
+	for _, blockTxsAcceptanceData := range virtualTxsAcceptanceData {
+		entries += len(blockTxsAcceptanceData.TxAcceptanceData)
+	}
+	txsAcceptedByVirtual = make(map[daghash.TxID]bool, entries)
+
+	// Copy virtualTxsAcceptanceData to txsAcceptedByVirtual
+	for _, blockTxsAcceptanceData := range virtualTxsAcceptanceData {
+		for _, txAcceptanceData := range blockTxsAcceptanceData.TxAcceptanceData {
+			txsAcceptedByVirtual[*txAcceptanceData.Tx.ID()] = true
+		}
+	}
+
+	return nil
+}
+
+// TxIndex implements a transaction by hash index. That is to say, it supports
+// querying all transactions by their hash.
+type TxIndex struct {
+	db database.DB
+}
+
+// Ensure the TxIndex type implements the Indexer interface.
+var _ Indexer = (*TxIndex)(nil)
+
+// Init initializes the hash-based transaction index. In particular, it finds
+// the highest used block ID and stores it for later use when connecting or
+// disconnecting blocks.
+//
+// This is part of the Indexer interface.
+func (idx *TxIndex) Init(db database.DB, dag *blockdag.BlockDAG) error {
+	idx.db = db
+
+	// Initialize the txsAcceptedByVirtual index
+	virtualTxsAcceptanceData, err := dag.TxsAcceptedByVirtual()
+	if err != nil {
+		return err
+	}
+	err = updateTxsAcceptedByVirtual(virtualTxsAcceptanceData)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+// Key returns the database key to use for the index as a byte slice.
+//
+// This is part of the Indexer interface.
+func (idx *TxIndex) Key() []byte {
+	return includingBlocksIndexKey
+}
+
+// Name returns the human-readable name of the index.
+//
+// This is part of the Indexer interface.
+func (idx *TxIndex) Name() string {
+	return txIndexName
+}
+
+// Create is invoked when the indexer manager determines the index needs
+// to be created for the first time. It creates the buckets for the hash-based
+// transaction index and the internal block ID indexes.
+//
+// This is part of the Indexer interface.
+func (idx *TxIndex) Create(dbTx database.Tx) error {
+	meta := dbTx.Metadata()
+	if _, err := meta.CreateBucket(includingBlocksIndexKey); err != nil {
+		return err
+	}
+	_, err := meta.CreateBucket(acceptingBlocksIndexKey)
+	return err
+
+}
+
+// ConnectBlock is invoked by the index manager when a new block has been
+// connected to the DAG. This indexer adds a hash-to-transaction mapping
+// for every transaction in the passed block.
+//
+// This is part of the Indexer interface.
+func (idx *TxIndex) ConnectBlock(dbTx database.Tx, block *util.Block, blockID uint64, dag *blockdag.BlockDAG,
+	acceptedTxsData blockdag.MultiBlockTxsAcceptanceData, virtualTxsAcceptanceData blockdag.MultiBlockTxsAcceptanceData) error {
+	if err := dbAddTxIndexEntries(dbTx, block, blockID, acceptedTxsData); err != nil {
+		return err
+	}
+
+	err := updateTxsAcceptedByVirtual(virtualTxsAcceptanceData)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+// TxFirstBlockRegion returns the first block region for the provided transaction hash
+// from the transaction index. The block region can in turn be used to load the
+// raw transaction bytes. When there is no entry for the provided hash, nil
+// will be returned for the both the entry and the error.
+//
+// This function is safe for concurrent access.
+func (idx *TxIndex) TxFirstBlockRegion(txID *daghash.TxID) (*database.BlockRegion, error) {
+	var region *database.BlockRegion
+	err := idx.db.View(func(dbTx database.Tx) error {
+		var err error
+		region, err = dbFetchFirstTxRegion(dbTx, txID)
+		return err
+	})
+	return region, err
+}
+
+// TxBlocks returns the hashes of the blocks where the transaction exists
+func (idx *TxIndex) TxBlocks(txHash *daghash.Hash) ([]*daghash.Hash, error) {
+	blockHashes := make([]*daghash.Hash, 0)
+	err := idx.db.View(func(dbTx database.Tx) error {
+		var err error
+		blockHashes, err = dbFetchTxBlocks(dbTx, txHash)
+		if err != nil {
+			return err
+		}
+		return nil
+	})
+	return blockHashes, err
+}
+
+func dbFetchTxBlocks(dbTx database.Tx, txHash *daghash.Hash) ([]*daghash.Hash, error) {
+	blockHashes := make([]*daghash.Hash, 0)
+	bucket := dbTx.Metadata().Bucket(includingBlocksIndexKey).Bucket(txHash[:])
+	if bucket == nil {
+		return nil, database.Error{
+			ErrorCode: database.ErrCorruption,
+			Description: fmt.Sprintf("No including blocks "+
+				"were found for %s", txHash),
+		}
+	}
+	err := bucket.ForEach(func(serializedBlockID, _ []byte) error {
+		blockHash, err := blockdag.DBFetchBlockHashBySerializedID(dbTx, serializedBlockID)
+		if err != nil {
+			return err
+		}
+		blockHashes = append(blockHashes, blockHash)
+		return nil
+	})
+	if err != nil {
+		return nil, err
+	}
+	return blockHashes, nil
+}
+
+// BlockThatAcceptedTx returns the hash of the block where the transaction got accepted (from the virtual block point of view)
+func (idx *TxIndex) BlockThatAcceptedTx(dag *blockdag.BlockDAG, txID *daghash.TxID) (*daghash.Hash, error) {
+	var acceptingBlock *daghash.Hash
+	err := idx.db.View(func(dbTx database.Tx) error {
+		var err error
+		acceptingBlock, err = dbFetchTxAcceptingBlock(dbTx, txID, dag)
+		return err
+	})
+	return acceptingBlock, err
+}
+
+func dbFetchTxAcceptingBlock(dbTx database.Tx, txID *daghash.TxID, dag *blockdag.BlockDAG) (*daghash.Hash, error) {
+	// If the transaction was accepted by the current virtual,
+	// return the zeroHash immediately
+	if _, ok := txsAcceptedByVirtual[*txID]; ok {
+		return &daghash.ZeroHash, nil
+	}
+
+	bucket := dbTx.Metadata().Bucket(acceptingBlocksIndexKey).Bucket(txID[:])
+	if bucket == nil {
+		return nil, nil
+	}
+	cursor := bucket.Cursor()
+	if !cursor.First() {
+		return nil, database.Error{
+			ErrorCode: database.ErrCorruption,
+			Description: fmt.Sprintf("Accepting blocks bucket is "+
+				"empty for %s", txID),
+		}
+	}
+	for ; cursor.Key() != nil; cursor.Next() {
+		blockHash, err := blockdag.DBFetchBlockHashBySerializedID(dbTx, cursor.Key())
+		if err != nil {
+			return nil, err
+		}
+		isBlockInSelectedParentChain, err := dag.IsInSelectedParentChain(blockHash)
+		if err != nil {
+			return nil, err
+		}
+		if isBlockInSelectedParentChain {
+			return blockHash, nil
+		}
+	}
+	return nil, nil
+}
+
+// NewTxIndex returns a new instance of an indexer that is used to create a
+// mapping of the hashes of all transactions in the blockDAG to the respective
+// block, location within the block, and size of the transaction.
+//
+// It implements the Indexer interface which plugs into the IndexManager that in
+// turn is used by the blockdag package. This allows the index to be
+// seamlessly maintained along with the DAG.
+func NewTxIndex() *TxIndex {
+	return &TxIndex{}
+}
+
+// DropTxIndex drops the transaction index from the provided database if it
+// exists. Since the address index relies on it, the address index will also be
+// dropped when it exists.
+func DropTxIndex(db database.DB, interrupt <-chan struct{}) error {
+	err := dropIndex(db, addrIndexKey, addrIndexName, interrupt)
+	if err != nil {
+		return err
+	}
+
+	err = dropIndex(db, includingBlocksIndexKey, addrIndexName, interrupt)
+	if err != nil {
+		return err
+	}
+
+	return dropIndex(db, acceptingBlocksIndexKey, txIndexName, interrupt)
+}
+
+// Recover is invoked when the indexer wasn't turned on for several blocks
+// and the indexer needs to close the gaps.
+//
+// This is part of the Indexer interface.
+func (idx *TxIndex) Recover(dbTx database.Tx, currentBlockID, lastKnownBlockID uint64) error {
+	return errors.Errorf("txindex was turned off for %d blocks and can't be recovered."+
+		" To resume working drop the txindex with --droptxindex", lastKnownBlockID-currentBlockID)
+}

blockdag/indexers/txindex_test.go

@@ -0,0 +1,144 @@
+package indexers
+
+import (
+	"bytes"
+	"reflect"
+	"testing"
+
+	"github.com/kaspanet/kaspad/blockdag"
+	"github.com/kaspanet/kaspad/dagconfig"
+	"github.com/kaspanet/kaspad/mining"
+	"github.com/kaspanet/kaspad/txscript"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/wire"
+)
+
+func createTransaction(t *testing.T, value uint64, originTx *wire.MsgTx, outputIndex uint32) *wire.MsgTx {
+	signatureScript, err := txscript.PayToScriptHashSignatureScript(blockdag.OpTrueScript, nil)
+	if err != nil {
+		t.Fatalf("Error creating signature script: %s", err)
+	}
+	txIn := &wire.TxIn{
+		PreviousOutpoint: wire.Outpoint{
+			TxID:  *originTx.TxID(),
+			Index: outputIndex,
+		},
+		Sequence:        wire.MaxTxInSequenceNum,
+		SignatureScript: signatureScript,
+	}
+	txOut := wire.NewTxOut(value, blockdag.OpTrueScript)
+	tx := wire.NewNativeMsgTx(wire.TxVersion, []*wire.TxIn{txIn}, []*wire.TxOut{txOut})
+
+	return tx
+}
+
+func TestTxIndexConnectBlock(t *testing.T) {
+	blocks := make(map[daghash.Hash]*util.Block)
+
+	txIndex := NewTxIndex()
+	indexManager := NewManager([]Indexer{txIndex})
+
+	params := dagconfig.SimnetParams
+	params.BlockCoinbaseMaturity = 0
+	params.K = 1
+
+	config := blockdag.Config{
+		IndexManager: indexManager,
+		DAGParams:    &params,
+	}
+
+	dag, teardown, err := blockdag.DAGSetup("TestTxIndexConnectBlock", config)
+	if err != nil {
+		t.Fatalf("TestTxIndexConnectBlock: Failed to setup DAG instance: %v", err)
+	}
+	if teardown != nil {
+		defer teardown()
+	}
+
+	prepareAndProcessBlock := func(parentHashes []*daghash.Hash, transactions []*wire.MsgTx, blockName string) *wire.MsgBlock {
+		block, err := mining.PrepareBlockForTest(dag, &params, parentHashes, transactions, false)
+		if err != nil {
+			t.Fatalf("TestTxIndexConnectBlock: block %v got unexpected error from PrepareBlockForTest: %v", blockName, err)
+		}
+		utilBlock := util.NewBlock(block)
+		blocks[*block.BlockHash()] = utilBlock
+		isOrphan, isDelayed, err := dag.ProcessBlock(utilBlock, blockdag.BFNoPoWCheck)
+		if err != nil {
+			t.Fatalf("TestTxIndexConnectBlock: dag.ProcessBlock got unexpected error for block %v: %v", blockName, err)
+		}
+		if isDelayed {
+			t.Fatalf("TestTxIndexConnectBlock: block %s "+
+				"is too far in the future", blockName)
+		}
+		if isOrphan {
+			t.Fatalf("TestTxIndexConnectBlock: block %v was unexpectedly orphan", blockName)
+		}
+		return block
+	}
+
+	block1 := prepareAndProcessBlock([]*daghash.Hash{params.GenesisHash}, nil, "1")
+	block2Tx := createTransaction(t, block1.Transactions[0].TxOut[0].Value, block1.Transactions[0], 0)
+	block2 := prepareAndProcessBlock([]*daghash.Hash{block1.BlockHash()}, []*wire.MsgTx{block2Tx}, "2")
+	block3Tx := createTransaction(t, block2.Transactions[0].TxOut[0].Value, block2.Transactions[0], 0)
+	block3 := prepareAndProcessBlock([]*daghash.Hash{block2.BlockHash()}, []*wire.MsgTx{block3Tx}, "3")
+
+	block2TxID := block2Tx.TxID()
+	block2TxNewAcceptedBlock, err := txIndex.BlockThatAcceptedTx(dag, block2TxID)
+	if err != nil {
+		t.Errorf("TestTxIndexConnectBlock: TxAcceptedInBlock: %v", err)
+	}
+	block3Hash := block3.BlockHash()
+	if !block2TxNewAcceptedBlock.IsEqual(block3Hash) {
+		t.Errorf("TestTxIndexConnectBlock: block2Tx should've "+
+			"been accepted in block %v but instead got accepted in block %v", block3Hash, block2TxNewAcceptedBlock)
+	}
+
+	block3TxID := block3Tx.TxID()
+	block3TxNewAcceptedBlock, err := txIndex.BlockThatAcceptedTx(dag, block3TxID)
+	if err != nil {
+		t.Errorf("TestTxIndexConnectBlock: TxAcceptedInBlock: %v", err)
+	}
+	if !block3TxNewAcceptedBlock.IsEqual(&daghash.ZeroHash) {
+		t.Errorf("TestTxIndexConnectBlock: block3Tx should've "+
+			"been accepted by the virtual block but instead got accepted in block %v", block3TxNewAcceptedBlock)
+	}
+
+	block3A := prepareAndProcessBlock([]*daghash.Hash{block2.BlockHash()}, []*wire.MsgTx{block3Tx}, "3A")
+	block4 := prepareAndProcessBlock([]*daghash.Hash{block3.BlockHash()}, nil, "4")
+	prepareAndProcessBlock([]*daghash.Hash{block3A.BlockHash(), block4.BlockHash()}, nil, "5")
+
+	block2TxAcceptedBlock, err := txIndex.BlockThatAcceptedTx(dag, block2TxID)
+	if err != nil {
+		t.Errorf("TestTxIndexConnectBlock: TxAcceptedInBlock: %v", err)
+	}
+
+	if !block2TxAcceptedBlock.IsEqual(block3Hash) {
+		t.Errorf("TestTxIndexConnectBlock: block2Tx should've "+
+			"been accepted in block %v but instead got accepted in block %v", block3Hash, block2TxAcceptedBlock)
+	}
+
+	region, err := txIndex.TxFirstBlockRegion(block3TxID)
+	if err != nil {
+		t.Fatalf("TestTxIndexConnectBlock: no block region was found for block3Tx")
+	}
+	regionBlock, ok := blocks[*region.Hash]
+	if !ok {
+		t.Fatalf("TestTxIndexConnectBlock: couldn't find block with hash %v", region.Hash)
+	}
+
+	regionBlockBytes, err := regionBlock.Bytes()
+	if err != nil {
+		t.Fatalf("TestTxIndexConnectBlock: Couldn't serialize block to bytes")
+	}
+	block3TxInBlock := regionBlockBytes[region.Offset : region.Offset+region.Len]
+
+	block3TxBuf := bytes.NewBuffer(make([]byte, 0, block3Tx.SerializeSize()))
+	block3Tx.KaspaEncode(block3TxBuf, 0)
+	blockTxBytes := block3TxBuf.Bytes()
+
+	if !reflect.DeepEqual(blockTxBytes, block3TxInBlock) {
+		t.Errorf("TestTxIndexConnectBlock: the block region that was in the bucket doesn't match block3Tx")
+	}
+
+}

blockdag/log.go

@@ -0,0 +1,13 @@
+// Copyright (c) 2013-2016 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"github.com/kaspanet/kaspad/logger"
+	"github.com/kaspanet/kaspad/util/panics"
+)
+
+var log, _ = logger.Get(logger.SubsystemTags.BDAG)
+var spawn = panics.GoroutineWrapperFunc(log)

blockdag/mediantime.go

@@ -0,0 +1,206 @@
+// Copyright (c) 2013-2014 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"math"
+	"sort"
+	"sync"
+	"time"
+)
+
+const (
+	// maxAllowedOffsetSeconds is the maximum number of seconds in either
+	// direction that local clock will be adjusted. When the median time
+	// of the network is outside of this range, no offset will be applied.
+	maxAllowedOffsetSecs = 70 * 60 // 1 hour 10 minutes
+
+	// similarTimeSecs is the number of seconds in either direction from the
+	// local clock that is used to determine that it is likley wrong and
+	// hence to show a warning.
+	similarTimeSecs = 5 * 60 // 5 minutes
+)
+
+var (
+	// maxMedianTimeEntries is the maximum number of entries allowed in the
+	// median time data. This is a variable as opposed to a constant so the
+	// test code can modify it.
+	maxMedianTimeEntries = 200
+)
+
+// MedianTimeSource provides a mechanism to add several time samples which are
+// used to determine a median time which is then used as an offset to the local
+// clock.
+type MedianTimeSource interface {
+	// AdjustedTime returns the current time adjusted by the median time
+	// offset as calculated from the time samples added by AddTimeSample.
+	AdjustedTime() time.Time
+
+	// AddTimeSample adds a time sample that is used when determining the
+	// median time of the added samples.
+	AddTimeSample(id string, timeVal time.Time)
+
+	// Offset returns the number of seconds to adjust the local clock based
+	// upon the median of the time samples added by AddTimeData.
+	Offset() time.Duration
+}
+
+// int64Sorter implements sort.Interface to allow a slice of 64-bit integers to
+// be sorted.
+type int64Sorter []int64
+
+// Len returns the number of 64-bit integers in the slice. It is part of the
+// sort.Interface implementation.
+func (s int64Sorter) Len() int {
+	return len(s)
+}
+
+// Swap swaps the 64-bit integers at the passed indices. It is part of the
+// sort.Interface implementation.
+func (s int64Sorter) Swap(i, j int) {
+	s[i], s[j] = s[j], s[i]
+}
+
+// Less returns whether the 64-bit integer with index i should sort before the
+// 64-bit integer with index j. It is part of the sort.Interface
+// implementation.
+func (s int64Sorter) Less(i, j int) bool {
+	return s[i] < s[j]
+}
+
+// medianTime provides an implementation of the MedianTimeSource interface.
+type medianTime struct {
+	mtx                sync.Mutex
+	knownIDs           map[string]struct{}
+	offsets            []int64
+	offsetSecs         int64
+	invalidTimeChecked bool
+}
+
+// Ensure the medianTime type implements the MedianTimeSource interface.
+var _ MedianTimeSource = (*medianTime)(nil)
+
+// AdjustedTime returns the current time adjusted by the median time offset as
+// calculated from the time samples added by AddTimeSample.
+//
+// This function is safe for concurrent access and is part of the
+// MedianTimeSource interface implementation.
+func (m *medianTime) AdjustedTime() time.Time {
+	m.mtx.Lock()
+	defer m.mtx.Unlock()
+
+	// Limit the adjusted time to 1 second precision.
+	now := time.Unix(time.Now().Unix(), 0)
+	return now.Add(time.Duration(m.offsetSecs) * time.Second)
+}
+
+// AddTimeSample adds a time sample that is used when determining the median
+// time of the added samples.
+//
+// This function is safe for concurrent access and is part of the
+// MedianTimeSource interface implementation.
+func (m *medianTime) AddTimeSample(sourceID string, timeVal time.Time) {
+	m.mtx.Lock()
+	defer m.mtx.Unlock()
+
+	// Don't add time data from the same source.
+	if _, exists := m.knownIDs[sourceID]; exists {
+		return
+	}
+	m.knownIDs[sourceID] = struct{}{}
+
+	// Truncate the provided offset to seconds and append it to the slice
+	// of offsets while respecting the maximum number of allowed entries by
+	// replacing the oldest entry with the new entry once the maximum number
+	// of entries is reached.
+	now := time.Unix(time.Now().Unix(), 0)
+	offsetSecs := int64(timeVal.Sub(now).Seconds())
+	numOffsets := len(m.offsets)
+	if numOffsets == maxMedianTimeEntries && maxMedianTimeEntries > 0 {
+		m.offsets = m.offsets[1:]
+		numOffsets--
+	}
+	m.offsets = append(m.offsets, offsetSecs)
+	numOffsets++
+
+	// Sort the offsets so the median can be obtained as needed later.
+	sortedOffsets := make([]int64, numOffsets)
+	copy(sortedOffsets, m.offsets)
+	sort.Sort(int64Sorter(sortedOffsets))
+
+	offsetDuration := time.Duration(offsetSecs) * time.Second
+	log.Debugf("Added time sample of %s (total: %d)", offsetDuration,
+		numOffsets)
+
+	// The median offset is only updated when there are enough offsets and
+	// the number of offsets is odd so the middle value is the true median.
+	// Thus, there is nothing to do when those conditions are not met.
+	if numOffsets < 5 || numOffsets&0x01 != 1 {
+		return
+	}
+
+	// At this point the number of offsets in the list is odd, so the
+	// middle value of the sorted offsets is the median.
+	median := sortedOffsets[numOffsets/2]
+
+	// Set the new offset when the median offset is within the allowed
+	// offset range.
+	if math.Abs(float64(median)) < maxAllowedOffsetSecs {
+		m.offsetSecs = median
+	} else {
+		// The median offset of all added time data is larger than the
+		// maximum allowed offset, so don't use an offset. This
+		// effectively limits how far the local clock can be skewed.
+		m.offsetSecs = 0
+
+		if !m.invalidTimeChecked {
+			m.invalidTimeChecked = true
+
+			// Find if any time samples have a time that is close
+			// to the local time.
+			var remoteHasCloseTime bool
+			for _, offset := range sortedOffsets {
+				if math.Abs(float64(offset)) < similarTimeSecs {
+					remoteHasCloseTime = true
+					break
+				}
+			}
+
+			// Warn if none of the time samples are close.
+			if !remoteHasCloseTime {
+				log.Warnf("Please check your date and time " +
+					"are correct!  kaspad will not work " +
+					"properly with an invalid time")
+			}
+		}
+	}
+
+	medianDuration := time.Duration(m.offsetSecs) * time.Second
+	log.Debugf("New time offset: %d", medianDuration)
+}
+
+// Offset returns the number of seconds to adjust the local clock based upon the
+// median of the time samples added by AddTimeData.
+//
+// This function is safe for concurrent access and is part of the
+// MedianTimeSource interface implementation.
+func (m *medianTime) Offset() time.Duration {
+	m.mtx.Lock()
+	defer m.mtx.Unlock()
+
+	return time.Duration(m.offsetSecs) * time.Second
+}
+
+// NewMedianTime returns a new instance of concurrency-safe implementation of
+// the MedianTimeSource interface. The returned implementation contains the
+// rules necessary for proper time handling in the DAG consensus rules and
+// expects the time samples to be added from the timestamp field of the version
+// message received from remote peers that successfully connect and negotiate.
+func NewMedianTime() MedianTimeSource {
+	return &medianTime{
+		knownIDs: make(map[string]struct{}),
+		offsets:  make([]int64, 0, maxMedianTimeEntries),
+	}
+}

blockdag/mediantime_test.go

@@ -0,0 +1,102 @@
+// Copyright (c) 2013-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"strconv"
+	"testing"
+	"time"
+)
+
+// TestMedianTime tests the medianTime implementation.
+func TestMedianTime(t *testing.T) {
+	tests := []struct {
+		in         []int64
+		wantOffset int64
+		useDupID   bool
+	}{
+		// Not enough samples must result in an offset of 0.
+		{in: []int64{1}, wantOffset: 0},
+		{in: []int64{1, 2}, wantOffset: 0},
+		{in: []int64{1, 2, 3}, wantOffset: 0},
+		{in: []int64{1, 2, 3, 4}, wantOffset: 0},
+
+		// Various number of entries. The expected offset is only
+		// updated on odd number of elements.
+		{in: []int64{-13, 57, -4, -23, -12}, wantOffset: -12},
+		{in: []int64{55, -13, 61, -52, 39, 55}, wantOffset: 39},
+		{in: []int64{-62, -58, -30, -62, 51, -30, 15}, wantOffset: -30},
+		{in: []int64{29, -47, 39, 54, 42, 41, 8, -33}, wantOffset: 39},
+		{in: []int64{37, 54, 9, -21, -56, -36, 5, -11, -39}, wantOffset: -11},
+		{in: []int64{57, -28, 25, -39, 9, 63, -16, 19, -60, 25}, wantOffset: 9},
+		{in: []int64{-5, -4, -3, -2, -1}, wantOffset: -3, useDupID: true},
+
+		// The offset stops being updated once the max number of entries
+		// has been reached.
+		{in: []int64{-67, 67, -50, 24, 63, 17, 58, -14, 5, -32, -52}, wantOffset: 17},
+		{in: []int64{-67, 67, -50, 24, 63, 17, 58, -14, 5, -32, -52, 45}, wantOffset: 17},
+		{in: []int64{-67, 67, -50, 24, 63, 17, 58, -14, 5, -32, -52, 45, 4}, wantOffset: 17},
+
+		// Offsets that are too far away from the local time should
+		// be ignored.
+		{in: []int64{-4201, 4202, -4203, 4204, -4205}, wantOffset: 0},
+
+		// Exercise the condition where the median offset is greater
+		// than the max allowed adjustment, but there is at least one
+		// sample that is close enough to the current time to avoid
+		// triggering a warning about an invalid local clock.
+		{in: []int64{4201, 4202, 4203, 4204, -299}, wantOffset: 0},
+	}
+
+	// Modify the max number of allowed median time entries for these tests.
+	maxMedianTimeEntries = 10
+	defer func() { maxMedianTimeEntries = 200 }()
+
+	for i, test := range tests {
+		filter := NewMedianTime()
+		for j, offset := range test.in {
+			id := strconv.Itoa(j)
+			now := time.Unix(time.Now().Unix(), 0)
+			tOffset := now.Add(time.Duration(offset) * time.Second)
+			filter.AddTimeSample(id, tOffset)
+
+			// Ensure the duplicate IDs are ignored.
+			if test.useDupID {
+				// Modify the offsets to ensure the final median
+				// would be different if the duplicate is added.
+				tOffset = tOffset.Add(time.Duration(offset) *
+					time.Second)
+				filter.AddTimeSample(id, tOffset)
+			}
+		}
+
+		// Since it is possible that the time.Now call in AddTimeSample
+		// and the time.Now calls here in the tests will be off by one
+		// second, allow a fudge factor to compensate.
+		gotOffset := filter.Offset()
+		wantOffset := time.Duration(test.wantOffset) * time.Second
+		wantOffset2 := time.Duration(test.wantOffset-1) * time.Second
+		if gotOffset != wantOffset && gotOffset != wantOffset2 {
+			t.Errorf("Offset #%d: unexpected offset -- got %v, "+
+				"want %v or %v", i, gotOffset, wantOffset,
+				wantOffset2)
+			continue
+		}
+
+		// Since it is possible that the time.Now call in AdjustedTime
+		// and the time.Now call here in the tests will be off by one
+		// second, allow a fudge factor to compensate.
+		adjustedTime := filter.AdjustedTime()
+		now := time.Unix(time.Now().Unix(), 0)
+		wantTime := now.Add(filter.Offset())
+		wantTime2 := now.Add(filter.Offset() - time.Second)
+		if !adjustedTime.Equal(wantTime) && !adjustedTime.Equal(wantTime2) {
+			t.Errorf("AdjustedTime #%d: unexpected result -- got %v, "+
+				"want %v or %v", i, adjustedTime, wantTime,
+				wantTime2)
+			continue
+		}
+	}
+}

blockdag/merkle.go

@@ -0,0 +1,134 @@
+// Copyright (c) 2013-2016 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"math"
+
+	"github.com/kaspanet/kaspad/util"
+	"github.com/kaspanet/kaspad/util/daghash"
+)
+
+// MerkleTree holds the hashes of a merkle tree
+type MerkleTree []*daghash.Hash
+
+// Root returns the root of the merkle tree
+func (mt MerkleTree) Root() *daghash.Hash {
+	return mt[len(mt)-1]
+}
+
+// nextPowerOfTwo returns the next highest power of two from a given number if
+// it is not already a power of two. This is a helper function used during the
+// calculation of a merkle tree.
+func nextPowerOfTwo(n int) int {
+	// Return the number if it's already a power of 2.
+	if n&(n-1) == 0 {
+		return n
+	}
+
+	// Figure out and return the next power of two.
+	exponent := uint(math.Log2(float64(n))) + 1
+	return 1 << exponent // 2^exponent
+}
+
+// HashMerkleBranches takes two hashes, treated as the left and right tree
+// nodes, and returns the hash of their concatenation. This is a helper
+// function used to aid in the generation of a merkle tree.
+func HashMerkleBranches(left *daghash.Hash, right *daghash.Hash) *daghash.Hash {
+	// Concatenate the left and right nodes.
+	var hash [daghash.HashSize * 2]byte
+	copy(hash[:daghash.HashSize], left[:])
+	copy(hash[daghash.HashSize:], right[:])
+
+	newHash := daghash.DoubleHashH(hash[:])
+	return &newHash
+}
+
+// BuildHashMerkleTreeStore creates a merkle tree from a slice of transactions, based
+// on their hash. See `buildMerkleTreeStore` for more info.
+func BuildHashMerkleTreeStore(transactions []*util.Tx) MerkleTree {
+	txHashes := make([]*daghash.Hash, len(transactions))
+	for i, tx := range transactions {
+		txHashes[i] = tx.Hash()
+	}
+	return buildMerkleTreeStore(txHashes)
+}
+
+// BuildIDMerkleTreeStore creates a merkle tree from a slice of transactions, based
+// on their ID. See `buildMerkleTreeStore` for more info.
+func BuildIDMerkleTreeStore(transactions []*util.Tx) MerkleTree {
+	txIDs := make([]*daghash.Hash, len(transactions))
+	for i, tx := range transactions {
+		txIDs[i] = (*daghash.Hash)(tx.ID())
+	}
+	return buildMerkleTreeStore(txIDs)
+}
+
+// buildMerkleTreeStore creates a merkle tree from a slice of hashes,
+// stores it using a linear array, and returns a slice of the backing array. A
+// linear array was chosen as opposed to an actual tree structure since it uses
+// about half as much memory. The following describes a merkle tree and how it
+// is stored in a linear array.
+//
+// A merkle tree is a tree in which every non-leaf node is the hash of its
+// children nodes. A diagram depicting how this works for kaspa transactions
+// where h(x) is a double sha256 follows:
+//
+//	         root = h1234 = h(h12 + h34)
+//	        /                           \
+//	  h12 = h(h1 + h2)            h34 = h(h3 + h4)
+//	   /            \              /            \
+//	h1 = h(tx1)  h2 = h(tx2)    h3 = h(tx3)  h4 = h(tx4)
+//
+// The above stored as a linear array is as follows:
+//
+// 	[h1 h2 h3 h4 h12 h34 root]
+//
+// As the above shows, the merkle root is always the last element in the array.
+//
+// The number of inputs is not always a power of two which results in a
+// balanced tree structure as above. In that case, parent nodes with no
+// children are also zero and parent nodes with only a single left node
+// are calculated by concatenating the left node with itself before hashing.
+// Since this function uses nodes that are pointers to the hashes, empty nodes
+// will be nil.
+func buildMerkleTreeStore(hashes []*daghash.Hash) MerkleTree {
+	// Calculate how many entries are required to hold the binary merkle
+	// tree as a linear array and create an array of that size.
+	nextPoT := nextPowerOfTwo(len(hashes))
+	arraySize := nextPoT*2 - 1
+	merkles := make(MerkleTree, arraySize)
+
+	// Create the base transaction hashes and populate the array with them.
+	for i, hash := range hashes {
+		merkles[i] = hash
+	}
+
+	// Start the array offset after the last transaction and adjusted to the
+	// next power of two.
+	offset := nextPoT
+	for i := 0; i < arraySize-1; i += 2 {
+		switch {
+		// When there is no left child node, the parent is nil too.
+		case merkles[i] == nil:
+			merkles[offset] = nil
+
+		// When there is no right child, the parent is generated by
+		// hashing the concatenation of the left child with itself.
+		case merkles[i+1] == nil:
+			newHash := HashMerkleBranches(merkles[i], merkles[i])
+			merkles[offset] = newHash
+
+		// The normal case sets the parent node to the double sha256
+		// of the concatentation of the left and right children.
+		default:
+			newHash := HashMerkleBranches(merkles[i], merkles[i+1])
+			merkles[offset] = newHash
+		}
+		offset++
+	}
+
+	return merkles
+}

blockdag/merkle_test.go

@@ -0,0 +1,36 @@
+// Copyright (c) 2013-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"github.com/kaspanet/kaspad/util/daghash"
+	"testing"
+
+	"github.com/kaspanet/kaspad/util"
+)
+
+// TestMerkle tests the BuildHashMerkleTreeStore API.
+func TestMerkle(t *testing.T) {
+	block := util.NewBlock(&Block100000)
+
+	hashMerkleTree := BuildHashMerkleTreeStore(block.Transactions())
+	calculatedHashMerkleRoot := hashMerkleTree.Root()
+	wantHashMerkleRoot := Block100000.Header.HashMerkleRoot
+	if !wantHashMerkleRoot.IsEqual(calculatedHashMerkleRoot) {
+		t.Errorf("BuildHashMerkleTreeStore: hash merkle root mismatch - "+
+			"got %v, want %v", calculatedHashMerkleRoot, wantHashMerkleRoot)
+	}
+
+	idMerkleTree := BuildIDMerkleTreeStore(block.Transactions())
+	calculatedIDMerkleRoot := idMerkleTree.Root()
+	wantIDMerkleRoot, err := daghash.NewHashFromStr("3f69feb7edf5d0d67930afc990c8ec931e3428d7c7a65d7af6b81079319eb110")
+	if err != nil {
+		t.Errorf("BuildIDMerkleTreeStore: unexpected error: %s", err)
+	}
+	if !calculatedIDMerkleRoot.IsEqual(wantIDMerkleRoot) {
+		t.Errorf("BuildIDMerkleTreeStore: ID merkle root mismatch - "+
+			"got %v, want %v", calculatedIDMerkleRoot, wantIDMerkleRoot)
+	}
+}

blockdag/mining.go

@@ -0,0 +1,124 @@
+package blockdag
+
+import (
+	"bytes"
+	"encoding/binary"
+	"github.com/kaspanet/kaspad/txscript"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/kaspanet/kaspad/util/subnetworkid"
+	"github.com/kaspanet/kaspad/wire"
+	"sort"
+	"time"
+)
+
+// BlockForMining returns a block with the given transactions
+// that points to the current DAG tips, that is valid from
+// all aspects except proof of work.
+func (dag *BlockDAG) BlockForMining(transactions []*util.Tx) (*wire.MsgBlock, error) {
+	blockTimestamp := dag.NextBlockTime()
+	requiredDifficulty := dag.NextRequiredDifficulty(blockTimestamp)
+
+	// Calculate the next expected block version based on the state of the
+	// rule change deployments.
+	nextBlockVersion, err := dag.CalcNextBlockVersion()
+	if err != nil {
+		return nil, err
+	}
+
+	// Sort transactions by subnetwork ID before building Merkle tree
+	sort.Slice(transactions, func(i, j int) bool {
+		if transactions[i].MsgTx().SubnetworkID.IsEqual(subnetworkid.SubnetworkIDCoinbase) {
+			return true
+		}
+		if transactions[j].MsgTx().SubnetworkID.IsEqual(subnetworkid.SubnetworkIDCoinbase) {
+			return false
+		}
+		return subnetworkid.Less(&transactions[i].MsgTx().SubnetworkID, &transactions[j].MsgTx().SubnetworkID)
+	})
+
+	// Create a new block ready to be solved.
+	hashMerkleTree := BuildHashMerkleTreeStore(transactions)
+	acceptedIDMerkleRoot, err := dag.NextAcceptedIDMerkleRootNoLock()
+	if err != nil {
+		return nil, err
+	}
+	var msgBlock wire.MsgBlock
+	for _, tx := range transactions {
+		msgBlock.AddTransaction(tx.MsgTx())
+	}
+
+	utxoWithTransactions, err := dag.UTXOSet().WithTransactions(msgBlock.Transactions, UnacceptedBlueScore, false)
+	if err != nil {
+		return nil, err
+	}
+	utxoCommitment := utxoWithTransactions.Multiset().Hash()
+
+	msgBlock.Header = wire.BlockHeader{
+		Version:              nextBlockVersion,
+		ParentHashes:         dag.TipHashes(),
+		HashMerkleRoot:       hashMerkleTree.Root(),
+		AcceptedIDMerkleRoot: acceptedIDMerkleRoot,
+		UTXOCommitment:       utxoCommitment,
+		Timestamp:            blockTimestamp,
+		Bits:                 requiredDifficulty,
+	}
+
+	return &msgBlock, nil
+}
+
+// CoinbasePayloadExtraData returns coinbase payload extra data parameter
+// which is built from extra nonce and coinbase flags.
+func CoinbasePayloadExtraData(extraNonce uint64, coinbaseFlags string) ([]byte, error) {
+	extraNonceBytes := make([]byte, 8)
+	binary.LittleEndian.PutUint64(extraNonceBytes, extraNonce)
+	w := &bytes.Buffer{}
+	_, err := w.Write(extraNonceBytes)
+	if err != nil {
+		return nil, err
+	}
+	_, err = w.Write([]byte(coinbaseFlags))
+	if err != nil {
+		return nil, err
+	}
+	return w.Bytes(), nil
+}
+
+// NextCoinbaseFromAddress returns a coinbase transaction for the
+// next block with the given address and extra data in its payload.
+func (dag *BlockDAG) NextCoinbaseFromAddress(payToAddress util.Address, extraData []byte) (*util.Tx, error) {
+	coinbasePayloadScriptPubKey, err := txscript.PayToAddrScript(payToAddress)
+	if err != nil {
+		return nil, err
+	}
+	coinbaseTx, err := dag.NextBlockCoinbaseTransactionNoLock(coinbasePayloadScriptPubKey, extraData)
+	if err != nil {
+		return nil, err
+	}
+	return coinbaseTx, nil
+}
+
+// NextBlockMinimumTime returns the minimum allowed timestamp for a block building
+// on the end of the DAG. In particular, it is one second after
+// the median timestamp of the last several blocks per the DAG consensus
+// rules.
+func (dag *BlockDAG) NextBlockMinimumTime() time.Time {
+	return dag.CalcPastMedianTime().Add(time.Second)
+}
+
+// NextBlockTime returns a valid block time for the
+// next block that will point to the existing DAG tips.
+func (dag *BlockDAG) NextBlockTime() time.Time {
+	// The timestamp for the block must not be before the median timestamp
+	// of the last several blocks. Thus, choose the maximum between the
+	// current time and one second after the past median time. The current
+	// timestamp is truncated to a second boundary before comparison since a
+	// block timestamp does not supported a precision greater than one
+	// second.
+	newTimestamp := dag.AdjustedTime()
+	minTimestamp := dag.NextBlockMinimumTime()
+	if newTimestamp.Before(minTimestamp) {
+		newTimestamp = minTimestamp
+	}
+
+	return newTimestamp
+}

blockdag/notifications.go

@@ -0,0 +1,89 @@
+// Copyright (c) 2013-2016 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"fmt"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/kaspanet/kaspad/util/daghash"
+)
+
+// NotificationType represents the type of a notification message.
+type NotificationType int
+
+// NotificationCallback is used for a caller to provide a callback for
+// notifications about various blockDAG events.
+type NotificationCallback func(*Notification)
+
+// Constants for the type of a notification message.
+const (
+	// NTBlockAdded indicates the associated block was added into
+	// the blockDAG.
+	NTBlockAdded NotificationType = iota
+
+	// NTChainChanged indicates that selected parent
+	// chain had changed.
+	NTChainChanged
+)
+
+// notificationTypeStrings is a map of notification types back to their constant
+// names for pretty printing.
+var notificationTypeStrings = map[NotificationType]string{
+	NTBlockAdded:   "NTBlockAdded",
+	NTChainChanged: "NTChainChanged",
+}
+
+// String returns the NotificationType in human-readable form.
+func (n NotificationType) String() string {
+	if s, ok := notificationTypeStrings[n]; ok {
+		return s
+	}
+	return fmt.Sprintf("Unknown Notification Type (%d)", int(n))
+}
+
+// Notification defines notification that is sent to the caller via the callback
+// function provided during the call to New and consists of a notification type
+// as well as associated data that depends on the type as follows:
+// 	- Added:     *util.Block
+type Notification struct {
+	Type NotificationType
+	Data interface{}
+}
+
+// Subscribe to block DAG notifications. Registers a callback to be executed
+// when various events take place. See the documentation on Notification and
+// NotificationType for details on the types and contents of notifications.
+func (dag *BlockDAG) Subscribe(callback NotificationCallback) {
+	dag.notificationsLock.Lock()
+	dag.notifications = append(dag.notifications, callback)
+	dag.notificationsLock.Unlock()
+}
+
+// sendNotification sends a notification with the passed type and data if the
+// caller requested notifications by providing a callback function in the call
+// to New.
+func (dag *BlockDAG) sendNotification(typ NotificationType, data interface{}) {
+	// Generate and send the notification.
+	n := Notification{Type: typ, Data: data}
+	dag.notificationsLock.RLock()
+	for _, callback := range dag.notifications {
+		callback(&n)
+	}
+	dag.notificationsLock.RUnlock()
+}
+
+// BlockAddedNotificationData defines data to be sent along with a BlockAdded
+// notification
+type BlockAddedNotificationData struct {
+	Block         *util.Block
+	WasUnorphaned bool
+}
+
+// ChainChangedNotificationData defines data to be sent along with a ChainChanged
+// notification
+type ChainChangedNotificationData struct {
+	RemovedChainBlockHashes []*daghash.Hash
+	AddedChainBlockHashes   []*daghash.Hash
+}

blockdag/notifications_test.go

@@ -0,0 +1,61 @@
+// Copyright (c) 2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"path/filepath"
+	"testing"
+
+	"github.com/kaspanet/kaspad/dagconfig"
+)
+
+// TestNotifications ensures that notification callbacks are fired on events.
+func TestNotifications(t *testing.T) {
+	blocks, err := LoadBlocks(filepath.Join("testdata/blk_0_to_4.dat"))
+	if err != nil {
+		t.Fatalf("Error loading file: %v\n", err)
+	}
+
+	// Create a new database and dag instance to run tests against.
+	dag, teardownFunc, err := DAGSetup("notifications", Config{
+		DAGParams: &dagconfig.SimnetParams,
+	})
+	if err != nil {
+		t.Fatalf("Failed to setup dag instance: %v", err)
+	}
+	defer teardownFunc()
+
+	notificationCount := 0
+	callback := func(notification *Notification) {
+		if notification.Type == NTBlockAdded {
+			notificationCount++
+		}
+	}
+
+	// Register callback multiple times then assert it is called that many
+	// times.
+	const numSubscribers = 3
+	for i := 0; i < numSubscribers; i++ {
+		dag.Subscribe(callback)
+	}
+
+	isOrphan, isDelayed, err := dag.ProcessBlock(blocks[1], BFNone)
+	if err != nil {
+		t.Fatalf("ProcessBlock fail on block 1: %v\n", err)
+	}
+	if isDelayed {
+		t.Fatalf("ProcessBlock: block 1 " +
+			"is too far in the future")
+	}
+	if isOrphan {
+		t.Fatalf("ProcessBlock incorrectly returned block " +
+			"is an orphan\n")
+	}
+
+	if notificationCount != numSubscribers {
+		t.Fatalf("Expected notification callback to be executed %d "+
+			"times, found %d", numSubscribers, notificationCount)
+	}
+}

blockdag/process.go

@@ -0,0 +1,263 @@
+// Copyright (c) 2013-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"fmt"
+	"github.com/kaspanet/kaspad/dagconfig"
+	"time"
+
+	"github.com/kaspanet/kaspad/util"
+	"github.com/kaspanet/kaspad/util/daghash"
+)
+
+// BehaviorFlags is a bitmask defining tweaks to the normal behavior when
+// performing DAG processing and consensus rules checks.
+type BehaviorFlags uint32
+
+const (
+	// BFFastAdd may be set to indicate that several checks can be avoided
+	// for the block since it is already known to fit into the DAG due to
+	// already proving it correct links into the DAG.
+	BFFastAdd BehaviorFlags = 1 << iota
+
+	// BFNoPoWCheck may be set to indicate the proof of work check which
+	// ensures a block hashes to a value less than the required target will
+	// not be performed.
+	BFNoPoWCheck
+
+	// BFWasUnorphaned may be set to indicate that a block was just now
+	// unorphaned
+	BFWasUnorphaned
+
+	// BFAfterDelay may be set to indicate that a block had timestamp too far
+	// in the future, just finished the delay
+	BFAfterDelay
+
+	// BFIsSync may be set to indicate that the block was sent as part of the
+	// netsync process
+	BFIsSync
+
+	// BFWasStored is set to indicate that the block was previously stored
+	// in the block index but was never fully processed
+	BFWasStored
+
+	// BFNone is a convenience value to specifically indicate no flags.
+	BFNone BehaviorFlags = 0
+)
+
+// BlockExists determines whether a block with the given hash exists in
+// the DAG.
+//
+// This function is safe for concurrent access.
+func (dag *BlockDAG) BlockExists(hash *daghash.Hash) bool {
+	return dag.index.HaveBlock(hash)
+}
+
+// processOrphans determines if there are any orphans which depend on the passed
+// block hash (they are no longer orphans if true) and potentially accepts them.
+// It repeats the process for the newly accepted blocks (to detect further
+// orphans which may no longer be orphans) until there are no more.
+//
+// The flags do not modify the behavior of this function directly, however they
+// are needed to pass along to maybeAcceptBlock.
+//
+// This function MUST be called with the DAG state lock held (for writes).
+func (dag *BlockDAG) processOrphans(hash *daghash.Hash, flags BehaviorFlags) error {
+	// Start with processing at least the passed hash. Leave a little room
+	// for additional orphan blocks that need to be processed without
+	// needing to grow the array in the common case.
+	processHashes := make([]*daghash.Hash, 0, 10)
+	processHashes = append(processHashes, hash)
+	for len(processHashes) > 0 {
+		// Pop the first hash to process from the slice.
+		processHash := processHashes[0]
+		processHashes[0] = nil // Prevent GC leak.
+		processHashes = processHashes[1:]
+
+		// Look up all orphans that are parented by the block we just
+		// accepted.  An indexing for loop is
+		// intentionally used over a range here as range does not
+		// reevaluate the slice on each iteration nor does it adjust the
+		// index for the modified slice.
+		for i := 0; i < len(dag.prevOrphans[*processHash]); i++ {
+			orphan := dag.prevOrphans[*processHash][i]
+			if orphan == nil {
+				log.Warnf("Found a nil entry at index %d in the "+
+					"orphan dependency list for block %s", i,
+					processHash)
+				continue
+			}
+
+			// Skip this orphan if one or more of its parents are
+			// still missing.
+			_, err := lookupParentNodes(orphan.block, dag)
+			if err != nil {
+				if ruleErr, ok := err.(RuleError); ok && ruleErr.ErrorCode == ErrParentBlockUnknown {
+					continue
+				}
+				return err
+			}
+
+			// Remove the orphan from the orphan pool.
+			orphanHash := orphan.block.Hash()
+			dag.removeOrphanBlock(orphan)
+			i--
+
+			// Potentially accept the block into the block DAG.
+			err = dag.maybeAcceptBlock(orphan.block, flags|BFWasUnorphaned)
+			if err != nil {
+				// Since we don't want to reject the original block because of
+				// a bad unorphaned child, only return an error if it's not a RuleError.
+				if _, ok := err.(RuleError); !ok {
+					return err
+				}
+				log.Warnf("Verification failed for orphan block %s: %s", orphanHash, err)
+			}
+
+			// Add this block to the list of blocks to process so
+			// any orphan blocks that depend on this block are
+			// handled too.
+			processHashes = append(processHashes, orphanHash)
+		}
+	}
+	return nil
+}
+
+// ProcessBlock is the main workhorse for handling insertion of new blocks into
+// the block DAG. It includes functionality such as rejecting duplicate
+// blocks, ensuring blocks follow all rules, orphan handling, and insertion into
+// the block DAG.
+//
+// When no errors occurred during processing, the first return value indicates
+// whether or not the block is an orphan.
+//
+// This function is safe for concurrent access.
+func (dag *BlockDAG) ProcessBlock(block *util.Block, flags BehaviorFlags) (isOrphan bool, isDelayed bool, err error) {
+	dag.dagLock.Lock()
+	defer dag.dagLock.Unlock()
+	return dag.processBlockNoLock(block, flags)
+}
+
+func (dag *BlockDAG) processBlockNoLock(block *util.Block, flags BehaviorFlags) (isOrphan bool, isDelayed bool, err error) {
+	isAfterDelay := flags&BFAfterDelay == BFAfterDelay
+	wasBlockStored := flags&BFWasStored == BFWasStored
+
+	blockHash := block.Hash()
+	log.Tracef("Processing block %s", blockHash)
+
+	// The block must not already exist in the DAG.
+	if dag.BlockExists(blockHash) && !wasBlockStored {
+		str := fmt.Sprintf("already have block %s", blockHash)
+		return false, false, ruleError(ErrDuplicateBlock, str)
+	}
+
+	// The block must not already exist as an orphan.
+	if _, exists := dag.orphans[*blockHash]; exists {
+		str := fmt.Sprintf("already have block (orphan) %s", blockHash)
+		return false, false, ruleError(ErrDuplicateBlock, str)
+	}
+
+	if dag.isKnownDelayedBlock(blockHash) {
+		str := fmt.Sprintf("already have block (delayed) %s", blockHash)
+		return false, false, ruleError(ErrDuplicateBlock, str)
+	}
+
+	if !isAfterDelay {
+		// Perform preliminary sanity checks on the block and its transactions.
+		delay, err := dag.checkBlockSanity(block, flags)
+		if err != nil {
+			return false, false, err
+		}
+
+		if delay != 0 {
+			err = dag.addDelayedBlock(block, delay)
+			if err != nil {
+				return false, false, err
+			}
+			return false, true, nil
+		}
+	}
+
+	var missingParents []*daghash.Hash
+	for _, parentHash := range block.MsgBlock().Header.ParentHashes {
+		if !dag.BlockExists(parentHash) {
+			missingParents = append(missingParents, parentHash)
+		}
+	}
+
+	// Handle the case of a block with a valid timestamp(non-delayed) which points to a delayed block.
+	delay, isParentDelayed := dag.maxDelayOfParents(missingParents)
+	if isParentDelayed {
+		// Add Nanosecond to ensure that parent process time will be after its child.
+		delay += time.Nanosecond
+		err := dag.addDelayedBlock(block, delay)
+		if err != nil {
+			return false, false, err
+		}
+		return false, true, err
+	}
+
+	// Handle orphan blocks.
+	if len(missingParents) > 0 {
+		// Some orphans during netsync are a normal part of the process, since the anticone
+		// of the chain-split is never explicitly requested.
+		// Therefore, if we are during netsync - don't report orphans to default logs.
+		//
+		// The number K*2 was chosen since in peace times anticone is limited to K blocks,
+		// while some red block can make it a bit bigger, but much more than that indicates
+		// there might be some problem with the netsync process.
+		if flags&BFIsSync == BFIsSync && dagconfig.KType(len(dag.orphans)) < dag.dagParams.K*2 {
+			log.Debugf("Adding orphan block %s. This is normal part of netsync process", blockHash)
+		} else {
+			log.Infof("Adding orphan block %s", blockHash)
+		}
+		dag.addOrphanBlock(block)
+
+		return true, false, nil
+	}
+
+	// The block has passed all context independent checks and appears sane
+	// enough to potentially accept it into the block DAG.
+	err = dag.maybeAcceptBlock(block, flags)
+	if err != nil {
+		return false, false, err
+	}
+
+	// Accept any orphan blocks that depend on this block (they are
+	// no longer orphans) and repeat for those accepted blocks until
+	// there are no more.
+	err = dag.processOrphans(blockHash, flags)
+	if err != nil {
+		return false, false, err
+	}
+
+	if !isAfterDelay {
+		err = dag.processDelayedBlocks()
+		if err != nil {
+			return false, false, err
+		}
+	}
+
+	log.Debugf("Accepted block %s", blockHash)
+
+	return false, false, nil
+}
+
+// maxDelayOfParents returns the maximum delay of the given block hashes.
+// Note that delay could be 0, but isDelayed will return true. This is the case where the parent process time is due.
+func (dag *BlockDAG) maxDelayOfParents(parentHashes []*daghash.Hash) (delay time.Duration, isDelayed bool) {
+	for _, parentHash := range parentHashes {
+		if delayedParent, exists := dag.delayedBlocks[*parentHash]; exists {
+			isDelayed = true
+			parentDelay := delayedParent.processTime.Sub(dag.AdjustedTime())
+			if parentDelay > delay {
+				delay = parentDelay
+			}
+		}
+	}
+
+	return delay, isDelayed
+}

blockdag/process_test.go

@@ -0,0 +1,71 @@
+package blockdag
+
+import (
+	"path/filepath"
+	"testing"
+
+	"github.com/kaspanet/kaspad/dagconfig"
+	"github.com/kaspanet/kaspad/util/daghash"
+)
+
+func TestProcessOrphans(t *testing.T) {
+	dag, teardownFunc, err := DAGSetup("TestProcessOrphans", Config{
+		DAGParams: &dagconfig.SimnetParams,
+	})
+	if err != nil {
+		t.Errorf("Failed to setup dag instance: %v", err)
+		return
+	}
+	defer teardownFunc()
+
+	dag.TestSetCoinbaseMaturity(0)
+
+	blocksFile := "blk_0_to_4.dat"
+	blocks, err := LoadBlocks(filepath.Join("testdata/", blocksFile))
+	if err != nil {
+		t.Fatalf("TestProcessOrphans: "+
+			"Error loading file '%s': %s\n", blocksFile, err)
+	}
+
+	// Get a reference to a parent block
+	parentBlock := blocks[1]
+
+	// Get a reference to a child block and mess with it so that:
+	// a. It gets added to the orphan pool
+	// b. It gets rejected once it's unorphaned
+	childBlock := blocks[2]
+	childBlock.MsgBlock().Header.UTXOCommitment = &daghash.ZeroHash
+
+	// Process the child block so that it gets added to the orphan pool
+	isOrphan, isDelayed, err := dag.ProcessBlock(childBlock, BFNoPoWCheck)
+	if err != nil {
+		t.Fatalf("TestProcessOrphans: child block unexpectedly returned an error: %s", err)
+	}
+	if isDelayed {
+		t.Fatalf("TestProcessOrphans: child block is too far in the future")
+	}
+	if !isOrphan {
+		t.Fatalf("TestProcessOrphans: incorrectly returned that child block is not an orphan")
+	}
+
+	// Process the parent block. Note that this will attempt to unorphan the child block
+	isOrphan, isDelayed, err = dag.ProcessBlock(parentBlock, BFNone)
+	if err != nil {
+		t.Fatalf("TestProcessOrphans: parent block unexpectedly returned an error: %s", err)
+	}
+	if isDelayed {
+		t.Fatalf("TestProcessOrphans: parent block is too far in the future")
+	}
+	if isOrphan {
+		t.Fatalf("TestProcessOrphans: incorrectly returned that parent block is an orphan")
+	}
+
+	// Make sure that the child block had been rejected
+	node := dag.index.LookupNode(childBlock.Hash())
+	if node == nil {
+		t.Fatalf("TestProcessOrphans: child block missing from block index")
+	}
+	if !dag.index.NodeStatus(node).KnownInvalid() {
+		t.Fatalf("TestProcessOrphans: child block erroneously not marked as invalid")
+	}
+}

blockdag/reachability.go

@@ -0,0 +1,589 @@
+package blockdag
+
+import (
+	"fmt"
+	"github.com/pkg/errors"
+	"math"
+	"strings"
+)
+
+// reachabilityInterval represents an interval to be used within the
+// tree reachability algorithm. See reachabilityTreeNode for further
+// details.
+type reachabilityInterval struct {
+	start uint64
+	end   uint64
+}
+
+func newReachabilityInterval(start uint64, end uint64) *reachabilityInterval {
+	return &reachabilityInterval{start: start, end: end}
+}
+
+// size returns the size of this interval. Note that intervals are
+// inclusive from both sides.
+func (ri *reachabilityInterval) size() uint64 {
+	return ri.end - ri.start + 1
+}
+
+// splitInHalf splits this interval by a fraction of 0.5.
+// See splitFraction for further details.
+func (ri *reachabilityInterval) splitInHalf() (
+	left *reachabilityInterval, right *reachabilityInterval, err error) {
+
+	return ri.splitFraction(0.5)
+}
+
+// splitFraction splits this interval to two parts such that their
+// union is equal to the original interval and the first (left) part
+// contains the given fraction of the original interval's size.
+// Note: if the split results in fractional parts, this method rounds
+// the first part up and the last part down.
+func (ri *reachabilityInterval) splitFraction(fraction float64) (
+	left *reachabilityInterval, right *reachabilityInterval, err error) {
+
+	if fraction < 0 || fraction > 1 {
+		return nil, nil, errors.Errorf("fraction must be between 0 and 1")
+	}
+	if ri.size() == 0 {
+		return nil, nil, errors.Errorf("cannot split an empty interval")
+	}
+
+	allocationSize := uint64(math.Ceil(float64(ri.size()) * fraction))
+	left = newReachabilityInterval(ri.start, ri.start+allocationSize-1)
+	right = newReachabilityInterval(ri.start+allocationSize, ri.end)
+	return left, right, nil
+}
+
+// splitExact splits this interval to exactly |sizes| parts where
+// |part_i| = sizes[i]. This method expects sum(sizes) to be exactly
+// equal to the interval's size.
+func (ri *reachabilityInterval) splitExact(sizes []uint64) ([]*reachabilityInterval, error) {
+	sizesSum := uint64(0)
+	for _, size := range sizes {
+		sizesSum += size
+	}
+	if sizesSum != ri.size() {
+		return nil, errors.Errorf("sum of sizes must be equal to the interval's size")
+	}
+
+	intervals := make([]*reachabilityInterval, len(sizes))
+	start := ri.start
+	for i, size := range sizes {
+		intervals[i] = newReachabilityInterval(start, start+size-1)
+		start += size
+	}
+	return intervals, nil
+}
+
+// splitWithExponentialBias splits this interval to |sizes| parts
+// by the allocation rule described below. This method expects sum(sizes)
+// to be smaller or equal to the interval's size. Every part_i is
+// allocated at least sizes[i] capacity. The remaining budget is
+// split by an exponentially biased rule described below.
+//
+// This rule follows the GHOSTDAG protocol behavior where the child
+// with the largest subtree is expected to dominate the competition
+// for new blocks and thus grow the most. However, we may need to
+// add slack for non-largest subtrees in order to make CPU reindexing
+// attacks unworthy.
+func (ri *reachabilityInterval) splitWithExponentialBias(sizes []uint64) ([]*reachabilityInterval, error) {
+	intervalSize := ri.size()
+	sizesSum := uint64(0)
+	for _, size := range sizes {
+		sizesSum += size
+	}
+	if sizesSum > intervalSize {
+		return nil, errors.Errorf("sum of sizes must be less than or equal to the interval's size")
+	}
+	if sizesSum == intervalSize {
+		return ri.splitExact(sizes)
+	}
+
+	// Add a fractional bias to every size in the given sizes
+	totalBias := intervalSize - sizesSum
+	remainingBias := totalBias
+	biasedSizes := make([]uint64, len(sizes))
+	fractions := exponentialFractions(sizes)
+	for i, fraction := range fractions {
+		var bias uint64
+		if i == len(fractions)-1 {
+			bias = remainingBias
+		} else {
+			bias = uint64(math.Round(float64(totalBias) * fraction))
+			if bias > remainingBias {
+				bias = remainingBias
+			}
+		}
+		biasedSizes[i] = sizes[i] + bias
+		remainingBias -= bias
+	}
+	return ri.splitExact(biasedSizes)
+}
+
+// exponentialFractions returns a fraction of each size in sizes
+// as follows:
+//   fraction[i] = 2^size[i] / sum_j(2^size[j])
+// In the code below the above equation is divided by 2^max(size)
+// to avoid exploding numbers. Note that in 1 / 2^(max(size)-size[i])
+// we divide 1 by potentially a very large number, which will
+// result in loss of float precision. This is not a problem - all
+// numbers close to 0 bear effectively the same weight.
+func exponentialFractions(sizes []uint64) []float64 {
+	maxSize := uint64(0)
+	for _, size := range sizes {
+		if size > maxSize {
+			maxSize = size
+		}
+	}
+	fractions := make([]float64, len(sizes))
+	for i, size := range sizes {
+		fractions[i] = 1 / math.Pow(2, float64(maxSize-size))
+	}
+	fractionsSum := float64(0)
+	for _, fraction := range fractions {
+		fractionsSum += fraction
+	}
+	for i, fraction := range fractions {
+		fractions[i] = fraction / fractionsSum
+	}
+	return fractions
+}
+
+// isAncestorOf checks if this interval's node is a reachability tree
+// ancestor of the other interval's node. The condition below is relying on the
+// property of reachability intervals that intervals are either completely disjoint,
+// or one strictly contains the other.
+func (ri *reachabilityInterval) isAncestorOf(other *reachabilityInterval) bool {
+	return ri.start <= other.end && other.end <= ri.end
+}
+
+// String returns a string representation of the interval.
+func (ri *reachabilityInterval) String() string {
+	return fmt.Sprintf("[%d,%d]", ri.start, ri.end)
+}
+
+// reachabilityTreeNode represents a node in the reachability tree
+// of some DAG block. It mainly provides the ability to query *tree*
+// reachability with O(1) query time. It does so by managing an
+// index interval for each node and making sure all nodes in its
+// subtree are indexed within the interval, so the query
+// B ∈ subtree(A) simply becomes B.interval ⊂ A.interval.
+//
+// The main challenge of maintaining such intervals is that our tree
+// is an ever-growing tree and as such pre-allocated intervals may
+// not suffice as per future events. This is where the reindexing
+// algorithm below comes into place.
+// We use the reasonable assumption that the initial root interval
+// (e.g., [0, 2^64-1]) should always suffice for any practical use-
+// case, and so reindexing should always succeed unless more than
+// 2^64 blocks are added to the DAG/tree.
+type reachabilityTreeNode struct {
+	blockNode *blockNode
+
+	children []*reachabilityTreeNode
+	parent   *reachabilityTreeNode
+
+	// interval is the index interval containing all intervals of
+	// blocks in this node's subtree
+	interval *reachabilityInterval
+
+	// remainingInterval is the not-yet allocated interval (within
+	// this node's interval) awaiting new children
+	remainingInterval *reachabilityInterval
+
+	// subtreeSize is a helper field used only during reindexing
+	// (expected to be 0 any other time).
+	// See countSubtrees for further details.
+	subtreeSize uint64
+}
+
+func newReachabilityTreeNode(blockNode *blockNode) *reachabilityTreeNode {
+	// Please see the comment above reachabilityTreeNode to understand why
+	// we use these initial values.
+	interval := newReachabilityInterval(1, math.MaxUint64-1)
+	// We subtract 1 from the end of the remaining interval to prevent the node from allocating
+	// the entire interval to its child, so its interval would *strictly* contain the interval of its child.
+	remainingInterval := newReachabilityInterval(interval.start, interval.end-1)
+	return &reachabilityTreeNode{blockNode: blockNode, interval: interval, remainingInterval: remainingInterval}
+}
+
+// addChild adds child to this tree node. If this node has no
+// remaining interval to allocate, a reindexing is triggered.
+// This method returns a list of reachabilityTreeNodes modified
+// by it.
+func (rtn *reachabilityTreeNode) addChild(child *reachabilityTreeNode) ([]*reachabilityTreeNode, error) {
+	// Set the parent-child relationship
+	rtn.children = append(rtn.children, child)
+	child.parent = rtn
+
+	// No allocation space left -- reindex
+	if rtn.remainingInterval.size() == 0 {
+		return rtn.reindexIntervals()
+	}
+
+	// Allocate from the remaining space
+	allocated, remaining, err := rtn.remainingInterval.splitInHalf()
+	if err != nil {
+		return nil, err
+	}
+	child.setInterval(allocated)
+	rtn.remainingInterval = remaining
+	return []*reachabilityTreeNode{rtn, child}, nil
+}
+
+// setInterval sets the reachability interval for this node.
+func (rtn *reachabilityTreeNode) setInterval(interval *reachabilityInterval) {
+	rtn.interval = interval
+
+	// Reserve a single interval index for the current node. This
+	// is necessary to ensure that ancestor intervals are strictly
+	// supersets of any descendant intervals and not equal
+	rtn.remainingInterval = newReachabilityInterval(interval.start, interval.end-1)
+}
+
+// reindexIntervals traverses the reachability subtree that's
+// defined by this node and reallocates reachability interval space
+// such that another reindexing is unlikely to occur shortly
+// thereafter. It does this by traversing down the reachability
+// tree until it finds a node with a subreeSize that's greater than
+// its interval size. See propagateInterval for further details.
+// This method returns a list of reachabilityTreeNodes modified by it.
+func (rtn *reachabilityTreeNode) reindexIntervals() ([]*reachabilityTreeNode, error) {
+	current := rtn
+
+	// Initial interval and subtree sizes
+	intervalSize := current.interval.size()
+	subtreeSize := current.countSubtrees()
+
+	// Find the first ancestor that has sufficient interval space
+	for intervalSize < subtreeSize {
+		if current.parent == nil {
+			// If we ended up here it means that there are more
+			// than 2^64 blocks, which shouldn't ever happen.
+			return nil, errors.Errorf("missing tree " +
+				"parent during reindexing. Theoretically, this " +
+				"should only ever happen if there are more " +
+				"than 2^64 blocks in the DAG.")
+		}
+		current = current.parent
+		intervalSize = current.interval.size()
+		subtreeSize = current.countSubtrees()
+	}
+
+	// Propagate the interval to the subtree
+	return current.propagateInterval()
+}
+
+// countSubtrees counts the size of each subtree under this node.
+// It is equivalent to the following recursive implementation:
+//
+// func (rtn *reachabilityTreeNode) countSubtrees() uint64 {
+//     subtreeSize := uint64(0)
+//     for _, child := range rtn.children {
+//         subtreeSize += child.countSubtrees()
+//     }
+//     return subtreeSize + 1
+// }
+//
+// However, we are expecting (linearly) deep trees, and so a
+// recursive stack-based approach is inefficient and will hit
+// recursion limits. Instead, the same logic was implemented
+// using a (queue-based) BFS method. At a high level, the
+// algorithm uses BFS for reaching all leaves and pushes
+// intermediate updates from leaves via parent chains until all
+// size information is gathered at the root of the operation
+// (i.e. at rtn).
+//
+// Note the role of the subtreeSize field in the algorithm.
+// For each node rtn this field is initialized to 0. The field
+// has two possible states:
+// * rtn.subtreeSize > |rtn.children|:
+//   indicates that rtn's subtree size is already known and
+//   calculated.
+// * rtn.subtreeSize <= |rtn.children|:
+//   indicates that we are still in the counting stage of
+//   tracking which of rtn's children has already calculated
+//   its subtree size.
+//   This way, once rtn.subtree_size = |rtn.children| we know we
+//   can pull subtree sizes from children and continue pushing
+//   the readiness signal further up.
+func (rtn *reachabilityTreeNode) countSubtrees() uint64 {
+	queue := []*reachabilityTreeNode{rtn}
+	for len(queue) > 0 {
+		var current *reachabilityTreeNode
+		current, queue = queue[0], queue[1:]
+		if len(current.children) == 0 {
+			// We reached a leaf
+			current.subtreeSize = 1
+		}
+		if current.subtreeSize <= uint64(len(current.children)) {
+			// We haven't yet calculated the subtree size of
+			// the current node. Add all its children to the
+			// queue
+			for _, child := range current.children {
+				queue = append(queue, child)
+			}
+			continue
+		}
+
+		// We reached a leaf or a pre-calculated subtree.
+		// Push information up
+		for current != rtn {
+			current = current.parent
+			current.subtreeSize++
+			if current.subtreeSize != uint64(len(current.children)) {
+				// Not all subtrees of the current node are ready
+				break
+			}
+			// All subtrees of current have reported readiness.
+			// Count actual subtree size and continue pushing up.
+			childSubtreeSizeSum := uint64(0)
+			for _, child := range current.children {
+				childSubtreeSizeSum += child.subtreeSize
+			}
+			current.subtreeSize = childSubtreeSizeSum + 1
+		}
+	}
+	return rtn.subtreeSize
+}
+
+// propagateInterval propagates the new interval using a BFS traversal.
+// Subtree intervals are recursively allocated according to subtree sizes and
+// the allocation rule in splitWithExponentialBias. This method returns
+// a list of reachabilityTreeNodes modified by it.
+func (rtn *reachabilityTreeNode) propagateInterval() ([]*reachabilityTreeNode, error) {
+	// We set the interval to reset its remainingInterval, so we could reallocate it while reindexing.
+	rtn.setInterval(rtn.interval)
+	modifiedNodes := []*reachabilityTreeNode{rtn}
+	queue := []*reachabilityTreeNode{rtn}
+	for len(queue) > 0 {
+		var current *reachabilityTreeNode
+		current, queue = queue[0], queue[1:]
+		if len(current.children) > 0 {
+			sizes := make([]uint64, len(current.children))
+			for i, child := range current.children {
+				sizes[i] = child.subtreeSize
+			}
+			intervals, err := current.remainingInterval.splitWithExponentialBias(sizes)
+			if err != nil {
+				return nil, err
+			}
+			for i, child := range current.children {
+				childInterval := intervals[i]
+				child.setInterval(childInterval)
+				queue = append(queue, child)
+			}
+
+			// Empty up remaining interval
+			current.remainingInterval.start = current.remainingInterval.end + 1
+		}
+
+		modifiedNodes = append(modifiedNodes, current)
+
+		// Cleanup temp info for future reindexing
+		current.subtreeSize = 0
+	}
+	return modifiedNodes, nil
+}
+
+// isAncestorOf checks if this node is a reachability tree ancestor
+// of the other node.
+func (rtn *reachabilityTreeNode) isAncestorOf(other *reachabilityTreeNode) bool {
+	return rtn.interval.isAncestorOf(other.interval)
+}
+
+// String returns a string representation of a reachability tree node
+// and its children.
+func (rtn *reachabilityTreeNode) String() string {
+	queue := []*reachabilityTreeNode{rtn}
+	lines := []string{rtn.interval.String()}
+	for len(queue) > 0 {
+		var current *reachabilityTreeNode
+		current, queue = queue[0], queue[1:]
+		if len(current.children) == 0 {
+			break
+		}
+
+		line := ""
+		for _, child := range current.children {
+			line += child.interval.String()
+			queue = append(queue, child)
+		}
+		lines = append([]string{line}, lines...)
+	}
+	return strings.Join(lines, "\n")
+}
+
+// futureCoveringBlockSet represents a collection of blocks in the future of
+// a certain block. Once a block B is added to the DAG, every block A_i in
+// B's selected parent anticone must register B in its futureCoveringBlockSet. This allows
+// to relatively quickly (O(log(|futureCoveringBlockSet|))) query whether B
+// is a descendent (is in the "future") of any block that previously
+// registered it.
+//
+// Note that futureCoveringBlockSet is meant to be queried only if B is not
+// a reachability tree descendant of the block in question, as reachability
+// tree queries are always O(1).
+//
+// See insertBlock, isInFuture, and dag.isAncestorOf for further details.
+type futureCoveringBlockSet []*futureCoveringBlock
+
+// futureCoveringBlock represents a block in the future of some other block.
+type futureCoveringBlock struct {
+	blockNode *blockNode
+	treeNode  *reachabilityTreeNode
+}
+
+// insertBlock inserts the given block into this futureCoveringBlockSet
+// while keeping futureCoveringBlockSet ordered by interval.
+// If a block B ∈ futureCoveringBlockSet exists such that its interval
+// contains block's interval, block need not be added. If block's
+// interval contains B's interval, it replaces it.
+//
+// Notes:
+// * Intervals never intersect unless one contains the other
+//   (this follows from the tree structure and the indexing rule).
+// * Since futureCoveringBlockSet is kept ordered, a binary search can be
+//   used for insertion/queries.
+// * Although reindexing may change a block's interval, the
+//   is-superset relation will by definition
+//   be always preserved.
+func (fb *futureCoveringBlockSet) insertBlock(block *futureCoveringBlock) {
+	blockInterval := block.treeNode.interval
+	i := fb.findIndex(block)
+	if i > 0 {
+		candidate := (*fb)[i-1]
+		candidateInterval := candidate.treeNode.interval
+		if candidateInterval.isAncestorOf(blockInterval) {
+			// candidate is an ancestor of block, no need to insert
+			return
+		}
+		if blockInterval.isAncestorOf(candidateInterval) {
+			// block is an ancestor of candidate, and can thus replace it
+			(*fb)[i-1] = block
+			return
+		}
+	}
+
+	// Insert block in the correct index to maintain futureCoveringBlockSet as
+	// a sorted-by-interval list.
+	// Note that i might be equal to len(futureCoveringBlockSet)
+	left := (*fb)[:i]
+	right := append([]*futureCoveringBlock{block}, (*fb)[i:]...)
+	*fb = append(left, right...)
+}
+
+// isInFuture resolves whether the given block is in the subtree of
+// any block in this futureCoveringBlockSet.
+// See insertBlock method for the complementary insertion behavior.
+//
+// Like the insert method, this method also relies on the fact that
+// futureCoveringBlockSet is kept ordered by interval to efficiently perform a
+// binary search over futureCoveringBlockSet and answer the query in
+// O(log(|futureCoveringBlockSet|)).
+func (fb futureCoveringBlockSet) isInFuture(block *futureCoveringBlock) bool {
+	i := fb.findIndex(block)
+	if i == 0 {
+		// No candidate to contain block
+		return false
+	}
+
+	candidate := fb[i-1]
+	return candidate.treeNode.isAncestorOf(block.treeNode)
+}
+
+// findIndex finds the index of the block with the maximum start that is below
+// the given block.
+func (fb futureCoveringBlockSet) findIndex(block *futureCoveringBlock) int {
+	blockInterval := block.treeNode.interval
+	end := blockInterval.end
+
+	low := 0
+	high := len(fb)
+	for low < high {
+		middle := (low + high) / 2
+		middleInterval := fb[middle].treeNode.interval
+		if end < middleInterval.start {
+			high = middle
+		} else {
+			low = middle + 1
+		}
+	}
+	return low
+}
+
+// String returns a string representation of the intervals in this futureCoveringBlockSet.
+func (fb futureCoveringBlockSet) String() string {
+	intervalsString := ""
+	for _, block := range fb {
+		intervalsString += block.treeNode.interval.String()
+	}
+	return intervalsString
+}
+
+func (dag *BlockDAG) updateReachability(node *blockNode, selectedParentAnticone []*blockNode) error {
+	// Allocate a new reachability tree node
+	newTreeNode := newReachabilityTreeNode(node)
+
+	// If this is the genesis node, simply initialize it and return
+	if node.isGenesis() {
+		return dag.reachabilityStore.setTreeNode(newTreeNode)
+	}
+
+	// Insert the node into the selected parent's reachability tree
+	selectedParentTreeNode, err := dag.reachabilityStore.treeNodeByBlockNode(node.selectedParent)
+	if err != nil {
+		return err
+	}
+	modifiedTreeNodes, err := selectedParentTreeNode.addChild(newTreeNode)
+	if err != nil {
+		return err
+	}
+	for _, modifiedTreeNode := range modifiedTreeNodes {
+		err = dag.reachabilityStore.setTreeNode(modifiedTreeNode)
+		if err != nil {
+			return err
+		}
+	}
+
+	// Add the block to the futureCoveringSets of all the blocks
+	// in the selected parent's anticone
+	for _, current := range selectedParentAnticone {
+		currentFutureCoveringSet, err := dag.reachabilityStore.futureCoveringSetByBlockNode(current)
+		if err != nil {
+			return err
+		}
+		currentFutureCoveringSet.insertBlock(&futureCoveringBlock{blockNode: node, treeNode: newTreeNode})
+		err = dag.reachabilityStore.setFutureCoveringSet(current, currentFutureCoveringSet)
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// isAncestorOf returns true if this node is in the past of the other node
+// in the DAG. The complexity of this method is O(log(|this.futureCoveringBlockSet|))
+func (dag *BlockDAG) isAncestorOf(this *blockNode, other *blockNode) (bool, error) {
+	// First, check if this node is a reachability tree ancestor of the
+	// other node
+	thisTreeNode, err := dag.reachabilityStore.treeNodeByBlockNode(this)
+	if err != nil {
+		return false, err
+	}
+	otherTreeNode, err := dag.reachabilityStore.treeNodeByBlockNode(other)
+	if err != nil {
+		return false, err
+	}
+	if thisTreeNode.isAncestorOf(otherTreeNode) {
+		return true, nil
+	}
+
+	// Otherwise, use previously registered future blocks to complete the
+	// reachability test
+	thisFutureCoveringSet, err := dag.reachabilityStore.futureCoveringSetByBlockNode(this)
+	if err != nil {
+		return false, err
+	}
+	return thisFutureCoveringSet.isInFuture(&futureCoveringBlock{blockNode: other, treeNode: otherTreeNode}), nil
+}

blockdag/reachability_test.go

@@ -0,0 +1,475 @@
+package blockdag
+
+import (
+	"reflect"
+	"testing"
+)
+
+func TestAddChild(t *testing.T) {
+	// Scenario 1: test addChild in a chain
+	//             root -> a -> b -> c...
+	// Create the root node of a new reachability tree
+	root := newReachabilityTreeNode(&blockNode{})
+	root.setInterval(newReachabilityInterval(1, 100))
+
+	// Add a chain of child nodes just before a reindex occurs (2^6=64 < 100)
+	currentTip := root
+	for i := 0; i < 6; i++ {
+		node := newReachabilityTreeNode(&blockNode{})
+		modifiedNodes, err := currentTip.addChild(node)
+		if err != nil {
+			t.Fatalf("TestAddChild: addChild failed: %s", err)
+		}
+
+		// Expect only the node and its parent to be affected
+		expectedModifiedNodes := []*reachabilityTreeNode{currentTip, node}
+		if !reflect.DeepEqual(modifiedNodes, expectedModifiedNodes) {
+			t.Fatalf("TestAddChild: unexpected modifiedNodes. "+
+				"want: %s, got: %s", expectedModifiedNodes, modifiedNodes)
+		}
+
+		currentTip = node
+	}
+
+	// Add another node to the tip of the chain to trigger a reindex (100 < 2^7=128)
+	lastChild := newReachabilityTreeNode(&blockNode{})
+	modifiedNodes, err := currentTip.addChild(lastChild)
+	if err != nil {
+		t.Fatalf("TestAddChild: addChild failed: %s", err)
+	}
+
+	// Expect more than just the node and its parent to be modified but not
+	// all the nodes
+	if len(modifiedNodes) <= 2 && len(modifiedNodes) >= 7 {
+		t.Fatalf("TestAddChild: unexpected amount of modifiedNodes.")
+	}
+
+	// Expect the tip to have an interval of 1 and remaining interval of 0
+	tipInterval := lastChild.interval.size()
+	if tipInterval != 1 {
+		t.Fatalf("TestAddChild: unexpected tip interval size: want: 1, got: %d", tipInterval)
+	}
+	tipRemainingInterval := lastChild.remainingInterval.size()
+	if tipRemainingInterval != 0 {
+		t.Fatalf("TestAddChild: unexpected tip interval size: want: 0, got: %d", tipRemainingInterval)
+	}
+
+	// Expect all nodes to be descendant nodes of root
+	currentNode := currentTip
+	for currentNode != nil {
+		if !root.isAncestorOf(currentNode) {
+			t.Fatalf("TestAddChild: currentNode is not a descendant of root")
+		}
+		currentNode = currentNode.parent
+	}
+
+	// Scenario 2: test addChild where all nodes are direct descendants of root
+	//             root -> a, b, c...
+	// Create the root node of a new reachability tree
+	root = newReachabilityTreeNode(&blockNode{})
+	root.setInterval(newReachabilityInterval(1, 100))
+
+	// Add child nodes to root just before a reindex occurs (2^6=64 < 100)
+	childNodes := make([]*reachabilityTreeNode, 6)
+	for i := 0; i < len(childNodes); i++ {
+		childNodes[i] = newReachabilityTreeNode(&blockNode{})
+		modifiedNodes, err := root.addChild(childNodes[i])
+		if err != nil {
+			t.Fatalf("TestAddChild: addChild failed: %s", err)
+		}
+
+		// Expect only the node and the root to be affected
+		expectedModifiedNodes := []*reachabilityTreeNode{root, childNodes[i]}
+		if !reflect.DeepEqual(modifiedNodes, expectedModifiedNodes) {
+			t.Fatalf("TestAddChild: unexpected modifiedNodes. "+
+				"want: %s, got: %s", expectedModifiedNodes, modifiedNodes)
+		}
+	}
+
+	// Add another node to the root to trigger a reindex (100 < 2^7=128)
+	lastChild = newReachabilityTreeNode(&blockNode{})
+	modifiedNodes, err = root.addChild(lastChild)
+	if err != nil {
+		t.Fatalf("TestAddChild: addChild failed: %s", err)
+	}
+
+	// Expect more than just the node and the root to be modified but not
+	// all the nodes
+	if len(modifiedNodes) <= 2 && len(modifiedNodes) >= 7 {
+		t.Fatalf("TestAddChild: unexpected amount of modifiedNodes.")
+	}
+
+	// Expect the last-added child to have an interval of 1 and remaining interval of 0
+	lastChildInterval := lastChild.interval.size()
+	if lastChildInterval != 1 {
+		t.Fatalf("TestAddChild: unexpected lastChild interval size: want: 1, got: %d", lastChildInterval)
+	}
+	lastChildRemainingInterval := lastChild.remainingInterval.size()
+	if lastChildRemainingInterval != 0 {
+		t.Fatalf("TestAddChild: unexpected lastChild interval size: want: 0, got: %d", lastChildRemainingInterval)
+	}
+
+	// Expect all nodes to be descendant nodes of root
+	for _, childNode := range childNodes {
+		if !root.isAncestorOf(childNode) {
+			t.Fatalf("TestAddChild: childNode is not a descendant of root")
+		}
+	}
+}
+
+func TestSplitFraction(t *testing.T) {
+	tests := []struct {
+		interval      *reachabilityInterval
+		fraction      float64
+		expectedLeft  *reachabilityInterval
+		expectedRight *reachabilityInterval
+	}{
+		{
+			interval:      newReachabilityInterval(1, 100),
+			fraction:      0.5,
+			expectedLeft:  newReachabilityInterval(1, 50),
+			expectedRight: newReachabilityInterval(51, 100),
+		},
+		{
+			interval:      newReachabilityInterval(2, 100),
+			fraction:      0.5,
+			expectedLeft:  newReachabilityInterval(2, 51),
+			expectedRight: newReachabilityInterval(52, 100),
+		},
+		{
+			interval:      newReachabilityInterval(1, 99),
+			fraction:      0.5,
+			expectedLeft:  newReachabilityInterval(1, 50),
+			expectedRight: newReachabilityInterval(51, 99),
+		},
+		{
+			interval:      newReachabilityInterval(1, 100),
+			fraction:      0.2,
+			expectedLeft:  newReachabilityInterval(1, 20),
+			expectedRight: newReachabilityInterval(21, 100),
+		},
+		{
+			interval:      newReachabilityInterval(1, 100),
+			fraction:      0,
+			expectedLeft:  newReachabilityInterval(1, 0),
+			expectedRight: newReachabilityInterval(1, 100),
+		},
+		{
+			interval:      newReachabilityInterval(1, 100),
+			fraction:      1,
+			expectedLeft:  newReachabilityInterval(1, 100),
+			expectedRight: newReachabilityInterval(101, 100),
+		},
+	}
+
+	for i, test := range tests {
+		left, right, err := test.interval.splitFraction(test.fraction)
+		if err != nil {
+			t.Fatalf("TestSplitFraction: splitFraction unexpectedly failed in test #%d: %s", i, err)
+		}
+		if !reflect.DeepEqual(left, test.expectedLeft) {
+			t.Errorf("TestSplitFraction: unexpected left in test #%d. "+
+				"want: %s, got: %s", i, test.expectedLeft, left)
+		}
+		if !reflect.DeepEqual(right, test.expectedRight) {
+			t.Errorf("TestSplitFraction: unexpected right in test #%d. "+
+				"want: %s, got: %s", i, test.expectedRight, right)
+		}
+	}
+}
+
+func TestSplitExact(t *testing.T) {
+	tests := []struct {
+		interval          *reachabilityInterval
+		sizes             []uint64
+		expectedIntervals []*reachabilityInterval
+	}{
+		{
+			interval: newReachabilityInterval(1, 100),
+			sizes:    []uint64{100},
+			expectedIntervals: []*reachabilityInterval{
+				newReachabilityInterval(1, 100),
+			},
+		},
+		{
+			interval: newReachabilityInterval(1, 100),
+			sizes:    []uint64{50, 50},
+			expectedIntervals: []*reachabilityInterval{
+				newReachabilityInterval(1, 50),
+				newReachabilityInterval(51, 100),
+			},
+		},
+		{
+			interval: newReachabilityInterval(1, 100),
+			sizes:    []uint64{10, 20, 30, 40},
+			expectedIntervals: []*reachabilityInterval{
+				newReachabilityInterval(1, 10),
+				newReachabilityInterval(11, 30),
+				newReachabilityInterval(31, 60),
+				newReachabilityInterval(61, 100),
+			},
+		},
+		{
+			interval: newReachabilityInterval(1, 100),
+			sizes:    []uint64{0, 100},
+			expectedIntervals: []*reachabilityInterval{
+				newReachabilityInterval(1, 0),
+				newReachabilityInterval(1, 100),
+			},
+		},
+		{
+			interval: newReachabilityInterval(1, 100),
+			sizes:    []uint64{100, 0},
+			expectedIntervals: []*reachabilityInterval{
+				newReachabilityInterval(1, 100),
+				newReachabilityInterval(101, 100),
+			},
+		},
+	}
+
+	for i, test := range tests {
+		intervals, err := test.interval.splitExact(test.sizes)
+		if err != nil {
+			t.Fatalf("TestSplitExact: splitExact unexpectedly failed in test #%d: %s", i, err)
+		}
+		if !reflect.DeepEqual(intervals, test.expectedIntervals) {
+			t.Errorf("TestSplitExact: unexpected intervals in test #%d. "+
+				"want: %s, got: %s", i, test.expectedIntervals, intervals)
+		}
+	}
+}
+
+func TestSplitWithExponentialBias(t *testing.T) {
+	tests := []struct {
+		interval          *reachabilityInterval
+		sizes             []uint64
+		expectedIntervals []*reachabilityInterval
+	}{
+		{
+			interval: newReachabilityInterval(1, 100),
+			sizes:    []uint64{100},
+			expectedIntervals: []*reachabilityInterval{
+				newReachabilityInterval(1, 100),
+			},
+		},
+		{
+			interval: newReachabilityInterval(1, 100),
+			sizes:    []uint64{50, 50},
+			expectedIntervals: []*reachabilityInterval{
+				newReachabilityInterval(1, 50),
+				newReachabilityInterval(51, 100),
+			},
+		},
+		{
+			interval: newReachabilityInterval(1, 100),
+			sizes:    []uint64{10, 20, 30, 40},
+			expectedIntervals: []*reachabilityInterval{
+				newReachabilityInterval(1, 10),
+				newReachabilityInterval(11, 30),
+				newReachabilityInterval(31, 60),
+				newReachabilityInterval(61, 100),
+			},
+		},
+		{
+			interval: newReachabilityInterval(1, 100),
+			sizes:    []uint64{25, 25},
+			expectedIntervals: []*reachabilityInterval{
+				newReachabilityInterval(1, 50),
+				newReachabilityInterval(51, 100),
+			},
+		},
+		{
+			interval: newReachabilityInterval(1, 100),
+			sizes:    []uint64{1, 1},
+			expectedIntervals: []*reachabilityInterval{
+				newReachabilityInterval(1, 50),
+				newReachabilityInterval(51, 100),
+			},
+		},
+		{
+			interval: newReachabilityInterval(1, 100),
+			sizes:    []uint64{33, 33, 33},
+			expectedIntervals: []*reachabilityInterval{
+				newReachabilityInterval(1, 33),
+				newReachabilityInterval(34, 66),
+				newReachabilityInterval(67, 100),
+			},
+		},
+		{
+			interval: newReachabilityInterval(1, 100),
+			sizes:    []uint64{10, 15, 25},
+			expectedIntervals: []*reachabilityInterval{
+				newReachabilityInterval(1, 10),
+				newReachabilityInterval(11, 25),
+				newReachabilityInterval(26, 100),
+			},
+		},
+		{
+			interval: newReachabilityInterval(1, 100),
+			sizes:    []uint64{25, 15, 10},
+			expectedIntervals: []*reachabilityInterval{
+				newReachabilityInterval(1, 75),
+				newReachabilityInterval(76, 90),
+				newReachabilityInterval(91, 100),
+			},
+		},
+		{
+			interval: newReachabilityInterval(1, 10_000),
+			sizes:    []uint64{10, 10, 20},
+			expectedIntervals: []*reachabilityInterval{
+				newReachabilityInterval(1, 20),
+				newReachabilityInterval(21, 40),
+				newReachabilityInterval(41, 10_000),
+			},
+		},
+	}
+
+	for i, test := range tests {
+		intervals, err := test.interval.splitWithExponentialBias(test.sizes)
+		if err != nil {
+			t.Fatalf("TestSplitWithExponentialBias: splitWithExponentialBias unexpectedly failed in test #%d: %s", i, err)
+		}
+		if !reflect.DeepEqual(intervals, test.expectedIntervals) {
+			t.Errorf("TestSplitWithExponentialBias: unexpected intervals in test #%d. "+
+				"want: %s, got: %s", i, test.expectedIntervals, intervals)
+		}
+	}
+}
+
+func TestIsInFuture(t *testing.T) {
+	blocks := futureCoveringBlockSet{
+		{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(2, 3)}},
+		{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(4, 67)}},
+		{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(67, 77)}},
+		{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(657, 789)}},
+		{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1000, 1000)}},
+		{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1920, 1921)}},
+	}
+
+	tests := []struct {
+		block          *futureCoveringBlock
+		expectedResult bool
+	}{
+		{
+			block:          &futureCoveringBlock{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1, 1)}},
+			expectedResult: false,
+		},
+		{
+			block:          &futureCoveringBlock{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(5, 7)}},
+			expectedResult: true,
+		},
+		{
+			block:          &futureCoveringBlock{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(67, 76)}},
+			expectedResult: true,
+		},
+		{
+			block:          &futureCoveringBlock{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(78, 100)}},
+			expectedResult: false,
+		},
+		{
+			block:          &futureCoveringBlock{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1980, 2000)}},
+			expectedResult: false,
+		},
+		{
+			block:          &futureCoveringBlock{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1920, 1920)}},
+			expectedResult: true,
+		},
+	}
+
+	for i, test := range tests {
+		result := blocks.isInFuture(test.block)
+		if result != test.expectedResult {
+			t.Errorf("TestIsInFuture: unexpected result in test #%d. Want: %t, got: %t",
+				i, test.expectedResult, result)
+		}
+	}
+}
+
+func TestInsertBlock(t *testing.T) {
+	blocks := futureCoveringBlockSet{
+		{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1, 3)}},
+		{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(4, 67)}},
+		{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(67, 77)}},
+		{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(657, 789)}},
+		{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1000, 1000)}},
+		{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1920, 1921)}},
+	}
+
+	tests := []struct {
+		toInsert       []*futureCoveringBlock
+		expectedResult futureCoveringBlockSet
+	}{
+		{
+			toInsert: []*futureCoveringBlock{
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(5, 7)}},
+			},
+			expectedResult: futureCoveringBlockSet{
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1, 3)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(4, 67)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(67, 77)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(657, 789)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1000, 1000)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1920, 1921)}},
+			},
+		},
+		{
+			toInsert: []*futureCoveringBlock{
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(65, 78)}},
+			},
+			expectedResult: futureCoveringBlockSet{
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1, 3)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(4, 67)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(65, 78)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(657, 789)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1000, 1000)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1920, 1921)}},
+			},
+		},
+		{
+			toInsert: []*futureCoveringBlock{
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(88, 97)}},
+			},
+			expectedResult: futureCoveringBlockSet{
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1, 3)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(4, 67)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(67, 77)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(88, 97)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(657, 789)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1000, 1000)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1920, 1921)}},
+			},
+		},
+		{
+			toInsert: []*futureCoveringBlock{
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(88, 97)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(3000, 3010)}},
+			},
+			expectedResult: futureCoveringBlockSet{
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1, 3)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(4, 67)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(67, 77)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(88, 97)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(657, 789)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1000, 1000)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(1920, 1921)}},
+				{treeNode: &reachabilityTreeNode{interval: newReachabilityInterval(3000, 3010)}},
+			},
+		},
+	}
+
+	for i, test := range tests {
+		// Create a clone of blocks so that we have a clean start for every test
+		blocksClone := make(futureCoveringBlockSet, len(blocks))
+		for i, block := range blocks {
+			blocksClone[i] = block
+		}
+
+		for _, block := range test.toInsert {
+			blocksClone.insertBlock(block)
+		}
+		if !reflect.DeepEqual(blocksClone, test.expectedResult) {
+			t.Errorf("TestInsertBlock: unexpected result in test #%d. Want: %s, got: %s",
+				i, test.expectedResult, blocksClone)
+		}
+	}
+}

blockdag/reachabilitystore.go

@@ -0,0 +1,392 @@
+package blockdag
+
+import (
+	"bytes"
+	"github.com/kaspanet/kaspad/database"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/wire"
+	"github.com/pkg/errors"
+	"io"
+)
+
+type reachabilityData struct {
+	treeNode          *reachabilityTreeNode
+	futureCoveringSet futureCoveringBlockSet
+}
+
+type reachabilityStore struct {
+	dag    *BlockDAG
+	dirty  map[daghash.Hash]struct{}
+	loaded map[daghash.Hash]*reachabilityData
+}
+
+func newReachabilityStore(dag *BlockDAG) *reachabilityStore {
+	return &reachabilityStore{
+		dag:    dag,
+		dirty:  make(map[daghash.Hash]struct{}),
+		loaded: make(map[daghash.Hash]*reachabilityData),
+	}
+}
+
+func (store *reachabilityStore) setTreeNode(treeNode *reachabilityTreeNode) error {
+	// load the reachability data from DB to store.loaded
+	node := treeNode.blockNode
+	_, exists := store.reachabilityDataByHash(node.hash)
+	if !exists {
+		store.loaded[*node.hash] = &reachabilityData{}
+	}
+
+	store.loaded[*node.hash].treeNode = treeNode
+	store.setBlockAsDirty(node.hash)
+	return nil
+}
+
+func (store *reachabilityStore) setFutureCoveringSet(node *blockNode, futureCoveringSet futureCoveringBlockSet) error {
+	// load the reachability data from DB to store.loaded
+	_, exists := store.reachabilityDataByHash(node.hash)
+	if !exists {
+		return reachabilityNotFoundError(node)
+	}
+
+	store.loaded[*node.hash].futureCoveringSet = futureCoveringSet
+	store.setBlockAsDirty(node.hash)
+	return nil
+}
+
+func (store *reachabilityStore) setBlockAsDirty(blockHash *daghash.Hash) {
+	store.dirty[*blockHash] = struct{}{}
+}
+
+func reachabilityNotFoundError(node *blockNode) error {
+	return errors.Errorf("Couldn't find reachability data for block %s", node.hash)
+}
+
+func (store *reachabilityStore) treeNodeByBlockNode(node *blockNode) (*reachabilityTreeNode, error) {
+	reachabilityData, exists := store.reachabilityDataByHash(node.hash)
+	if !exists {
+		return nil, reachabilityNotFoundError(node)
+	}
+	return reachabilityData.treeNode, nil
+}
+
+func (store *reachabilityStore) futureCoveringSetByBlockNode(node *blockNode) (futureCoveringBlockSet, error) {
+	reachabilityData, exists := store.reachabilityDataByHash(node.hash)
+	if !exists {
+		return nil, reachabilityNotFoundError(node)
+	}
+	return reachabilityData.futureCoveringSet, nil
+}
+
+func (store *reachabilityStore) reachabilityDataByHash(hash *daghash.Hash) (*reachabilityData, bool) {
+	reachabilityData, ok := store.loaded[*hash]
+	return reachabilityData, ok
+}
+
+// flushToDB writes all dirty reachability data to the database.
+func (store *reachabilityStore) flushToDB(dbTx database.Tx) error {
+	if len(store.dirty) == 0 {
+		return nil
+	}
+
+	for hash := range store.dirty {
+		reachabilityData := store.loaded[hash]
+		err := store.dbStoreReachabilityData(dbTx, &hash, reachabilityData)
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (store *reachabilityStore) clearDirtyEntries() {
+	store.dirty = make(map[daghash.Hash]struct{})
+}
+
+func (store *reachabilityStore) init(dbTx database.Tx) error {
+	bucket := dbTx.Metadata().Bucket(reachabilityDataBucketName)
+
+	// TODO: (Stas) This is a quick and dirty hack.
+	// We iterate over the entire bucket twice:
+	// * First, populate the loaded set with all entries
+	// * Second, connect the parent/children pointers in each entry
+	//   with other nodes, which are now guaranteed to exist
+	cursor := bucket.Cursor()
+	for ok := cursor.First(); ok; ok = cursor.Next() {
+		err := store.initReachabilityData(cursor)
+		if err != nil {
+			return err
+		}
+	}
+	cursor = bucket.Cursor()
+	for ok := cursor.First(); ok; ok = cursor.Next() {
+		err := store.loadReachabilityDataFromCursor(cursor)
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (store *reachabilityStore) initReachabilityData(cursor database.Cursor) error {
+	hash, err := daghash.NewHash(cursor.Key())
+	if err != nil {
+		return err
+	}
+
+	store.loaded[*hash] = &reachabilityData{
+		treeNode:          &reachabilityTreeNode{},
+		futureCoveringSet: nil,
+	}
+	return nil
+}
+
+func (store *reachabilityStore) loadReachabilityDataFromCursor(cursor database.Cursor) error {
+	hash, err := daghash.NewHash(cursor.Key())
+	if err != nil {
+		return err
+	}
+
+	reachabilityData, ok := store.reachabilityDataByHash(hash)
+	if !ok {
+		return errors.Errorf("cannot find reachability data for block hash: %s", hash)
+	}
+
+	err = store.deserializeReachabilityData(cursor.Value(), reachabilityData)
+	if err != nil {
+		return err
+	}
+
+	// Connect the treeNode with its blockNode
+	reachabilityData.treeNode.blockNode = store.dag.index.LookupNode(hash)
+
+	return nil
+}
+
+// dbStoreReachabilityData stores the reachability data to the database.
+// This overwrites the current entry if there exists one.
+func (store *reachabilityStore) dbStoreReachabilityData(dbTx database.Tx, hash *daghash.Hash, reachabilityData *reachabilityData) error {
+	serializedReachabilyData, err := store.serializeReachabilityData(reachabilityData)
+	if err != nil {
+		return err
+	}
+
+	return dbTx.Metadata().Bucket(reachabilityDataBucketName).Put(hash[:], serializedReachabilyData)
+}
+
+func (store *reachabilityStore) serializeReachabilityData(reachabilityData *reachabilityData) ([]byte, error) {
+	w := &bytes.Buffer{}
+	err := store.serializeTreeNode(w, reachabilityData.treeNode)
+	if err != nil {
+		return nil, err
+	}
+	err = store.serializeFutureCoveringSet(w, reachabilityData.futureCoveringSet)
+	if err != nil {
+		return nil, err
+	}
+	return w.Bytes(), nil
+}
+
+func (store *reachabilityStore) serializeTreeNode(w io.Writer, treeNode *reachabilityTreeNode) error {
+	// Serialize the interval
+	err := store.serializeReachabilityInterval(w, treeNode.interval)
+	if err != nil {
+		return err
+	}
+
+	// Serialize the remaining interval
+	err = store.serializeReachabilityInterval(w, treeNode.remainingInterval)
+	if err != nil {
+		return err
+	}
+
+	// Serialize the parent
+	// If this is the genesis block, write the zero hash instead
+	parentHash := &daghash.ZeroHash
+	if treeNode.parent != nil {
+		parentHash = treeNode.parent.blockNode.hash
+	}
+	err = wire.WriteElement(w, parentHash)
+	if err != nil {
+		return err
+	}
+
+	// Serialize the amount of children
+	err = wire.WriteVarInt(w, uint64(len(treeNode.children)))
+	if err != nil {
+		return err
+	}
+
+	// Serialize the children
+	for _, child := range treeNode.children {
+		err = wire.WriteElement(w, child.blockNode.hash)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (store *reachabilityStore) serializeReachabilityInterval(w io.Writer, interval *reachabilityInterval) error {
+	// Serialize start
+	err := wire.WriteElement(w, interval.start)
+	if err != nil {
+		return err
+	}
+
+	// Serialize end
+	err = wire.WriteElement(w, interval.end)
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (store *reachabilityStore) serializeFutureCoveringSet(w io.Writer, futureCoveringSet futureCoveringBlockSet) error {
+	// Serialize the set size
+	err := wire.WriteVarInt(w, uint64(len(futureCoveringSet)))
+	if err != nil {
+		return err
+	}
+
+	// Serialize each block in the set
+	for _, block := range futureCoveringSet {
+		err = wire.WriteElement(w, block.blockNode.hash)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (store *reachabilityStore) deserializeReachabilityData(
+	serializedReachabilityDataBytes []byte, destination *reachabilityData) error {
+
+	r := bytes.NewBuffer(serializedReachabilityDataBytes)
+
+	// Deserialize the tree node
+	err := store.deserializeTreeNode(r, destination)
+	if err != nil {
+		return err
+	}
+
+	// Deserialize the future covering set
+	err = store.deserializeFutureCoveringSet(r, destination)
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (store *reachabilityStore) deserializeTreeNode(r io.Reader, destination *reachabilityData) error {
+	// Deserialize the interval
+	interval, err := store.deserializeReachabilityInterval(r)
+	if err != nil {
+		return err
+	}
+	destination.treeNode.interval = interval
+
+	// Deserialize the remaining interval
+	remainingInterval, err := store.deserializeReachabilityInterval(r)
+	if err != nil {
+		return err
+	}
+	destination.treeNode.remainingInterval = remainingInterval
+
+	// Deserialize the parent
+	// If this is the zero hash, this node is the genesis and as such doesn't have a parent
+	parentHash := &daghash.Hash{}
+	err = wire.ReadElement(r, parentHash)
+	if err != nil {
+		return err
+	}
+	if !daghash.ZeroHash.IsEqual(parentHash) {
+		parentReachabilityData, ok := store.reachabilityDataByHash(parentHash)
+		if !ok {
+			return errors.Errorf("parent reachability data not found for hash: %s", parentHash)
+		}
+		destination.treeNode.parent = parentReachabilityData.treeNode
+	}
+
+	// Deserialize the amount of children
+	childCount, err := wire.ReadVarInt(r)
+	if err != nil {
+		return err
+	}
+
+	// Deserialize the children
+	children := make([]*reachabilityTreeNode, childCount)
+	for i := uint64(0); i < childCount; i++ {
+		childHash := &daghash.Hash{}
+		err = wire.ReadElement(r, childHash)
+		if err != nil {
+			return err
+		}
+		childReachabilityData, ok := store.reachabilityDataByHash(childHash)
+		if !ok {
+			return errors.Errorf("child reachability data not found for hash: %s", parentHash)
+		}
+		children[i] = childReachabilityData.treeNode
+	}
+	destination.treeNode.children = children
+
+	return nil
+}
+
+func (store *reachabilityStore) deserializeReachabilityInterval(r io.Reader) (*reachabilityInterval, error) {
+	interval := &reachabilityInterval{}
+
+	// Deserialize start
+	start := uint64(0)
+	err := wire.ReadElement(r, &start)
+	if err != nil {
+		return nil, err
+	}
+	interval.start = start
+
+	// Deserialize end
+	end := uint64(0)
+	err = wire.ReadElement(r, &end)
+	if err != nil {
+		return nil, err
+	}
+	interval.end = end
+
+	return interval, nil
+}
+
+func (store *reachabilityStore) deserializeFutureCoveringSet(r io.Reader, destination *reachabilityData) error {
+	// Deserialize the set size
+	setSize, err := wire.ReadVarInt(r)
+	if err != nil {
+		return err
+	}
+
+	// Deserialize each block in the set
+	futureCoveringSet := make(futureCoveringBlockSet, setSize)
+	for i := uint64(0); i < setSize; i++ {
+		blockHash := &daghash.Hash{}
+		err = wire.ReadElement(r, blockHash)
+		if err != nil {
+			return err
+		}
+		blockNode := store.dag.index.LookupNode(blockHash)
+		if blockNode == nil {
+			return errors.Errorf("blockNode not found for hash %s", blockHash)
+		}
+		blockReachabilityData, ok := store.reachabilityDataByHash(blockHash)
+		if !ok {
+			return errors.Errorf("block reachability data not found for hash: %s", blockHash)
+		}
+		futureCoveringSet[i] = &futureCoveringBlock{
+			blockNode: blockNode,
+			treeNode:  blockReachabilityData.treeNode,
+		}
+	}
+	destination.futureCoveringSet = futureCoveringSet
+
+	return nil
+}

blockdag/scriptval.go

@@ -0,0 +1,241 @@
+// Copyright (c) 2013-2016 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"fmt"
+	"runtime"
+	"time"
+
+	"github.com/kaspanet/kaspad/txscript"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/kaspanet/kaspad/wire"
+)
+
+// txValidateItem holds a transaction along with which input to validate.
+type txValidateItem struct {
+	txInIndex int
+	txIn      *wire.TxIn
+	tx        *util.Tx
+}
+
+// txValidator provides a type which asynchronously validates transaction
+// inputs. It provides several channels for communication and a processing
+// function that is intended to be in run multiple goroutines.
+type txValidator struct {
+	validateChan chan *txValidateItem
+	quitChan     chan struct{}
+	resultChan   chan error
+	utxoSet      UTXOSet
+	flags        txscript.ScriptFlags
+	sigCache     *txscript.SigCache
+}
+
+// sendResult sends the result of a script pair validation on the internal
+// result channel while respecting the quit channel. This allows orderly
+// shutdown when the validation process is aborted early due to a validation
+// error in one of the other goroutines.
+func (v *txValidator) sendResult(result error) {
+	select {
+	case v.resultChan <- result:
+	case <-v.quitChan:
+	}
+}
+
+// validateHandler consumes items to validate from the internal validate channel
+// and returns the result of the validation on the internal result channel. It
+// must be run as a goroutine.
+func (v *txValidator) validateHandler() {
+out:
+	for {
+		select {
+		case txVI := <-v.validateChan:
+			// Ensure the referenced input utxo is available.
+			txIn := txVI.txIn
+			entry, ok := v.utxoSet.Get(txIn.PreviousOutpoint)
+			if !ok {
+				str := fmt.Sprintf("unable to find unspent "+
+					"output %s referenced from "+
+					"transaction %s input %d",
+					txIn.PreviousOutpoint, txVI.tx.ID(),
+					txVI.txInIndex)
+				err := ruleError(ErrMissingTxOut, str)
+				v.sendResult(err)
+				break out
+			}
+
+			// Create a new script engine for the script pair.
+			sigScript := txIn.SignatureScript
+			scriptPubKey := entry.ScriptPubKey()
+			vm, err := txscript.NewEngine(scriptPubKey, txVI.tx.MsgTx(),
+				txVI.txInIndex, v.flags, v.sigCache)
+			if err != nil {
+				str := fmt.Sprintf("failed to parse input "+
+					"%s:%d which references output %s - "+
+					"%s (input script bytes %x, prev "+
+					"output script bytes %x)",
+					txVI.tx.ID(), txVI.txInIndex,
+					txIn.PreviousOutpoint, err, sigScript, scriptPubKey)
+				err := ruleError(ErrScriptMalformed, str)
+				v.sendResult(err)
+				break out
+			}
+
+			// Execute the script pair.
+			if err := vm.Execute(); err != nil {
+				str := fmt.Sprintf("failed to validate input "+
+					"%s:%d which references output %s - "+
+					"%s (input script bytes %x, prev output "+
+					"script bytes %x)",
+					txVI.tx.ID(), txVI.txInIndex,
+					txIn.PreviousOutpoint, err, sigScript, scriptPubKey)
+				err := ruleError(ErrScriptValidation, str)
+				v.sendResult(err)
+				break out
+			}
+
+			// Validation succeeded.
+			v.sendResult(nil)
+
+		case <-v.quitChan:
+			break out
+		}
+	}
+}
+
+// Validate validates the scripts for all of the passed transaction inputs using
+// multiple goroutines.
+func (v *txValidator) Validate(items []*txValidateItem) error {
+	if len(items) == 0 {
+		return nil
+	}
+
+	// Limit the number of goroutines to do script validation based on the
+	// number of processor cores. This helps ensure the system stays
+	// reasonably responsive under heavy load.
+	maxGoRoutines := runtime.NumCPU() * 3
+	if maxGoRoutines <= 0 {
+		maxGoRoutines = 1
+	}
+	if maxGoRoutines > len(items) {
+		maxGoRoutines = len(items)
+	}
+
+	// Start up validation handlers that are used to asynchronously
+	// validate each transaction input.
+	for i := 0; i < maxGoRoutines; i++ {
+		spawn(v.validateHandler)
+	}
+
+	// Validate each of the inputs. The quit channel is closed when any
+	// errors occur so all processing goroutines exit regardless of which
+	// input had the validation error.
+	numInputs := len(items)
+	currentItem := 0
+	processedItems := 0
+	for processedItems < numInputs {
+		// Only send items while there are still items that need to
+		// be processed. The select statement will never select a nil
+		// channel.
+		var validateChan chan *txValidateItem
+		var item *txValidateItem
+		if currentItem < numInputs {
+			validateChan = v.validateChan
+			item = items[currentItem]
+		}
+
+		select {
+		case validateChan <- item:
+			currentItem++
+
+		case err := <-v.resultChan:
+			processedItems++
+			if err != nil {
+				close(v.quitChan)
+				return err
+			}
+		}
+	}
+
+	close(v.quitChan)
+	return nil
+}
+
+// newTxValidator returns a new instance of txValidator to be used for
+// validating transaction scripts asynchronously.
+func newTxValidator(utxoSet UTXOSet, flags txscript.ScriptFlags, sigCache *txscript.SigCache) *txValidator {
+	return &txValidator{
+		validateChan: make(chan *txValidateItem),
+		quitChan:     make(chan struct{}),
+		resultChan:   make(chan error),
+		utxoSet:      utxoSet,
+		sigCache:     sigCache,
+		flags:        flags,
+	}
+}
+
+// ValidateTransactionScripts validates the scripts for the passed transaction
+// using multiple goroutines.
+func ValidateTransactionScripts(tx *util.Tx, utxoSet UTXOSet, flags txscript.ScriptFlags, sigCache *txscript.SigCache) error {
+	// Don't validate coinbase transaction scripts.
+	if tx.IsCoinBase() {
+		return nil
+	}
+
+	// Collect all of the transaction inputs and required information for
+	// validation.
+	txIns := tx.MsgTx().TxIn
+	txValItems := make([]*txValidateItem, 0, len(txIns))
+	for txInIdx, txIn := range txIns {
+		txVI := &txValidateItem{
+			txInIndex: txInIdx,
+			txIn:      txIn,
+			tx:        tx,
+		}
+		txValItems = append(txValItems, txVI)
+	}
+
+	// Validate all of the inputs.
+	validator := newTxValidator(utxoSet, flags, sigCache)
+	return validator.Validate(txValItems)
+}
+
+// checkBlockScripts executes and validates the scripts for all transactions in
+// the passed block using multiple goroutines.
+func checkBlockScripts(block *blockNode, utxoSet UTXOSet, transactions []*util.Tx, scriptFlags txscript.ScriptFlags, sigCache *txscript.SigCache) error {
+	// Collect all of the transaction inputs and required information for
+	// validation for all transactions in the block into a single slice.
+	numInputs := 0
+	for _, tx := range transactions {
+		numInputs += len(tx.MsgTx().TxIn)
+	}
+	txValItems := make([]*txValidateItem, 0, numInputs)
+	for _, tx := range transactions {
+		// Skip coinbase transactions.
+		if tx.IsCoinBase() {
+			continue
+		}
+		for txInIdx, txIn := range tx.MsgTx().TxIn {
+			txVI := &txValidateItem{
+				txInIndex: txInIdx,
+				txIn:      txIn,
+				tx:        tx,
+			}
+			txValItems = append(txValItems, txVI)
+		}
+	}
+
+	// Validate all of the inputs.
+	validator := newTxValidator(utxoSet, scriptFlags, sigCache)
+	start := time.Now()
+	if err := validator.Validate(txValItems); err != nil {
+		return err
+	}
+	elapsed := time.Since(start)
+
+	log.Tracef("block %s took %s to verify", block.hash, elapsed)
+
+	return nil
+}

blockdag/scriptval_test.go

@@ -0,0 +1,55 @@
+// Copyright (c) 2013-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"fmt"
+	"path/filepath"
+	"runtime"
+	"testing"
+
+	"github.com/kaspanet/kaspad/txscript"
+)
+
+// TestCheckBlockScripts ensures that validating the all of the scripts in a
+// known-good block doesn't return an error.
+func TestCheckBlockScripts(t *testing.T) {
+	t.Skip() // TODO: Reactivate this test once we have blocks from testnet.
+	runtime.GOMAXPROCS(runtime.NumCPU())
+
+	testBlockNum := 277647
+	blockDataFile := fmt.Sprintf("%d.dat", testBlockNum)
+	blocks, err := LoadBlocks(filepath.Join("testdata/", blockDataFile))
+	if err != nil {
+		t.Errorf("Error loading file: %v\n", err)
+		return
+	}
+	if len(blocks) > 1 {
+		t.Errorf("The test block file must only have one block in it")
+		return
+	}
+	if len(blocks) == 0 {
+		t.Errorf("The test block file may not be empty")
+		return
+	}
+
+	storeDataFile := fmt.Sprintf("%d.utxostore", testBlockNum)
+	utxoSet, err := loadUTXOSet(storeDataFile)
+	if err != nil {
+		t.Errorf("Error loading txstore: %v\n", err)
+		return
+	}
+
+	node := &blockNode{
+		hash: blocks[0].Hash(),
+	}
+
+	scriptFlags := txscript.ScriptNoFlags
+	err = checkBlockScripts(node, utxoSet, blocks[0].Transactions(), scriptFlags, nil)
+	if err != nil {
+		t.Errorf("Transaction script validation failed: %v\n", err)
+		return
+	}
+}

blockdag/subnetworks.go

@@ -0,0 +1,194 @@
+package blockdag
+
+import (
+	"bytes"
+	"encoding/binary"
+	"fmt"
+	"github.com/pkg/errors"
+
+	"github.com/kaspanet/kaspad/util"
+
+	"github.com/kaspanet/kaspad/database"
+	"github.com/kaspanet/kaspad/util/subnetworkid"
+	"github.com/kaspanet/kaspad/wire"
+)
+
+// SubnetworkStore stores the subnetworks data
+type SubnetworkStore struct {
+	db database.DB
+}
+
+func newSubnetworkStore(db database.DB) *SubnetworkStore {
+	return &SubnetworkStore{
+		db: db,
+	}
+}
+
+// registerSubnetworks scans a list of transactions, singles out
+// subnetwork registry transactions, validates them, and registers a new
+// subnetwork based on it.
+// This function returns an error if one or more transactions are invalid
+func registerSubnetworks(dbTx database.Tx, txs []*util.Tx) error {
+	subnetworkRegistryTxs := make([]*wire.MsgTx, 0)
+	for _, tx := range txs {
+		msgTx := tx.MsgTx()
+
+		if msgTx.SubnetworkID.IsEqual(subnetworkid.SubnetworkIDRegistry) {
+			subnetworkRegistryTxs = append(subnetworkRegistryTxs, msgTx)
+		}
+
+		if subnetworkid.Less(subnetworkid.SubnetworkIDRegistry, &msgTx.SubnetworkID) {
+			// Transactions are ordered by subnetwork, so we can safely assume
+			// that the rest of the transactions will not be subnetwork registry
+			// transactions.
+			break
+		}
+	}
+
+	for _, registryTx := range subnetworkRegistryTxs {
+		subnetworkID, err := TxToSubnetworkID(registryTx)
+		if err != nil {
+			return err
+		}
+		sNet, err := dbGetSubnetwork(dbTx, subnetworkID)
+		if err != nil {
+			return err
+		}
+		if sNet == nil {
+			createdSubnetwork := newSubnetwork(registryTx)
+			err := dbRegisterSubnetwork(dbTx, subnetworkID, createdSubnetwork)
+			if err != nil {
+				return errors.Errorf("failed registering subnetwork"+
+					"for tx '%s': %s", registryTx.TxHash(), err)
+			}
+		}
+	}
+
+	return nil
+}
+
+// validateSubnetworkRegistryTransaction makes sure that a given subnetwork registry
+// transaction is valid. Such a transaction is valid iff:
+// - Its entire payload is a uint64 (8 bytes)
+func validateSubnetworkRegistryTransaction(tx *wire.MsgTx) error {
+	if len(tx.Payload) != 8 {
+		return ruleError(ErrSubnetworkRegistry, fmt.Sprintf("validation failed: subnetwork registry"+
+			"tx '%s' has an invalid payload", tx.TxHash()))
+	}
+
+	return nil
+}
+
+// TxToSubnetworkID creates a subnetwork ID from a subnetwork registry transaction
+func TxToSubnetworkID(tx *wire.MsgTx) (*subnetworkid.SubnetworkID, error) {
+	txHash := tx.TxHash()
+	return subnetworkid.New(util.Hash160(txHash[:]))
+}
+
+// subnetwork returns a registered subnetwork. If the subnetwork does not exist
+// this method returns an error.
+func (s *SubnetworkStore) subnetwork(subnetworkID *subnetworkid.SubnetworkID) (*subnetwork, error) {
+	var sNet *subnetwork
+	var err error
+	dbErr := s.db.View(func(dbTx database.Tx) error {
+		sNet, err = dbGetSubnetwork(dbTx, subnetworkID)
+		return nil
+	})
+	if dbErr != nil {
+		return nil, errors.Errorf("could not retrieve subnetwork '%d': %s", subnetworkID, dbErr)
+	}
+	if err != nil {
+		return nil, errors.Errorf("could not retrieve subnetwork '%d': %s", subnetworkID, err)
+	}
+
+	return sNet, nil
+}
+
+// GasLimit returns the gas limit of a registered subnetwork. If the subnetwork does not
+// exist this method returns an error.
+func (s *SubnetworkStore) GasLimit(subnetworkID *subnetworkid.SubnetworkID) (uint64, error) {
+	sNet, err := s.subnetwork(subnetworkID)
+	if err != nil {
+		return 0, err
+	}
+	if sNet == nil {
+		return 0, errors.Errorf("subnetwork '%s' not found", subnetworkID)
+	}
+
+	return sNet.gasLimit, nil
+}
+
+// dbRegisterSubnetwork stores mappings from ID of the subnetwork to the subnetwork data.
+func dbRegisterSubnetwork(dbTx database.Tx, subnetworkID *subnetworkid.SubnetworkID, network *subnetwork) error {
+	// Serialize the subnetwork
+	serializedSubnetwork, err := serializeSubnetwork(network)
+	if err != nil {
+		return errors.Errorf("failed to serialize sub-netowrk '%s': %s", subnetworkID, err)
+	}
+
+	// Store the subnetwork
+	subnetworksBucket := dbTx.Metadata().Bucket(subnetworksBucketName)
+	err = subnetworksBucket.Put(subnetworkID[:], serializedSubnetwork)
+	if err != nil {
+		return errors.Errorf("failed to write sub-netowrk '%s': %s", subnetworkID, err)
+	}
+
+	return nil
+}
+
+// dbGetSubnetwork returns the subnetwork associated with subnetworkID or nil if the subnetwork was not found.
+func dbGetSubnetwork(dbTx database.Tx, subnetworkID *subnetworkid.SubnetworkID) (*subnetwork, error) {
+	bucket := dbTx.Metadata().Bucket(subnetworksBucketName)
+	serializedSubnetwork := bucket.Get(subnetworkID[:])
+	if serializedSubnetwork == nil {
+		return nil, nil
+	}
+
+	return deserializeSubnetwork(serializedSubnetwork)
+}
+
+type subnetwork struct {
+	gasLimit uint64
+}
+
+func newSubnetwork(tx *wire.MsgTx) *subnetwork {
+	return &subnetwork{
+		gasLimit: ExtractGasLimit(tx),
+	}
+}
+
+// ExtractGasLimit extracts the gas limit from the transaction payload
+func ExtractGasLimit(tx *wire.MsgTx) uint64 {
+	return binary.LittleEndian.Uint64(tx.Payload[:8])
+}
+
+// serializeSubnetwork serializes a subnetwork into the following binary format:
+// | gasLimit (8 bytes) |
+func serializeSubnetwork(sNet *subnetwork) ([]byte, error) {
+	serializedSNet := bytes.NewBuffer(make([]byte, 0, 8))
+
+	// Write the gas limit
+	err := binary.Write(serializedSNet, byteOrder, sNet.gasLimit)
+	if err != nil {
+		return nil, errors.Errorf("failed to serialize subnetwork: %s", err)
+	}
+
+	return serializedSNet.Bytes(), nil
+}
+
+// deserializeSubnetwork deserializes a byte slice into a subnetwork.
+// See serializeSubnetwork for the binary format.
+func deserializeSubnetwork(serializedSNetBytes []byte) (*subnetwork, error) {
+	serializedSNet := bytes.NewBuffer(serializedSNetBytes)
+
+	// Read the gas limit
+	var gasLimit uint64
+	err := binary.Read(serializedSNet, byteOrder, &gasLimit)
+	if err != nil {
+		return nil, errors.Errorf("failed to deserialize subnetwork: %s", err)
+	}
+
+	return &subnetwork{
+		gasLimit: gasLimit,
+	}, nil
+}

blockdag/subnetworks_test.go

@@ -0,0 +1,26 @@
+package blockdag
+
+import (
+	"reflect"
+	"testing"
+)
+
+func TestSerializeSubnetwork(t *testing.T) {
+	sNet := &subnetwork{
+		gasLimit: 1000,
+	}
+
+	serializedSNet, err := serializeSubnetwork(sNet)
+	if err != nil {
+		t.Fatalf("subnetwork serialization unexpectedly failed: %s", err)
+	}
+
+	deserializedSNet, err := deserializeSubnetwork(serializedSNet)
+	if err != nil {
+		t.Fatalf("subnetwork deserialization unexpectedly failed: %s", err)
+	}
+
+	if !reflect.DeepEqual(sNet, deserializedSNet) {
+		t.Errorf("original subnetwork and deserialized subnetwork are not equal")
+	}
+}

blockdag/test_utils.go

@@ -0,0 +1,311 @@
+package blockdag
+
+// This file functions are not considered safe for regular use, and should be used for test purposes only.
+
+import (
+	"compress/bzip2"
+	"encoding/binary"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/pkg/errors"
+	"io"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+
+	"github.com/kaspanet/kaspad/util/subnetworkid"
+
+	"github.com/kaspanet/kaspad/database"
+	_ "github.com/kaspanet/kaspad/database/ffldb" // blank import ffldb so that its init() function runs before tests
+	"github.com/kaspanet/kaspad/txscript"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/wire"
+)
+
+const (
+	// testDbType is the database backend type to use for the tests.
+	testDbType = "ffldb"
+
+	// testDbRoot is the root directory used to create all test databases.
+	testDbRoot = "testdbs"
+
+	// blockDataNet is the expected network in the test block data.
+	blockDataNet = wire.Mainnet
+)
+
+// isSupportedDbType returns whether or not the passed database type is
+// currently supported.
+func isSupportedDbType(dbType string) bool {
+	supportedDrivers := database.SupportedDrivers()
+	for _, driver := range supportedDrivers {
+		if dbType == driver {
+			return true
+		}
+	}
+
+	return false
+}
+
+// FileExists returns whether or not the named file or directory exists.
+func FileExists(name string) bool {
+	if _, err := os.Stat(name); err != nil {
+		if os.IsNotExist(err) {
+			return false
+		}
+	}
+	return true
+}
+
+// DAGSetup is used to create a new db and DAG instance with the genesis
+// block already inserted. In addition to the new DAG instance, it returns
+// a teardown function the caller should invoke when done testing to clean up.
+func DAGSetup(dbName string, config Config) (*BlockDAG, func(), error) {
+	if !isSupportedDbType(testDbType) {
+		return nil, nil, errors.Errorf("unsupported db type %s", testDbType)
+	}
+
+	var teardown func()
+
+	// To make sure that the teardown function is not called before any goroutines finished to run -
+	// overwrite `spawn` to count the number of running goroutines
+	spawnWaitGroup := sync.WaitGroup{}
+	realSpawn := spawn
+	spawn = func(f func()) {
+		spawnWaitGroup.Add(1)
+		realSpawn(func() {
+			f()
+			spawnWaitGroup.Done()
+		})
+	}
+
+	if config.DB == nil {
+		// Create the root directory for test databases.
+		if !FileExists(testDbRoot) {
+			if err := os.MkdirAll(testDbRoot, 0700); err != nil {
+				err := errors.Errorf("unable to create test db "+
+					"root: %s", err)
+				return nil, nil, err
+			}
+		}
+
+		dbPath := filepath.Join(testDbRoot, dbName)
+		_ = os.RemoveAll(dbPath)
+		var err error
+		config.DB, err = database.Create(testDbType, dbPath, blockDataNet)
+		if err != nil {
+			return nil, nil, errors.Errorf("error creating db: %s", err)
+		}
+
+		// Setup a teardown function for cleaning up. This function is
+		// returned to the caller to be invoked when it is done testing.
+		teardown = func() {
+			spawnWaitGroup.Wait()
+			spawn = realSpawn
+			config.DB.Close()
+			os.RemoveAll(dbPath)
+			os.RemoveAll(testDbRoot)
+		}
+	} else {
+		teardown = func() {
+			spawnWaitGroup.Wait()
+			spawn = realSpawn
+			config.DB.Close()
+		}
+	}
+
+	config.TimeSource = NewMedianTime()
+	config.SigCache = txscript.NewSigCache(1000)
+
+	// Create the DAG instance.
+	dag, err := New(&config)
+	if err != nil {
+		teardown()
+		err := errors.Errorf("failed to create dag instance: %s", err)
+		return nil, nil, err
+	}
+	return dag, teardown, nil
+}
+
+// OpTrueScript is script returning TRUE
+var OpTrueScript = []byte{txscript.OpTrue}
+
+type txSubnetworkData struct {
+	subnetworkID *subnetworkid.SubnetworkID
+	Gas          uint64
+	Payload      []byte
+}
+
+func createTxForTest(numInputs uint32, numOutputs uint32, outputValue uint64, subnetworkData *txSubnetworkData) *wire.MsgTx {
+	txIns := []*wire.TxIn{}
+	txOuts := []*wire.TxOut{}
+
+	for i := uint32(0); i < numInputs; i++ {
+		txIns = append(txIns, &wire.TxIn{
+			PreviousOutpoint: *wire.NewOutpoint(&daghash.TxID{}, i),
+			SignatureScript:  []byte{},
+			Sequence:         wire.MaxTxInSequenceNum,
+		})
+	}
+
+	for i := uint32(0); i < numOutputs; i++ {
+		txOuts = append(txOuts, &wire.TxOut{
+			ScriptPubKey: OpTrueScript,
+			Value:        outputValue,
+		})
+	}
+
+	if subnetworkData != nil {
+		return wire.NewSubnetworkMsgTx(wire.TxVersion, txIns, txOuts, subnetworkData.subnetworkID, subnetworkData.Gas, subnetworkData.Payload)
+	}
+
+	return wire.NewNativeMsgTx(wire.TxVersion, txIns, txOuts)
+}
+
+// VirtualForTest is an exported version for virtualBlock, so that it can be returned by exported test_util methods
+type VirtualForTest *virtualBlock
+
+// SetVirtualForTest replaces the dag's virtual block. This function is used for test purposes only
+func SetVirtualForTest(dag *BlockDAG, virtual VirtualForTest) VirtualForTest {
+	oldVirtual := dag.virtual
+	dag.virtual = virtual
+	return VirtualForTest(oldVirtual)
+}
+
+// GetVirtualFromParentsForTest generates a virtual block with the given parents.
+func GetVirtualFromParentsForTest(dag *BlockDAG, parentHashes []*daghash.Hash) (VirtualForTest, error) {
+	parents := newSet()
+	for _, hash := range parentHashes {
+		parent := dag.index.LookupNode(hash)
+		if parent == nil {
+			return nil, errors.Errorf("GetVirtualFromParentsForTest: didn't found node for hash %s", hash)
+		}
+		parents.add(parent)
+	}
+	virtual := newVirtualBlock(dag, parents)
+
+	pastUTXO, _, err := dag.pastUTXO(&virtual.blockNode)
+	if err != nil {
+		return nil, err
+	}
+	diffUTXO := pastUTXO.clone().(*DiffUTXOSet)
+	err = diffUTXO.meldToBase()
+	if err != nil {
+		return nil, err
+	}
+	virtual.utxoSet = diffUTXO.base
+
+	return VirtualForTest(virtual), nil
+}
+
+// LoadBlocks reads files containing kaspa gzipped block data from disk
+// and returns them as an array of util.Block.
+func LoadBlocks(filename string) (blocks []*util.Block, err error) {
+	var network = wire.Mainnet
+	var dr io.Reader
+	var fi io.ReadCloser
+
+	fi, err = os.Open(filename)
+	if err != nil {
+		return
+	}
+
+	if strings.HasSuffix(filename, ".bz2") {
+		dr = bzip2.NewReader(fi)
+	} else {
+		dr = fi
+	}
+	defer fi.Close()
+
+	var block *util.Block
+
+	err = nil
+	for height := uint64(0); err == nil; height++ {
+		var rintbuf uint32
+		err = binary.Read(dr, binary.LittleEndian, &rintbuf)
+		if err == io.EOF {
+			// hit end of file at expected offset: no warning
+			height--
+			err = nil
+			break
+		}
+		if err != nil {
+			break
+		}
+		if rintbuf != uint32(network) {
+			break
+		}
+		err = binary.Read(dr, binary.LittleEndian, &rintbuf)
+		blocklen := rintbuf
+
+		rbytes := make([]byte, blocklen)
+
+		// read block
+		dr.Read(rbytes)
+
+		block, err = util.NewBlockFromBytes(rbytes)
+		if err != nil {
+			return
+		}
+		blocks = append(blocks, block)
+	}
+
+	return
+}
+
+// opTrueAddress returns an address pointing to a P2SH anyone-can-spend script
+func opTrueAddress(prefix util.Bech32Prefix) (util.Address, error) {
+	return util.NewAddressScriptHash(OpTrueScript, prefix)
+}
+
+// PrepareBlockForTest generates a block with the proper merkle roots, coinbase transaction etc. This function is used for test purposes only
+func PrepareBlockForTest(dag *BlockDAG, parentHashes []*daghash.Hash, transactions []*wire.MsgTx) (*wire.MsgBlock, error) {
+	newVirtual, err := GetVirtualFromParentsForTest(dag, parentHashes)
+	if err != nil {
+		return nil, err
+	}
+	oldVirtual := SetVirtualForTest(dag, newVirtual)
+	defer SetVirtualForTest(dag, oldVirtual)
+
+	OpTrueAddr, err := opTrueAddress(dag.dagParams.Prefix)
+	if err != nil {
+		return nil, err
+	}
+
+	blockTransactions := make([]*util.Tx, len(transactions)+1)
+
+	extraNonce := generateDeterministicExtraNonceForTest()
+	coinbasePayloadExtraData, err := CoinbasePayloadExtraData(extraNonce, "")
+	if err != nil {
+		return nil, err
+	}
+
+	blockTransactions[0], err = dag.NextCoinbaseFromAddress(OpTrueAddr, coinbasePayloadExtraData)
+	if err != nil {
+		return nil, err
+	}
+
+	for i, tx := range transactions {
+		blockTransactions[i+1] = util.NewTx(tx)
+	}
+
+	block, err := dag.BlockForMining(blockTransactions)
+	if err != nil {
+		return nil, err
+	}
+	block.Header.Timestamp = dag.NextBlockMinimumTime()
+	block.Header.Bits = dag.NextRequiredDifficulty(block.Header.Timestamp)
+
+	return block, nil
+}
+
+// generateDeterministicExtraNonceForTest returns a unique deterministic extra nonce for coinbase data, in order to create unique coinbase transactions.
+func generateDeterministicExtraNonceForTest() uint64 {
+	extraNonceForTest++
+	return extraNonceForTest
+}
+
+func resetExtraNonceForTest() {
+	extraNonceForTest = 0
+}
+
+var extraNonceForTest = uint64(0)

blockdag/test_utils_test.go

@@ -0,0 +1,14 @@
+package blockdag
+
+import (
+	"testing"
+)
+
+func TestIsSupportedDbType(t *testing.T) {
+	if !isSupportedDbType("ffldb") {
+		t.Errorf("ffldb should be a supported DB driver")
+	}
+	if isSupportedDbType("madeUpDb") {
+		t.Errorf("madeUpDb should not be a supported DB driver")
+	}
+}

blockdag/testdata/reorgtest.hex

@@ -0,0 +1,180 @@
+File path:  reorgTest/blk_0_to_4.dat
+
+Block 0:
+   f9beb4d9
+   1d010000
+   
+   01000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
+   00000000 3ba3edfd 7a7b12b2 7ac72c3e 67768f61 7fc81bc3 888a5132 3a9fb8aa 
+   4b1e5e4a 29ab5f49 ffff001d 1dac2b7c 
+   01
+   
+   01000000 01000000 00000000 00000000 00000000 00000000 00000000 00000000 
+   00000000 00ffffff ff4d04ff ff001d01 04455468 65205469 6d657320 30332f4a 
+   616e2f32 30303920 4368616e 63656c6c 6f72206f 6e206272 696e6b20 6f662073 
+   65636f6e 64206261 696c6f75 7420666f 72206261 6e6b73ff ffffff01 00f2052a 
+   01000000 43410467 8afdb0fe 55482719 67f1a671 30b7105c d6a828e0 3909a679 
+   62e0ea1f 61deb649 f6bc3f4c ef38c4f3 5504e51e c112de5c 384df7ba 0b8d578a 
+   4c702b6b f11d5fac 00000000 
+Block 1:
+   f9beb4d9
+   d4000000
+   
+   01000000 6fe28c0a b6f1b372 c1a6a246 ae63f74f 931e8365 e15a089c 68d61900 
+   00000000 3bbd67ad e98fbbb7 0718cd80 f9e9acf9 3b5fae91 7bb2b41d 4c3bb82c 
+   77725ca5 81ad5f49 ffff001d 44e69904 
+   01
+   
+   01000000 01000000 00000000 00000000 00000000 00000000 00000000 00000000 
+   00000000 00ffffff ff04722f 2e2bffff ffff0100 f2052a01 00000043 41046868 
+   0737c76d abb801cb 2204f57d be4e4579 e4f710cd 67dc1b42 27592c81 e9b5cf02 
+   b5ac9e8b 4c9f49be 5251056b 6a6d011e 4c37f6b6 d17ede6b 55faa235 19e2ac00 
+   000000 
+Block 2:
+   f9beb4d9
+   95010000
+   
+   01000000 13ca7940 4c11c63e ca906bbd f190b751 2872b857 1b5143ae e8cb5737 
+   00000000 fc07c983 d7391736 0aeda657 29d0d4d3 2533eb84 76ee9d64 aa27538f 
+   9b4fc00a d9af5f49 ffff001d 630bea22 
+   02
+   
+   01000000 01000000 00000000 00000000 00000000 00000000 00000000 00000000 
+   00000000 00ffffff ff04eb96 14e5ffff ffff0100 f2052a01 00000043 41046868 
+   0737c76d abb801cb 2204f57d be4e4579 e4f710cd 67dc1b42 27592c81 e9b5cf02 
+   b5ac9e8b 4c9f49be 5251056b 6a6d011e 4c37f6b6 d17ede6b 55faa235 19e2ac00 
+   000000 
+   
+   01000000 0163451d 1002611c 1388d5ba 4ddfdf99 196a86b5 990fb5b0 dc786207 
+   4fdcb8ee d2000000 004a4930 46022100 3dde52c6 5e339f45 7fe1015e 70eed208 
+   872eb71e dd484c07 206b190e cb2ec3f8 02210011 c78dcfd0 3d43fa63 61242a33 
+   6291ba2a 8c1ef5bc d5472126 2468f2bf 8dee4d01 ffffffff 0200ca9a 3b000000 
+   001976a9 14cb2abd e8bccacc 32e893df 3a054b9e f7f227a4 ce88ac00 286bee00 
+   00000019 76a914ee 26c56fc1 d942be8d 7a24b2a1 001dd894 69398088 ac000000 
+   00 
+Block 3:
+   f9beb4d9
+   96020000
+   
+   01000000 7d338254 0506faab 0d4cf179 45dda023 49db51f9 6233f24c 28002258 
+   00000000 4806fe80 bf85931b 882ea645 77ca5a03 22bb8af2 3f277b20 55f160cd 
+   972c8e8b 31b25f49 ffff001d e8f0c653 
+   03
+   
+   01000000 01000000 00000000 00000000 00000000 00000000 00000000 00000000 
+   00000000 00ffffff ff044abd 8159ffff ffff0100 f2052a01 00000043 4104b95c 
+   249d84f4 17e3e395 a1274254 28b54067 1cc15881 eb828c17 b722a53f c599e21c 
+   a5e56c90 f340988d 3933acc7 6beb832f d64cab07 8ddf3ce7 32923031 d1a8ac00 
+   000000 
+   
+   01000000 01f287b5 e067e1cf 80f7da8a f89917b5 505094db d82412d9 35b665eb 
+   bad253d3 77010000 008c4930 46022100 96ee0d02 b35fd61e 4960b44f f396f67e 
+   01fe17f9 de4e0c17 b6a963bd ab2b50a6 02210034 920d4daa 7e9f8abe 5675c931 
+   495809f9 0b9c1189 d05fbaf1 dd6696a5 b0d8f301 41046868 0737c76d abb801cb 
+   2204f57d be4e4579 e4f710cd 67dc1b42 27592c81 e9b5cf02 b5ac9e8b 4c9f49be 
+   5251056b 6a6d011e 4c37f6b6 d17ede6b 55faa235 19e2ffff ffff0100 286bee00 
+   00000019 76a914c5 22664fb0 e55cdc5c 0cea73b4 aad97ec8 34323288 ac000000 
+   00 
+   
+   01000000 01f287b5 e067e1cf 80f7da8a f89917b5 505094db d82412d9 35b665eb 
+   bad253d3 77000000 008c4930 46022100 b08b922a c4bde411 1c229f92 9fe6eb6a 
+   50161f98 1f4cf47e a9214d35 bf74d380 022100d2 f6640327 e677a1e1 cc474991 
+   b9a48ba5 bd1e0c94 d1c8df49 f7b0193b 7ea4fa01 4104b95c 249d84f4 17e3e395 
+   a1274254 28b54067 1cc15881 eb828c17 b722a53f c599e21c a5e56c90 f340988d 
+   3933acc7 6beb832f d64cab07 8ddf3ce7 32923031 d1a8ffff ffff0100 ca9a3b00 
+   00000019 76a914c5 22664fb0 e55cdc5c 0cea73b4 aad97ec8 34323288 ac000000 
+   00 
+
+Block 4:
+   f9beb4d9
+   73010000
+   
+   01000000 5da36499 06f35e09 9be42a1d 87b6dd42 11bc1400 6c220694 0807eaae 
+   00000000 48eeeaed 2d9d8522 e6201173 743823fd 4b87cd8a ca8e6408 ec75ca38 
+   302c2ff0 89b45f49 ffff001d 00530839 
+   02
+   
+   01000000 01000000 00000000 00000000 00000000 00000000 00000000 00000000 
+   00000000 00ffffff ff04d41d 2213ffff ffff0100 f2052a01 00000043 4104678a 
+   fdb0fe55 48271967 f1a67130 b7105cd6 a828e039 09a67962 e0ea1f61 deb649f6 
+   bc3f4cef 38c4f355 04e51ec1 12de5c38 4df7ba0b 8d578a4c 702b6bf1 1d5fac00 
+   000000 
+   
+   01000000 0163451d 1002611c 1388d5ba 4ddfdf99 196a86b5 990fb5b0 dc786207 
+   4fdcb8ee d2000000 004a4930 46022100 8c8fd57b 48762135 8d8f3e69 19f33e08 
+   804736ff 83db47aa 248512e2 6df9b8ba 022100b0 c59e5ee7 bfcbfcd1 a4d83da9 
+   55fb260e fda7f42a 25522625 a3d6f2d9 1174a701 ffffffff 0100f205 2a010000 
+   001976a9 14c52266 4fb0e55c dc5c0cea 73b4aad9 7ec83432 3288ac00 000000 
+
+File path:  reorgTest/blk_3A.dat
+Block 3A:
+   f9beb4d9
+   96020000
+   
+   01000000 7d338254 0506faab 0d4cf179 45dda023 49db51f9 6233f24c 28002258 
+   00000000 5a15f573 1177a353 bdca7aab 20e16624 dfe90adc 70accadc 68016732 
+   302c20a7 31b25f49 ffff001d 6a901440 
+   03
+   
+   01000000 01000000 00000000 00000000 00000000 00000000 00000000 00000000 
+   00000000 00ffffff ff04ad1b e7d5ffff ffff0100 f2052a01 00000043 4104ed83 
+   704c95d8 29046f1a c2780621 1132102c 34e9ac7f fa1b7111 0658e5b9 d1bdedc4 
+   16f5cefc 1db0625c d0c75de8 192d2b59 2d7e3b00 bcfb4a0e 860d880f d1fcac00 
+   000000 
+   
+   01000000 01f287b5 e067e1cf 80f7da8a f89917b5 505094db d82412d9 35b665eb 
+   bad253d3 77010000 008c4930 46022100 96ee0d02 b35fd61e 4960b44f f396f67e 
+   01fe17f9 de4e0c17 b6a963bd ab2b50a6 02210034 920d4daa 7e9f8abe 5675c931 
+   495809f9 0b9c1189 d05fbaf1 dd6696a5 b0d8f301 41046868 0737c76d abb801cb 
+   2204f57d be4e4579 e4f710cd 67dc1b42 27592c81 e9b5cf02 b5ac9e8b 4c9f49be 
+   5251056b 6a6d011e 4c37f6b6 d17ede6b 55faa235 19e2ffff ffff0100 286bee00 
+   00000019 76a914c5 22664fb0 e55cdc5c 0cea73b4 aad97ec8 34323288 ac000000 
+   00 
+   
+   01000000 01f287b5 e067e1cf 80f7da8a f89917b5 505094db d82412d9 35b665eb 
+   bad253d3 77000000 008c4930 46022100 9cc67ddd aa6f592a 6b2babd4 d6ff954f 
+   25a784cf 4fe4bb13 afb9f49b 08955119 022100a2 d99545b7 94080757 fcf2b563 
+   f2e91287 86332f46 0ec6b90f f085fb28 41a69701 4104b95c 249d84f4 17e3e395 
+   a1274254 28b54067 1cc15881 eb828c17 b722a53f c599e21c a5e56c90 f340988d 
+   3933acc7 6beb832f d64cab07 8ddf3ce7 32923031 d1a8ffff ffff0100 ca9a3b00 
+   00000019 76a914ee 26c56fc1 d942be8d 7a24b2a1 001dd894 69398088 ac000000 
+   00 
+
+File path:  reorgTest/blk_4A.dat
+Block 4A:
+   f9beb4d9
+   d4000000
+   
+   01000000 aae77468 2205667d 4f413a58 47cc8fe8 9795f1d5 645d5b24 1daf3c92 
+   00000000 361c9cde a09637a0 d0c05c3b 4e7a5d91 9edb184a 0a4c7633 d92e2ddd 
+   f04cb854 89b45f49 ffff001d 9e9aa1e8 
+   01
+   
+   01000000 01000000 00000000 00000000 00000000 00000000 00000000 00000000 
+   00000000 00ffffff ff0401b8 f3eaffff ffff0100 f2052a01 00000043 4104678a 
+   fdb0fe55 48271967 f1a67130 b7105cd6 a828e039 09a67962 e0ea1f61 deb649f6 
+   bc3f4cef 38c4f355 04e51ec1 12de5c38 4df7ba0b 8d578a4c 702b6bf1 1d5fac00 
+   000000 
+
+File path:  reorgTest/blk_5A.dat
+Block 5A:
+   f9beb4d9
+   73010000
+   
+   01000000 ebc7d0de 9c31a71b 7f41d275 2c080ba4 11e1854b d45cb2cf 8c1e4624 
+   00000000 a607774b 79b8eb50 b52a5a32 c1754281 ec67f626 9561df28 57d1fe6a 
+   ea82c696 e1b65f49 ffff001d 4a263577 
+   02
+   
+   01000000 01000000 00000000 00000000 00000000 00000000 00000000 00000000 
+   00000000 00ffffff ff049971 0c7dffff ffff0100 f2052a01 00000043 4104678a 
+   fdb0fe55 48271967 f1a67130 b7105cd6 a828e039 09a67962 e0ea1f61 deb649f6 
+   bc3f4cef 38c4f355 04e51ec1 12de5c38 4df7ba0b 8d578a4c 702b6bf1 1d5fac00 
+   000000 
+   
+   01000000 0163451d 1002611c 1388d5ba 4ddfdf99 196a86b5 990fb5b0 dc786207 
+   4fdcb8ee d2000000 004a4930 46022100 8c8fd57b 48762135 8d8f3e69 19f33e08 
+   804736ff 83db47aa 248512e2 6df9b8ba 022100b0 c59e5ee7 bfcbfcd1 a4d83da9 
+   55fb260e fda7f42a 25522625 a3d6f2d9 1174a701 ffffffff 0100f205 2a010000 
+   001976a9 14c52266 4fb0e55c dc5c0cea 73b4aad9 7ec83432 3288ac00 000000 
+

blockdag/thresholdstate.go

@@ -0,0 +1,358 @@
+// Copyright (c) 2016-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"fmt"
+
+	"github.com/kaspanet/kaspad/util/daghash"
+)
+
+// ThresholdState define the various threshold states used when voting on
+// consensus changes.
+type ThresholdState byte
+
+// These constants are used to identify specific threshold states.
+const (
+	// ThresholdDefined is the first state for each deployment and is the
+	// state for the genesis block has by definition for all deployments.
+	ThresholdDefined ThresholdState = iota
+
+	// ThresholdStarted is the state for a deployment once its start time
+	// has been reached.
+	ThresholdStarted
+
+	// ThresholdLockedIn is the state for a deployment during the retarget
+	// period which is after the ThresholdStarted state period and the
+	// number of blocks that have voted for the deployment equal or exceed
+	// the required number of votes for the deployment.
+	ThresholdLockedIn
+
+	// ThresholdActive is the state for a deployment for all blocks after a
+	// retarget period in which the deployment was in the ThresholdLockedIn
+	// state.
+	ThresholdActive
+
+	// ThresholdFailed is the state for a deployment once its expiration
+	// time has been reached and it did not reach the ThresholdLockedIn
+	// state.
+	ThresholdFailed
+
+	// numThresholdsStates is the maximum number of threshold states used in
+	// tests.
+	numThresholdsStates
+)
+
+// thresholdStateStrings is a map of ThresholdState values back to their
+// constant names for pretty printing.
+var thresholdStateStrings = map[ThresholdState]string{
+	ThresholdDefined:  "ThresholdDefined",
+	ThresholdStarted:  "ThresholdStarted",
+	ThresholdLockedIn: "ThresholdLockedIn",
+	ThresholdActive:   "ThresholdActive",
+	ThresholdFailed:   "ThresholdFailed",
+}
+
+// String returns the ThresholdState as a human-readable name.
+func (t ThresholdState) String() string {
+	if s := thresholdStateStrings[t]; s != "" {
+		return s
+	}
+	return fmt.Sprintf("Unknown ThresholdState (%d)", int(t))
+}
+
+// thresholdConditionChecker provides a generic interface that is invoked to
+// determine when a consensus rule change threshold should be changed.
+type thresholdConditionChecker interface {
+	// BeginTime returns the unix timestamp for the median block time after
+	// which voting on a rule change starts (at the next window).
+	BeginTime() uint64
+
+	// EndTime returns the unix timestamp for the median block time after
+	// which an attempted rule change fails if it has not already been
+	// locked in or activated.
+	EndTime() uint64
+
+	// RuleChangeActivationThreshold is the number of blocks for which the
+	// condition must be true in order to lock in a rule change.
+	RuleChangeActivationThreshold() uint64
+
+	// MinerConfirmationWindow is the number of blocks in each threshold
+	// state retarget window.
+	MinerConfirmationWindow() uint64
+
+	// Condition returns whether or not the rule change activation condition
+	// has been met. This typically involves checking whether or not the
+	// bit associated with the condition is set, but can be more complex as
+	// needed.
+	Condition(*blockNode) (bool, error)
+}
+
+// thresholdStateCache provides a type to cache the threshold states of each
+// threshold window for a set of IDs.
+type thresholdStateCache struct {
+	entries map[daghash.Hash]ThresholdState
+}
+
+// Lookup returns the threshold state associated with the given hash along with
+// a boolean that indicates whether or not it is valid.
+func (c *thresholdStateCache) Lookup(hash *daghash.Hash) (ThresholdState, bool) {
+	state, ok := c.entries[*hash]
+	return state, ok
+}
+
+// Update updates the cache to contain the provided hash to threshold state
+// mapping.
+func (c *thresholdStateCache) Update(hash *daghash.Hash, state ThresholdState) {
+	c.entries[*hash] = state
+}
+
+// newThresholdCaches returns a new array of caches to be used when calculating
+// threshold states.
+func newThresholdCaches(numCaches uint32) []thresholdStateCache {
+	caches := make([]thresholdStateCache, numCaches)
+	for i := 0; i < len(caches); i++ {
+		caches[i] = thresholdStateCache{
+			entries: make(map[daghash.Hash]ThresholdState),
+		}
+	}
+	return caches
+}
+
+// thresholdState returns the current rule change threshold state for the block
+// AFTER the given node and deployment ID. The cache is used to ensure the
+// threshold states for previous windows are only calculated once.
+//
+// This function MUST be called with the DAG state lock held (for writes).
+func (dag *BlockDAG) thresholdState(prevNode *blockNode, checker thresholdConditionChecker, cache *thresholdStateCache) (ThresholdState, error) {
+	// The threshold state for the window that contains the genesis block is
+	// defined by definition.
+	confirmationWindow := checker.MinerConfirmationWindow()
+	if prevNode == nil || (prevNode.blueScore+1) < confirmationWindow {
+		return ThresholdDefined, nil
+	}
+
+	// Get the ancestor that is the last block of the previous confirmation
+	// window in order to get its threshold state. This can be done because
+	// the state is the same for all blocks within a given window.
+	prevNode = prevNode.SelectedAncestor(prevNode.blueScore -
+		(prevNode.blueScore+1)%confirmationWindow)
+
+	// Iterate backwards through each of the previous confirmation windows
+	// to find the most recently cached threshold state.
+	var neededStates []*blockNode
+	for prevNode != nil {
+		// Nothing more to do if the state of the block is already
+		// cached.
+		if _, ok := cache.Lookup(prevNode.hash); ok {
+			break
+		}
+
+		// The start and expiration times are based on the median block
+		// time, so calculate it now.
+		medianTime := prevNode.PastMedianTime(dag)
+
+		// The state is simply defined if the start time hasn't been
+		// been reached yet.
+		if uint64(medianTime.Unix()) < checker.BeginTime() {
+			cache.Update(prevNode.hash, ThresholdDefined)
+			break
+		}
+
+		// Add this node to the list of nodes that need the state
+		// calculated and cached.
+		neededStates = append(neededStates, prevNode)
+
+		// Get the ancestor that is the last block of the previous
+		// confirmation window.
+		prevNode = prevNode.RelativeAncestor(confirmationWindow)
+	}
+
+	// Start with the threshold state for the most recent confirmation
+	// window that has a cached state.
+	state := ThresholdDefined
+	if prevNode != nil {
+		var ok bool
+		state, ok = cache.Lookup(prevNode.hash)
+		if !ok {
+			return ThresholdFailed, AssertError(fmt.Sprintf(
+				"thresholdState: cache lookup failed for %s",
+				prevNode.hash))
+		}
+	}
+
+	// Since each threshold state depends on the state of the previous
+	// window, iterate starting from the oldest unknown window.
+	for neededNum := len(neededStates) - 1; neededNum >= 0; neededNum-- {
+		prevNode := neededStates[neededNum]
+
+		switch state {
+		case ThresholdDefined:
+			// The deployment of the rule change fails if it expires
+			// before it is accepted and locked in.
+			medianTime := prevNode.PastMedianTime(dag)
+			medianTimeUnix := uint64(medianTime.Unix())
+			if medianTimeUnix >= checker.EndTime() {
+				state = ThresholdFailed
+				break
+			}
+
+			// The state for the rule moves to the started state
+			// once its start time has been reached (and it hasn't
+			// already expired per the above).
+			if medianTimeUnix >= checker.BeginTime() {
+				state = ThresholdStarted
+			}
+
+		case ThresholdStarted:
+			// The deployment of the rule change fails if it expires
+			// before it is accepted and locked in.
+			medianTime := prevNode.PastMedianTime(dag)
+			if uint64(medianTime.Unix()) >= checker.EndTime() {
+				state = ThresholdFailed
+				break
+			}
+
+			// At this point, the rule change is still being voted
+			// on by the miners, so iterate backwards through the
+			// confirmation window to count all of the votes in it.
+			var count uint64
+			windowNodes := make([]*blockNode, 0, confirmationWindow)
+			windowNodes = append(windowNodes, prevNode)
+			windowNodes = append(windowNodes, blueBlockWindow(prevNode, confirmationWindow-1)...)
+			for _, current := range windowNodes {
+				condition, err := checker.Condition(current)
+				if err != nil {
+					return ThresholdFailed, err
+				}
+				if condition {
+					count++
+				}
+
+				// Get the previous block node.
+				current = current.selectedParent
+			}
+
+			// The state is locked in if the number of blocks in the
+			// period that voted for the rule change meets the
+			// activation threshold.
+			if count >= checker.RuleChangeActivationThreshold() {
+				state = ThresholdLockedIn
+			}
+
+		case ThresholdLockedIn:
+			// The new rule becomes active when its previous state
+			// was locked in.
+			state = ThresholdActive
+
+			// Nothing to do if the previous state is active or failed since
+			// they are both terminal states.
+		case ThresholdActive:
+		case ThresholdFailed:
+		}
+
+		// Update the cache to avoid recalculating the state in the
+		// future.
+		cache.Update(prevNode.hash, state)
+	}
+
+	return state, nil
+}
+
+// ThresholdState returns the current rule change threshold state of the given
+// deployment ID for the block AFTER the blueScore of the current DAG.
+//
+// This function is safe for concurrent access.
+func (dag *BlockDAG) ThresholdState(deploymentID uint32) (ThresholdState, error) {
+	dag.dagLock.Lock()
+	defer dag.dagLock.Unlock()
+	state, err := dag.deploymentState(dag.selectedTip(), deploymentID)
+
+	return state, err
+}
+
+// IsDeploymentActive returns true if the target deploymentID is active, and
+// false otherwise.
+//
+// This function is safe for concurrent access.
+func (dag *BlockDAG) IsDeploymentActive(deploymentID uint32) (bool, error) {
+	dag.dagLock.Lock()
+	defer dag.dagLock.Unlock()
+	state, err := dag.deploymentState(dag.selectedTip(), deploymentID)
+	if err != nil {
+		return false, err
+	}
+
+	return state == ThresholdActive, nil
+}
+
+// deploymentState returns the current rule change threshold for a given
+// deploymentID. The threshold is evaluated from the point of view of the block
+// node passed in as the first argument to this method.
+//
+// It is important to note that, as the variable name indicates, this function
+// expects the block node prior to the block for which the deployment state is
+// desired. In other words, the returned deployment state is for the block
+// AFTER the passed node.
+//
+// This function MUST be called with the DAG state lock held (for writes).
+func (dag *BlockDAG) deploymentState(prevNode *blockNode, deploymentID uint32) (ThresholdState, error) {
+	if deploymentID > uint32(len(dag.dagParams.Deployments)) {
+		return ThresholdFailed, DeploymentError(deploymentID)
+	}
+
+	deployment := &dag.dagParams.Deployments[deploymentID]
+	checker := deploymentChecker{deployment: deployment, dag: dag}
+	cache := &dag.deploymentCaches[deploymentID]
+
+	return dag.thresholdState(prevNode, checker, cache)
+}
+
+// initThresholdCaches initializes the threshold state caches for each warning
+// bit and defined deployment and provides warnings if the DAG is current per
+// the warnUnknownVersions and warnUnknownRuleActivations functions.
+func (dag *BlockDAG) initThresholdCaches() error {
+	// Initialize the warning and deployment caches by calculating the
+	// threshold state for each of them. This will ensure the caches are
+	// populated and any states that needed to be recalculated due to
+	// definition changes is done now.
+	prevNode := dag.selectedTip().selectedParent
+	for bit := uint32(0); bit < vbNumBits; bit++ {
+		checker := bitConditionChecker{bit: bit, dag: dag}
+		cache := &dag.warningCaches[bit]
+		_, err := dag.thresholdState(prevNode, checker, cache)
+		if err != nil {
+			return err
+		}
+	}
+	for id := 0; id < len(dag.dagParams.Deployments); id++ {
+		deployment := &dag.dagParams.Deployments[id]
+		cache := &dag.deploymentCaches[id]
+		checker := deploymentChecker{deployment: deployment, dag: dag}
+		_, err := dag.thresholdState(prevNode, checker, cache)
+		if err != nil {
+			return err
+		}
+	}
+
+	// No warnings about unknown rules or versions until the DAG is
+	// current.
+	if dag.isCurrent() {
+		// Warn if a high enough percentage of the last blocks have
+		// unexpected versions.
+		bestNode := dag.selectedTip()
+		if err := dag.warnUnknownVersions(bestNode); err != nil {
+			return err
+		}
+
+		// Warn if any unknown new rules are either about to activate or
+		// have already been activated.
+		if err := dag.warnUnknownRuleActivations(bestNode); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}

blockdag/thresholdstate_test.go

@@ -0,0 +1,134 @@
+// Copyright (c) 2016 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"testing"
+
+	"github.com/kaspanet/kaspad/util/daghash"
+)
+
+// TestThresholdStateStringer tests the stringized output for the
+// ThresholdState type.
+func TestThresholdStateStringer(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		in   ThresholdState
+		want string
+	}{
+		{ThresholdDefined, "ThresholdDefined"},
+		{ThresholdStarted, "ThresholdStarted"},
+		{ThresholdLockedIn, "ThresholdLockedIn"},
+		{ThresholdActive, "ThresholdActive"},
+		{ThresholdFailed, "ThresholdFailed"},
+		{0xff, "Unknown ThresholdState (255)"},
+	}
+
+	// Detect additional threshold states that don't have the stringer added.
+	if len(tests)-1 != int(numThresholdsStates) {
+		t.Errorf("It appears a threshold statewas added without " +
+			"adding an associated stringer test")
+	}
+
+	t.Logf("Running %d tests", len(tests))
+	for i, test := range tests {
+		result := test.in.String()
+		if result != test.want {
+			t.Errorf("String #%d\n got: %s want: %s", i, result,
+				test.want)
+			continue
+		}
+	}
+}
+
+// TestThresholdStateCache ensure the threshold state cache works as intended
+// including adding entries, updating existing entries, and flushing.
+func TestThresholdStateCache(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name       string
+		numEntries int
+		state      ThresholdState
+	}{
+		{name: "2 entries defined", numEntries: 2, state: ThresholdDefined},
+		{name: "7 entries started", numEntries: 7, state: ThresholdStarted},
+		{name: "10 entries active", numEntries: 10, state: ThresholdActive},
+		{name: "5 entries locked in", numEntries: 5, state: ThresholdLockedIn},
+		{name: "3 entries failed", numEntries: 3, state: ThresholdFailed},
+	}
+
+nextTest:
+	for _, test := range tests {
+		cache := &newThresholdCaches(1)[0]
+		for i := 0; i < test.numEntries; i++ {
+			var hash daghash.Hash
+			hash[0] = uint8(i + 1)
+
+			// Ensure the hash isn't available in the cache already.
+			_, ok := cache.Lookup(&hash)
+			if ok {
+				t.Errorf("Lookup (%s): has entry for hash %v",
+					test.name, hash)
+				continue nextTest
+			}
+
+			// Ensure hash that was added to the cache reports it's
+			// available and the state is the expected value.
+			cache.Update(&hash, test.state)
+			state, ok := cache.Lookup(&hash)
+			if !ok {
+				t.Errorf("Lookup (%s): missing entry for hash "+
+					"%v", test.name, hash)
+				continue nextTest
+			}
+			if state != test.state {
+				t.Errorf("Lookup (%s): state mismatch - got "+
+					"%v, want %v", test.name, state,
+					test.state)
+				continue nextTest
+			}
+
+			// Ensure adding an existing hash with the same state
+			// doesn't break the existing entry.
+			cache.Update(&hash, test.state)
+			state, ok = cache.Lookup(&hash)
+			if !ok {
+				t.Errorf("Lookup (%s): missing entry after "+
+					"second add for hash %v", test.name,
+					hash)
+				continue nextTest
+			}
+			if state != test.state {
+				t.Errorf("Lookup (%s): state mismatch after "+
+					"second add - got %v, want %v",
+					test.name, state, test.state)
+				continue nextTest
+			}
+
+			// Ensure adding an existing hash with a different state
+			// updates the existing entry.
+			newState := ThresholdFailed
+			if newState == test.state {
+				newState = ThresholdStarted
+			}
+			cache.Update(&hash, newState)
+			state, ok = cache.Lookup(&hash)
+			if !ok {
+				t.Errorf("Lookup (%s): missing entry after "+
+					"state change for hash %v", test.name,
+					hash)
+				continue nextTest
+			}
+			if state != newState {
+				t.Errorf("Lookup (%s): state mismatch after "+
+					"state change - got %v, want %v",
+					test.name, state, newState)
+				continue nextTest
+			}
+		}
+	}
+}

blockdag/timesorter.go

@@ -0,0 +1,27 @@
+// Copyright (c) 2013-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+// timeSorter implements sort.Interface to allow a slice of timestamps to
+// be sorted.
+type timeSorter []int64
+
+// Len returns the number of timestamps in the slice. It is part of the
+// sort.Interface implementation.
+func (s timeSorter) Len() int {
+	return len(s)
+}
+
+// Swap swaps the timestamps at the passed indices. It is part of the
+// sort.Interface implementation.
+func (s timeSorter) Swap(i, j int) {
+	s[i], s[j] = s[j], s[i]
+}
+
+// Less returns whether the timstamp with index i should sort before the
+// timestamp with index j. It is part of the sort.Interface implementation.
+func (s timeSorter) Less(i, j int) bool {
+	return s[i] < s[j]
+}

blockdag/timesorter_test.go

@@ -0,0 +1,47 @@
+// Copyright (c) 2013-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"reflect"
+	"sort"
+	"testing"
+)
+
+// TestTimeSorter tests the timeSorter implementation.
+func TestTimeSorter(t *testing.T) {
+	tests := []struct {
+		in   []int64
+		want []int64
+	}{
+		{
+			in: []int64{
+				1351228575, // Fri Oct 26 05:16:15 UTC 2012 (Block #205000)
+				1348310759, // Sat Sep 22 10:45:59 UTC 2012 (Block #200000)
+				1305758502, // Wed May 18 22:41:42 UTC 2011 (Block #125000)
+				1347777156, // Sun Sep 16 06:32:36 UTC 2012 (Block #199000)
+				1349492104, // Sat Oct  6 02:55:04 UTC 2012 (Block #202000)
+			},
+			want: []int64{
+				1305758502, // Wed May 18 22:41:42 UTC 2011 (Block #125000)
+				1347777156, // Sun Sep 16 06:32:36 UTC 2012 (Block #199000)
+				1348310759, // Sat Sep 22 10:45:59 UTC 2012 (Block #200000)
+				1349492104, // Sat Oct  6 02:55:04 UTC 2012 (Block #202000)
+				1351228575, // Fri Oct 26 05:16:15 UTC 2012 (Block #205000)
+			},
+		},
+	}
+
+	for i, test := range tests {
+		result := make([]int64, len(test.in))
+		copy(result, test.in)
+		sort.Sort(timeSorter(result))
+		if !reflect.DeepEqual(result, test.want) {
+			t.Errorf("timeSorter #%d got %v want %v", i, result,
+				test.want)
+			continue
+		}
+	}
+}

blockdag/utxo_ecmh.go

@@ -0,0 +1,32 @@
+package blockdag
+
+import (
+	"bytes"
+	"github.com/golang/groupcache/lru"
+	"github.com/kaspanet/kaspad/ecc"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/wire"
+)
+
+const ecmhCacheSize = 4_000_000
+
+var (
+	utxoToECMHCache = lru.New(ecmhCacheSize)
+)
+
+func utxoMultiset(entry *UTXOEntry, outpoint *wire.Outpoint) (*ecc.Multiset, error) {
+	w := &bytes.Buffer{}
+	err := serializeUTXO(w, entry, outpoint)
+	if err != nil {
+		return nil, err
+	}
+	serializedUTXO := w.Bytes()
+	utxoHash := daghash.DoubleHashH(serializedUTXO)
+
+	if cachedMSPoint, ok := utxoToECMHCache.Get(utxoHash); ok {
+		return cachedMSPoint.(*ecc.Multiset), nil
+	}
+	msPoint := ecc.NewMultiset(ecc.S256()).Add(serializedUTXO)
+	utxoToECMHCache.Add(utxoHash, msPoint)
+	return msPoint, nil
+}

blockdag/utxodiffstore.go

@@ -0,0 +1,207 @@
+package blockdag
+
+import (
+	"bytes"
+	"github.com/kaspanet/kaspad/database"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/util/locks"
+	"github.com/pkg/errors"
+)
+
+var multisetPointSize = 32
+
+type blockUTXODiffData struct {
+	diff      *UTXODiff
+	diffChild *blockNode
+}
+
+type utxoDiffStore struct {
+	dag    *BlockDAG
+	dirty  map[daghash.Hash]struct{}
+	loaded map[daghash.Hash]*blockUTXODiffData
+	mtx    *locks.PriorityMutex
+}
+
+func newUTXODiffStore(dag *BlockDAG) *utxoDiffStore {
+	return &utxoDiffStore{
+		dag:    dag,
+		dirty:  make(map[daghash.Hash]struct{}),
+		loaded: make(map[daghash.Hash]*blockUTXODiffData),
+		mtx:    locks.NewPriorityMutex(),
+	}
+}
+
+func (diffStore *utxoDiffStore) setBlockDiff(node *blockNode, diff *UTXODiff) error {
+	diffStore.mtx.HighPriorityWriteLock()
+	defer diffStore.mtx.HighPriorityWriteUnlock()
+	// load the diff data from DB to diffStore.loaded
+	_, exists, err := diffStore.diffDataByHash(node.hash)
+	if err != nil {
+		return err
+	}
+	if !exists {
+		diffStore.loaded[*node.hash] = &blockUTXODiffData{}
+	}
+
+	diffStore.loaded[*node.hash].diff = diff
+	diffStore.setBlockAsDirty(node.hash)
+	return nil
+}
+
+func (diffStore *utxoDiffStore) setBlockDiffChild(node *blockNode, diffChild *blockNode) error {
+	diffStore.mtx.HighPriorityWriteLock()
+	defer diffStore.mtx.HighPriorityWriteUnlock()
+	// load the diff data from DB to diffStore.loaded
+	_, exists, err := diffStore.diffDataByHash(node.hash)
+	if err != nil {
+		return err
+	}
+	if !exists {
+		return diffNotFoundError(node)
+	}
+
+	diffStore.loaded[*node.hash].diffChild = diffChild
+	diffStore.setBlockAsDirty(node.hash)
+	return nil
+}
+
+func (diffStore *utxoDiffStore) removeBlocksDiffData(dbTx database.Tx, blockHashes []*daghash.Hash) error {
+	for _, hash := range blockHashes {
+		err := diffStore.removeBlockDiffData(dbTx, hash)
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (diffStore *utxoDiffStore) removeBlockDiffData(dbTx database.Tx, blockHash *daghash.Hash) error {
+	diffStore.mtx.LowPriorityWriteLock()
+	defer diffStore.mtx.LowPriorityWriteUnlock()
+	delete(diffStore.loaded, *blockHash)
+	err := dbRemoveDiffData(dbTx, blockHash)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+func (diffStore *utxoDiffStore) setBlockAsDirty(blockHash *daghash.Hash) {
+	diffStore.dirty[*blockHash] = struct{}{}
+}
+
+func (diffStore *utxoDiffStore) diffDataByHash(hash *daghash.Hash) (*blockUTXODiffData, bool, error) {
+	if diffData, ok := diffStore.loaded[*hash]; ok {
+		return diffData, true, nil
+	}
+	diffData, err := diffStore.diffDataFromDB(hash)
+	if err != nil {
+		return nil, false, err
+	}
+	exists := diffData != nil
+	if exists {
+		diffStore.loaded[*hash] = diffData
+	}
+	return diffData, exists, nil
+}
+
+func diffNotFoundError(node *blockNode) error {
+	return errors.Errorf("Couldn't find diff data for block %s", node.hash)
+}
+
+func (diffStore *utxoDiffStore) diffByNode(node *blockNode) (*UTXODiff, error) {
+	diffStore.mtx.HighPriorityReadLock()
+	defer diffStore.mtx.HighPriorityReadUnlock()
+	diffData, exists, err := diffStore.diffDataByHash(node.hash)
+	if err != nil {
+		return nil, err
+	}
+	if !exists {
+		return nil, diffNotFoundError(node)
+	}
+	return diffData.diff, nil
+}
+
+func (diffStore *utxoDiffStore) diffChildByNode(node *blockNode) (*blockNode, error) {
+	diffStore.mtx.HighPriorityReadLock()
+	defer diffStore.mtx.HighPriorityReadUnlock()
+	diffData, exists, err := diffStore.diffDataByHash(node.hash)
+	if err != nil {
+		return nil, err
+	}
+	if !exists {
+		return nil, diffNotFoundError(node)
+	}
+	return diffData.diffChild, nil
+}
+
+func (diffStore *utxoDiffStore) diffDataFromDB(hash *daghash.Hash) (*blockUTXODiffData, error) {
+	var diffData *blockUTXODiffData
+	err := diffStore.dag.db.View(func(dbTx database.Tx) error {
+		bucket := dbTx.Metadata().Bucket(utxoDiffsBucketName)
+		serializedBlockDiffData := bucket.Get(hash[:])
+		if serializedBlockDiffData != nil {
+			var err error
+			diffData, err = diffStore.deserializeBlockUTXODiffData(serializedBlockDiffData)
+			return err
+		}
+		return nil
+	})
+	if err != nil {
+		return nil, err
+	}
+	return diffData, nil
+}
+
+// flushToDB writes all dirty diff data to the database. If all writes
+// succeed, this clears the dirty set.
+func (diffStore *utxoDiffStore) flushToDB(dbTx database.Tx) error {
+	diffStore.mtx.HighPriorityWriteLock()
+	defer diffStore.mtx.HighPriorityWriteUnlock()
+	if len(diffStore.dirty) == 0 {
+		return nil
+	}
+
+	// Allocate a buffer here to avoid needless allocations/grows
+	// while writing each entry.
+	buffer := &bytes.Buffer{}
+	for hash := range diffStore.dirty {
+		buffer.Reset()
+		diffData := diffStore.loaded[hash]
+		err := dbStoreDiffData(dbTx, buffer, &hash, diffData)
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (diffStore *utxoDiffStore) clearDirtyEntries() {
+	diffStore.dirty = make(map[daghash.Hash]struct{})
+}
+
+// dbStoreDiffData stores the UTXO diff data to the database.
+// This overwrites the current entry if there exists one.
+func dbStoreDiffData(dbTx database.Tx, writeBuffer *bytes.Buffer, hash *daghash.Hash, diffData *blockUTXODiffData) error {
+	// To avoid a ton of allocs, use the given writeBuffer
+	// instead of allocating one. We expect the buffer to
+	// already be initalized and, in most cases, to already
+	// be large enough to accommodate the serialized data
+	// without growing.
+	err := serializeBlockUTXODiffData(writeBuffer, diffData)
+	if err != nil {
+		return err
+	}
+
+	// Bucket.Put doesn't copy on its own, so we manually
+	// copy here. We do so because we expect the buffer
+	// to be reused once we're done with it.
+	serializedDiffData := make([]byte, writeBuffer.Len())
+	copy(serializedDiffData, writeBuffer.Bytes())
+
+	return dbTx.Metadata().Bucket(utxoDiffsBucketName).Put(hash[:], serializedDiffData)
+}
+
+func dbRemoveDiffData(dbTx database.Tx, hash *daghash.Hash) error {
+	return dbTx.Metadata().Bucket(utxoDiffsBucketName).Delete(hash[:])
+}

blockdag/utxodiffstore_test.go

@@ -0,0 +1,86 @@
+package blockdag
+
+import (
+	"fmt"
+	"github.com/kaspanet/kaspad/dagconfig"
+	"github.com/kaspanet/kaspad/database"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/wire"
+	"reflect"
+	"testing"
+)
+
+func TestUTXODiffStore(t *testing.T) {
+	// Create a new database and DAG instance to run tests against.
+	dag, teardownFunc, err := DAGSetup("TestUTXODiffStore", Config{
+		DAGParams: &dagconfig.SimnetParams,
+	})
+	if err != nil {
+		t.Fatalf("TestUTXODiffStore: Failed to setup DAG instance: %v", err)
+	}
+	defer teardownFunc()
+
+	nodeCounter := byte(0)
+	createNode := func() *blockNode {
+		nodeCounter++
+		node := &blockNode{hash: &daghash.Hash{nodeCounter}}
+		dag.index.AddNode(node)
+		return node
+	}
+
+	// Check that an error is returned when asking for non existing node
+	nonExistingNode := createNode()
+	_, err = dag.utxoDiffStore.diffByNode(nonExistingNode)
+	expectedErrString := fmt.Sprintf("Couldn't find diff data for block %s", nonExistingNode.hash)
+	if err == nil || err.Error() != expectedErrString {
+		t.Errorf("diffByNode: expected error %s but got %s", expectedErrString, err)
+	}
+
+	// Add node's diff data to the utxoDiffStore and check if it's checked correctly.
+	node := createNode()
+	diff := NewUTXODiff()
+	diff.toAdd.add(wire.Outpoint{TxID: daghash.TxID{0x01}, Index: 0}, &UTXOEntry{amount: 1, scriptPubKey: []byte{0x01}})
+	diff.toRemove.add(wire.Outpoint{TxID: daghash.TxID{0x02}, Index: 0}, &UTXOEntry{amount: 2, scriptPubKey: []byte{0x02}})
+	if err := dag.utxoDiffStore.setBlockDiff(node, diff); err != nil {
+		t.Fatalf("setBlockDiff: unexpected error: %s", err)
+	}
+	diffChild := createNode()
+	if err := dag.utxoDiffStore.setBlockDiffChild(node, diffChild); err != nil {
+		t.Fatalf("setBlockDiffChild: unexpected error: %s", err)
+	}
+
+	if storeDiff, err := dag.utxoDiffStore.diffByNode(node); err != nil {
+		t.Fatalf("diffByNode: unexpected error: %s", err)
+	} else if !reflect.DeepEqual(storeDiff, diff) {
+		t.Errorf("Expected diff and storeDiff to be equal")
+	}
+
+	if storeDiffChild, err := dag.utxoDiffStore.diffChildByNode(node); err != nil {
+		t.Fatalf("diffByNode: unexpected error: %s", err)
+	} else if !reflect.DeepEqual(storeDiffChild, diffChild) {
+		t.Errorf("Expected diff and storeDiff to be equal")
+	}
+
+	// Flush changes to db, delete them from the dag.utxoDiffStore.loaded
+	// map, and check if the diff data is re-fetched from the database.
+	err = dag.db.Update(func(dbTx database.Tx) error {
+		return dag.utxoDiffStore.flushToDB(dbTx)
+	})
+	if err != nil {
+		t.Fatalf("Error flushing utxoDiffStore data to DB: %s", err)
+	}
+	delete(dag.utxoDiffStore.loaded, *node.hash)
+
+	if storeDiff, err := dag.utxoDiffStore.diffByNode(node); err != nil {
+		t.Fatalf("diffByNode: unexpected error: %s", err)
+	} else if !reflect.DeepEqual(storeDiff, diff) {
+		t.Errorf("Expected diff and storeDiff to be equal")
+	}
+
+	// Check if getBlockDiff caches the result in dag.utxoDiffStore.loaded
+	if loadedDiffData, ok := dag.utxoDiffStore.loaded[*node.hash]; !ok {
+		t.Errorf("the diff data wasn't added to loaded map after requesting it")
+	} else if !reflect.DeepEqual(loadedDiffData.diff, diff) {
+		t.Errorf("Expected diff and loadedDiff to be equal")
+	}
+}

blockdag/utxoio.go

@@ -0,0 +1,313 @@
+package blockdag
+
+import (
+	"bytes"
+	"encoding/binary"
+	"fmt"
+	"github.com/pkg/errors"
+	"io"
+	"math/big"
+
+	"github.com/kaspanet/kaspad/ecc"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/wire"
+)
+
+// serializeBlockUTXODiffData serializes diff data in the following format:
+// 	Name         | Data type | Description
+//  ------------ | --------- | -----------
+// 	hasDiffChild | Boolean   | Indicates if a diff child exist
+//  diffChild    | Hash      | The diffChild's hash. Empty if hasDiffChild is true.
+//  diff		 | UTXODiff  | The diff data's diff
+func serializeBlockUTXODiffData(w io.Writer, diffData *blockUTXODiffData) error {
+	hasDiffChild := diffData.diffChild != nil
+	err := wire.WriteElement(w, hasDiffChild)
+	if err != nil {
+		return err
+	}
+	if hasDiffChild {
+		err := wire.WriteElement(w, diffData.diffChild.hash)
+		if err != nil {
+			return err
+		}
+	}
+
+	err = serializeUTXODiff(w, diffData.diff)
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+// utxoEntryHeaderCode returns the calculated header code to be used when
+// serializing the provided utxo entry.
+func utxoEntryHeaderCode(entry *UTXOEntry) uint64 {
+	// As described in the serialization format comments, the header code
+	// encodes the blue score shifted over one bit and the block reward flag
+	// in the lowest bit.
+	headerCode := uint64(entry.BlockBlueScore()) << 1
+	if entry.IsCoinbase() {
+		headerCode |= 0x01
+	}
+
+	return headerCode
+}
+
+func (diffStore *utxoDiffStore) deserializeBlockUTXODiffData(serializedDiffDataBytes []byte) (*blockUTXODiffData, error) {
+	diffData := &blockUTXODiffData{}
+	serializedDiffData := bytes.NewBuffer(serializedDiffDataBytes)
+
+	var hasDiffChild bool
+	err := wire.ReadElement(serializedDiffData, &hasDiffChild)
+	if err != nil {
+		return nil, err
+	}
+
+	if hasDiffChild {
+		hash := &daghash.Hash{}
+		err := wire.ReadElement(serializedDiffData, hash)
+		if err != nil {
+			return nil, err
+		}
+		diffData.diffChild = diffStore.dag.index.LookupNode(hash)
+	}
+
+	diffData.diff = &UTXODiff{
+		useMultiset: true,
+	}
+
+	diffData.diff.toAdd, err = deserializeDiffEntries(serializedDiffData)
+	if err != nil {
+		return nil, err
+	}
+
+	diffData.diff.toRemove, err = deserializeDiffEntries(serializedDiffData)
+	if err != nil {
+		return nil, err
+	}
+
+	diffData.diff.diffMultiset, err = deserializeMultiset(serializedDiffData)
+	if err != nil {
+		return nil, err
+	}
+
+	return diffData, nil
+}
+
+func deserializeDiffEntries(r io.Reader) (utxoCollection, error) {
+	count, err := wire.ReadVarInt(r)
+	if err != nil {
+		return nil, err
+	}
+	collection := utxoCollection{}
+	for i := uint64(0); i < count; i++ {
+		outpointSize, err := wire.ReadVarInt(r)
+		if err != nil {
+			return nil, err
+		}
+
+		serializedOutpoint := make([]byte, outpointSize)
+		err = binary.Read(r, byteOrder, serializedOutpoint)
+		if err != nil {
+			return nil, err
+		}
+		outpoint, err := deserializeOutpoint(serializedOutpoint)
+		if err != nil {
+			return nil, err
+		}
+
+		utxoEntrySize, err := wire.ReadVarInt(r)
+		if err != nil {
+			return nil, err
+		}
+		serializedEntry := make([]byte, utxoEntrySize)
+		err = binary.Read(r, byteOrder, serializedEntry)
+		if err != nil {
+			return nil, err
+		}
+		utxoEntry, err := deserializeUTXOEntry(serializedEntry)
+		if err != nil {
+			return nil, err
+		}
+		collection.add(*outpoint, utxoEntry)
+	}
+	return collection, nil
+}
+
+// deserializeMultiset deserializes an EMCH multiset.
+// See serializeMultiset for more details.
+func deserializeMultiset(r io.Reader) (*ecc.Multiset, error) {
+	xBytes := make([]byte, multisetPointSize)
+	yBytes := make([]byte, multisetPointSize)
+	err := binary.Read(r, byteOrder, xBytes)
+	if err != nil {
+		return nil, err
+	}
+	err = binary.Read(r, byteOrder, yBytes)
+	if err != nil {
+		return nil, err
+	}
+	var x, y big.Int
+	x.SetBytes(xBytes)
+	y.SetBytes(yBytes)
+	return ecc.NewMultisetFromPoint(ecc.S256(), &x, &y), nil
+}
+
+// serializeUTXODiff serializes UTXODiff by serializing
+// UTXODiff.toAdd, UTXODiff.toRemove and UTXODiff.Multiset one after the other.
+func serializeUTXODiff(w io.Writer, diff *UTXODiff) error {
+	if !diff.useMultiset {
+		return errors.New("Cannot serialize a UTXO diff without a multiset")
+	}
+	err := serializeUTXOCollection(w, diff.toAdd)
+	if err != nil {
+		return err
+	}
+
+	err = serializeUTXOCollection(w, diff.toRemove)
+	if err != nil {
+		return err
+	}
+	err = serializeMultiset(w, diff.diffMultiset)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+// serializeUTXOCollection serializes utxoCollection by iterating over
+// the utxo entries and serializing them and their corresponding outpoint
+// prefixed by a varint that indicates their size.
+func serializeUTXOCollection(w io.Writer, collection utxoCollection) error {
+	err := wire.WriteVarInt(w, uint64(len(collection)))
+	if err != nil {
+		return err
+	}
+	for outpoint, utxoEntry := range collection {
+		err := serializeUTXO(w, utxoEntry, &outpoint)
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// serializeMultiset serializes an ECMH multiset. The serialization
+// is done by taking the (x,y) coordinnates of the multiset point and
+// padding each one of them with 32 byte (it'll be 32 byte in most
+// cases anyway except one of the coordinates is zero) and writing
+// them one after the other.
+func serializeMultiset(w io.Writer, ms *ecc.Multiset) error {
+	x, y := ms.Point()
+	xBytes := make([]byte, multisetPointSize)
+	copy(xBytes, x.Bytes())
+	yBytes := make([]byte, multisetPointSize)
+	copy(yBytes, y.Bytes())
+
+	err := binary.Write(w, byteOrder, xBytes)
+	if err != nil {
+		return err
+	}
+	err = binary.Write(w, byteOrder, yBytes)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+// serializeUTXO serializes a utxo entry-outpoint pair
+func serializeUTXO(w io.Writer, entry *UTXOEntry, outpoint *wire.Outpoint) error {
+	serializedOutpoint := *outpointKey(*outpoint)
+	err := wire.WriteVarInt(w, uint64(len(serializedOutpoint)))
+	if err != nil {
+		return err
+	}
+
+	err = binary.Write(w, byteOrder, serializedOutpoint)
+	if err != nil {
+		return err
+	}
+
+	serializedUTXOEntry := serializeUTXOEntry(entry)
+	err = wire.WriteVarInt(w, uint64(len(serializedUTXOEntry)))
+	if err != nil {
+		return err
+	}
+	err = binary.Write(w, byteOrder, serializedUTXOEntry)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+// serializeUTXOEntry returns the entry serialized to a format that is suitable
+// for long-term storage. The format is described in detail above.
+func serializeUTXOEntry(entry *UTXOEntry) []byte {
+	// Encode the header code.
+	headerCode := utxoEntryHeaderCode(entry)
+
+	// Calculate the size needed to serialize the entry.
+	size := serializeSizeVLQ(headerCode) +
+		compressedTxOutSize(uint64(entry.Amount()), entry.ScriptPubKey())
+
+	// Serialize the header code followed by the compressed unspent
+	// transaction output.
+	serialized := make([]byte, size)
+	offset := putVLQ(serialized, headerCode)
+	offset += putCompressedTxOut(serialized[offset:], uint64(entry.Amount()),
+		entry.ScriptPubKey())
+
+	return serialized
+}
+
+// deserializeOutpoint decodes an outpoint from the passed serialized byte
+// slice into a new wire.Outpoint using a format that is suitable for long-
+// term storage. this format is described in detail above.
+func deserializeOutpoint(serialized []byte) (*wire.Outpoint, error) {
+	if len(serialized) <= daghash.HashSize {
+		return nil, errDeserialize("unexpected end of data")
+	}
+
+	txID := daghash.TxID{}
+	txID.SetBytes(serialized[:daghash.HashSize])
+	index, _ := deserializeVLQ(serialized[daghash.HashSize:])
+	return wire.NewOutpoint(&txID, uint32(index)), nil
+}
+
+// deserializeUTXOEntry decodes a UTXO entry from the passed serialized byte
+// slice into a new UTXOEntry using a format that is suitable for long-term
+// storage. The format is described in detail above.
+func deserializeUTXOEntry(serialized []byte) (*UTXOEntry, error) {
+	// Deserialize the header code.
+	code, offset := deserializeVLQ(serialized)
+	if offset >= len(serialized) {
+		return nil, errDeserialize("unexpected end of data after header")
+	}
+
+	// Decode the header code.
+	//
+	// Bit 0 indicates whether the containing transaction is a coinbase.
+	// Bits 1-x encode blue score of the containing transaction.
+	isCoinbase := code&0x01 != 0
+	blockBlueScore := code >> 1
+
+	// Decode the compressed unspent transaction output.
+	amount, scriptPubKey, _, err := decodeCompressedTxOut(serialized[offset:])
+	if err != nil {
+		return nil, errDeserialize(fmt.Sprintf("unable to decode "+
+			"UTXO: %s", err))
+	}
+
+	entry := &UTXOEntry{
+		amount:         amount,
+		scriptPubKey:   scriptPubKey,
+		blockBlueScore: blockBlueScore,
+		packedFlags:    0,
+	}
+	if isCoinbase {
+		entry.packedFlags |= tfCoinbase
+	}
+
+	return entry, nil
+}

blockdag/utxoset.go

@@ -0,0 +1,906 @@
+package blockdag
+
+import (
+	"fmt"
+	"github.com/pkg/errors"
+	"math"
+	"sort"
+	"strings"
+
+	"github.com/kaspanet/kaspad/ecc"
+	"github.com/kaspanet/kaspad/wire"
+)
+
+const (
+	// UnacceptedBlueScore is the blue score used for the "block" blueScore
+	// field of the contextual transaction information provided in a
+	// transaction store when it has not yet been accepted by a block.
+	UnacceptedBlueScore uint64 = math.MaxUint64
+)
+
+// UTXOEntry houses details about an individual transaction output in a utxo
+// set such as whether or not it was contained in a coinbase tx, the blue
+// score of the block that accepts the tx, its public key script, and how
+// much it pays.
+type UTXOEntry struct {
+	// NOTE: Additions, deletions, or modifications to the order of the
+	// definitions in this struct should not be changed without considering
+	// how it affects alignment on 64-bit platforms. The current order is
+	// specifically crafted to result in minimal padding. There will be a
+	// lot of these in memory, so a few extra bytes of padding adds up.
+
+	amount         uint64
+	scriptPubKey   []byte // The public key script for the output.
+	blockBlueScore uint64 // Blue score of the block accepting the tx.
+
+	// packedFlags contains additional info about output such as whether it
+	// is a coinbase, and whether it has been modified
+	// since it was loaded. This approach is used in order to reduce memory
+	// usage since there will be a lot of these in memory.
+	packedFlags txoFlags
+}
+
+// IsCoinbase returns whether or not the output was contained in a block
+// reward transaction.
+func (entry *UTXOEntry) IsCoinbase() bool {
+	return entry.packedFlags&tfCoinbase == tfCoinbase
+}
+
+// BlockBlueScore returns the blue score of the block accepting the output.
+func (entry *UTXOEntry) BlockBlueScore() uint64 {
+	return entry.blockBlueScore
+}
+
+// Amount returns the amount of the output.
+func (entry *UTXOEntry) Amount() uint64 {
+	return entry.amount
+}
+
+// ScriptPubKey returns the public key script for the output.
+func (entry *UTXOEntry) ScriptPubKey() []byte {
+	return entry.scriptPubKey
+}
+
+// IsUnaccepted returns true iff this UTXOEntry has been included in a block
+// but has not yet been accepted by any block.
+func (entry *UTXOEntry) IsUnaccepted() bool {
+	return entry.blockBlueScore == UnacceptedBlueScore
+}
+
+// txoFlags is a bitmask defining additional information and state for a
+// transaction output in a UTXO set.
+type txoFlags uint8
+
+const (
+	// tfCoinbase indicates that a txout was contained in a coinbase tx.
+	tfCoinbase txoFlags = 1 << iota
+)
+
+// NewUTXOEntry creates a new utxoEntry representing the given txOut
+func NewUTXOEntry(txOut *wire.TxOut, isCoinbase bool, blockBlueScore uint64) *UTXOEntry {
+	entry := &UTXOEntry{
+		amount:         txOut.Value,
+		scriptPubKey:   txOut.ScriptPubKey,
+		blockBlueScore: blockBlueScore,
+	}
+
+	if isCoinbase {
+		entry.packedFlags |= tfCoinbase
+	}
+
+	return entry
+}
+
+// utxoCollection represents a set of UTXOs indexed by their outpoints
+type utxoCollection map[wire.Outpoint]*UTXOEntry
+
+func (uc utxoCollection) String() string {
+	utxoStrings := make([]string, len(uc))
+
+	i := 0
+	for outpoint, utxoEntry := range uc {
+		utxoStrings[i] = fmt.Sprintf("(%s, %d) => %d, blueScore: %d",
+			outpoint.TxID, outpoint.Index, utxoEntry.amount, utxoEntry.blockBlueScore)
+		i++
+	}
+
+	// Sort strings for determinism.
+	sort.Strings(utxoStrings)
+
+	return fmt.Sprintf("[ %s ]", strings.Join(utxoStrings, ", "))
+}
+
+// add adds a new UTXO entry to this collection
+func (uc utxoCollection) add(outpoint wire.Outpoint, entry *UTXOEntry) {
+	uc[outpoint] = entry
+}
+
+// remove removes a UTXO entry from this collection if it exists
+func (uc utxoCollection) remove(outpoint wire.Outpoint) {
+	delete(uc, outpoint)
+}
+
+// get returns the UTXOEntry represented by provided outpoint,
+// and a boolean value indicating if said UTXOEntry is in the set or not
+func (uc utxoCollection) get(outpoint wire.Outpoint) (*UTXOEntry, bool) {
+	entry, ok := uc[outpoint]
+	return entry, ok
+}
+
+// contains returns a boolean value indicating whether a UTXO entry is in the set
+func (uc utxoCollection) contains(outpoint wire.Outpoint) bool {
+	_, ok := uc[outpoint]
+	return ok
+}
+
+// containsWithBlueScore returns a boolean value indicating whether a UTXOEntry
+// is in the set and its blue score is equal to the given blue score.
+func (uc utxoCollection) containsWithBlueScore(outpoint wire.Outpoint, blueScore uint64) bool {
+	entry, ok := uc.get(outpoint)
+	return ok && entry.blockBlueScore == blueScore
+}
+
+// clone returns a clone of this collection
+func (uc utxoCollection) clone() utxoCollection {
+	clone := utxoCollection{}
+	for outpoint, entry := range uc {
+		clone.add(outpoint, entry)
+	}
+
+	return clone
+}
+
+// UTXODiff represents a diff between two UTXO Sets.
+type UTXODiff struct {
+	toAdd        utxoCollection
+	toRemove     utxoCollection
+	diffMultiset *ecc.Multiset
+	useMultiset  bool
+}
+
+// NewUTXODiffWithoutMultiset creates a new, empty utxoDiff
+// without a multiset.
+func NewUTXODiffWithoutMultiset() *UTXODiff {
+	return &UTXODiff{
+		toAdd:       utxoCollection{},
+		toRemove:    utxoCollection{},
+		useMultiset: false,
+	}
+}
+
+// NewUTXODiff creates a new, empty utxoDiff.
+func NewUTXODiff() *UTXODiff {
+	return &UTXODiff{
+		toAdd:        utxoCollection{},
+		toRemove:     utxoCollection{},
+		useMultiset:  true,
+		diffMultiset: ecc.NewMultiset(ecc.S256()),
+	}
+}
+
+// diffFrom returns a new utxoDiff with the difference between this utxoDiff and another
+// Assumes that:
+// Both utxoDiffs are from the same base
+// If a txOut exists in both utxoDiffs, its underlying values would be the same
+//
+// diffFrom follows a set of rules represented by the following 3 by 3 table:
+//
+//          |           | this      |           |
+// ---------+-----------+-----------+-----------+-----------
+//          |           | toAdd     | toRemove  | None
+// ---------+-----------+-----------+-----------+-----------
+// other    | toAdd     | -         | X         | toAdd
+// ---------+-----------+-----------+-----------+-----------
+//          | toRemove  | X         | -         | toRemove
+// ---------+-----------+-----------+-----------+-----------
+//          | None      | toRemove  | toAdd     | -
+//
+// Key:
+// -		Don't add anything to the result
+// X		Return an error
+// toAdd	Add the UTXO into the toAdd collection of the result
+// toRemove	Add the UTXO into the toRemove collection of the result
+//
+// Examples:
+// 1. This diff contains a UTXO in toAdd, and the other diff contains it in toRemove
+//    diffFrom results in an error
+// 2. This diff contains a UTXO in toRemove, and the other diff does not contain it
+//    diffFrom results in the UTXO being added to toAdd
+func (d *UTXODiff) diffFrom(other *UTXODiff) (*UTXODiff, error) {
+	result := UTXODiff{
+		toAdd:       make(utxoCollection, len(d.toRemove)+len(other.toAdd)),
+		toRemove:    make(utxoCollection, len(d.toAdd)+len(other.toRemove)),
+		useMultiset: d.useMultiset,
+	}
+
+	// Note that the following cases are not accounted for, as they are impossible
+	// as long as the base utxoSet is the same:
+	// - if utxoEntry is in d.toAdd and other.toRemove
+	// - if utxoEntry is in d.toRemove and other.toAdd
+
+	// All transactions in d.toAdd:
+	// If they are not in other.toAdd - should be added in result.toRemove
+	// If they are in other.toRemove - base utxoSet is not the same
+	for outpoint, utxoEntry := range d.toAdd {
+		if !other.toAdd.containsWithBlueScore(outpoint, utxoEntry.blockBlueScore) {
+			result.toRemove.add(outpoint, utxoEntry)
+		}
+		if diffEntry, ok := other.toRemove.get(outpoint); ok {
+			// An exception is made for entries with unequal blue scores
+			// as long as the appropriate entry exists in either d.toRemove
+			// or other.toAdd.
+			// These are just "updates" to accepted blue score
+			if diffEntry.blockBlueScore != utxoEntry.blockBlueScore &&
+				(d.toRemove.containsWithBlueScore(outpoint, diffEntry.blockBlueScore) ||
+					other.toAdd.containsWithBlueScore(outpoint, utxoEntry.blockBlueScore)) {
+				continue
+			}
+			return nil, errors.Errorf("diffFrom: outpoint %s both in d.toAdd and in other.toRemove", outpoint)
+		}
+	}
+
+	// All transactions in d.toRemove:
+	// If they are not in other.toRemove - should be added in result.toAdd
+	// If they are in other.toAdd - base utxoSet is not the same
+	for outpoint, utxoEntry := range d.toRemove {
+		if !other.toRemove.containsWithBlueScore(outpoint, utxoEntry.blockBlueScore) {
+			result.toAdd.add(outpoint, utxoEntry)
+		}
+		if diffEntry, ok := other.toAdd.get(outpoint); ok {
+			// An exception is made for entries with unequal blue scores
+			// as long as the appropriate entry exists in either d.toAdd
+			// or other.toRemove.
+			// These are just "updates" to accepted blue score
+			if diffEntry.blockBlueScore != utxoEntry.blockBlueScore &&
+				(d.toAdd.containsWithBlueScore(outpoint, diffEntry.blockBlueScore) ||
+					other.toRemove.containsWithBlueScore(outpoint, utxoEntry.blockBlueScore)) {
+				continue
+			}
+			return nil, errors.New("diffFrom: transaction both in d.toRemove and in other.toAdd")
+		}
+	}
+
+	// All transactions in other.toAdd:
+	// If they are not in d.toAdd - should be added in result.toAdd
+	for outpoint, utxoEntry := range other.toAdd {
+		if !d.toAdd.containsWithBlueScore(outpoint, utxoEntry.blockBlueScore) {
+			result.toAdd.add(outpoint, utxoEntry)
+		}
+	}
+
+	// All transactions in other.toRemove:
+	// If they are not in d.toRemove - should be added in result.toRemove
+	for outpoint, utxoEntry := range other.toRemove {
+		if !d.toRemove.containsWithBlueScore(outpoint, utxoEntry.blockBlueScore) {
+			result.toRemove.add(outpoint, utxoEntry)
+		}
+	}
+
+	if d.useMultiset {
+		// Create a new diffMultiset as the subtraction of the two diffs.
+		result.diffMultiset = other.diffMultiset.Subtract(d.diffMultiset)
+	}
+
+	return &result, nil
+}
+
+// WithDiff applies provided diff to this diff, creating a new utxoDiff, that would be the result if
+// first d, and than diff were applied to the same base
+//
+// WithDiff follows a set of rules represented by the following 3 by 3 table:
+//
+//          |           | this      |           |
+// ---------+-----------+-----------+-----------+-----------
+//          |           | toAdd     | toRemove  | None
+// ---------+-----------+-----------+-----------+-----------
+// other    | toAdd     | X         | -         | toAdd
+// ---------+-----------+-----------+-----------+-----------
+//          | toRemove  | -         | X         | toRemove
+// ---------+-----------+-----------+-----------+-----------
+//          | None      | toAdd     | toRemove  | -
+//
+// Key:
+// -		Don't add anything to the result
+// X		Return an error
+// toAdd	Add the UTXO into the toAdd collection of the result
+// toRemove	Add the UTXO into the toRemove collection of the result
+//
+// Examples:
+// 1. This diff contains a UTXO in toAdd, and the other diff contains it in toRemove
+//    WithDiff results in nothing being added
+// 2. This diff contains a UTXO in toRemove, and the other diff does not contain it
+//    WithDiff results in the UTXO being added to toRemove
+func (d *UTXODiff) WithDiff(diff *UTXODiff) (*UTXODiff, error) {
+	result := UTXODiff{
+		toAdd:       make(utxoCollection, len(d.toAdd)+len(diff.toAdd)),
+		toRemove:    make(utxoCollection, len(d.toRemove)+len(diff.toRemove)),
+		useMultiset: d.useMultiset,
+	}
+
+	// All transactions in d.toAdd:
+	// If they are not in diff.toRemove - should be added in result.toAdd
+	// If they are in diff.toAdd - should throw an error
+	// Otherwise - should be ignored
+	for outpoint, utxoEntry := range d.toAdd {
+		if !diff.toRemove.containsWithBlueScore(outpoint, utxoEntry.blockBlueScore) {
+			result.toAdd.add(outpoint, utxoEntry)
+		}
+		if diffEntry, ok := diff.toAdd.get(outpoint); ok {
+			// An exception is made for entries with unequal blue scores
+			// as long as the appropriate entry exists in either d.toRemove
+			// or diff.toRemove.
+			// These are just "updates" to accepted blue score
+			if diffEntry.blockBlueScore != utxoEntry.blockBlueScore &&
+				(d.toRemove.containsWithBlueScore(outpoint, diffEntry.blockBlueScore) ||
+					diff.toRemove.containsWithBlueScore(outpoint, utxoEntry.blockBlueScore)) {
+				continue
+			}
+			return nil, ruleError(ErrWithDiff, fmt.Sprintf("WithDiff: outpoint %s both in d.toAdd and in other.toAdd", outpoint))
+		}
+	}
+
+	// All transactions in d.toRemove:
+	// If they are not in diff.toAdd - should be added in result.toRemove
+	// If they are in diff.toRemove - should throw an error
+	// Otherwise - should be ignored
+	for outpoint, utxoEntry := range d.toRemove {
+		if !diff.toAdd.containsWithBlueScore(outpoint, utxoEntry.blockBlueScore) {
+			result.toRemove.add(outpoint, utxoEntry)
+		}
+		if diffEntry, ok := diff.toRemove.get(outpoint); ok {
+			// An exception is made for entries with unequal blue scores
+			// as long as the appropriate entry exists in either d.toAdd
+			// or diff.toAdd.
+			// These are just "updates" to accepted blue score
+			if diffEntry.blockBlueScore != utxoEntry.blockBlueScore &&
+				(d.toAdd.containsWithBlueScore(outpoint, diffEntry.blockBlueScore) ||
+					diff.toAdd.containsWithBlueScore(outpoint, utxoEntry.blockBlueScore)) {
+				continue
+			}
+			return nil, ruleError(ErrWithDiff, "WithDiff: transaction both in d.toRemove and in other.toRemove")
+		}
+	}
+
+	// All transactions in diff.toAdd:
+	// If they are not in d.toRemove - should be added in result.toAdd
+	for outpoint, utxoEntry := range diff.toAdd {
+		if !d.toRemove.containsWithBlueScore(outpoint, utxoEntry.blockBlueScore) {
+			result.toAdd.add(outpoint, utxoEntry)
+		}
+	}
+
+	// All transactions in diff.toRemove:
+	// If they are not in d.toAdd - should be added in result.toRemove
+	for outpoint, utxoEntry := range diff.toRemove {
+		if !d.toAdd.containsWithBlueScore(outpoint, utxoEntry.blockBlueScore) {
+			result.toRemove.add(outpoint, utxoEntry)
+		}
+	}
+
+	// Apply diff.diffMultiset to d.diffMultiset
+	if d.useMultiset {
+		result.diffMultiset = d.diffMultiset.Union(diff.diffMultiset)
+	}
+
+	return &result, nil
+}
+
+// clone returns a clone of this utxoDiff
+func (d *UTXODiff) clone() *UTXODiff {
+	clone := &UTXODiff{
+		toAdd:       d.toAdd.clone(),
+		toRemove:    d.toRemove.clone(),
+		useMultiset: d.useMultiset,
+	}
+	if d.useMultiset {
+		clone.diffMultiset = d.diffMultiset.Clone()
+	}
+	return clone
+}
+
+// AddEntry adds a UTXOEntry to the diff
+//
+// If d.useMultiset is true, this function MUST be
+// called with the DAG lock held.
+func (d *UTXODiff) AddEntry(outpoint wire.Outpoint, entry *UTXOEntry) error {
+	if d.toRemove.containsWithBlueScore(outpoint, entry.blockBlueScore) {
+		d.toRemove.remove(outpoint)
+	} else if _, exists := d.toAdd[outpoint]; exists {
+		return errors.Errorf("AddEntry: Cannot add outpoint %s twice", outpoint)
+	} else {
+		d.toAdd.add(outpoint, entry)
+	}
+
+	if d.useMultiset {
+		newMs, err := addUTXOToMultiset(d.diffMultiset, entry, &outpoint)
+		if err != nil {
+			return err
+		}
+		d.diffMultiset = newMs
+	}
+	return nil
+}
+
+// RemoveEntry removes a UTXOEntry from the diff.
+//
+// If d.useMultiset is true, this function MUST be
+// called with the DAG lock held.
+func (d *UTXODiff) RemoveEntry(outpoint wire.Outpoint, entry *UTXOEntry) error {
+	if d.toAdd.containsWithBlueScore(outpoint, entry.blockBlueScore) {
+		d.toAdd.remove(outpoint)
+	} else if _, exists := d.toRemove[outpoint]; exists {
+		return errors.Errorf("removeEntry: Cannot remove outpoint %s twice", outpoint)
+	} else {
+		d.toRemove.add(outpoint, entry)
+	}
+
+	if d.useMultiset {
+		newMs, err := removeUTXOFromMultiset(d.diffMultiset, entry, &outpoint)
+		if err != nil {
+			return err
+		}
+		d.diffMultiset = newMs
+	}
+	return nil
+}
+
+func (d UTXODiff) String() string {
+	if d.useMultiset {
+		return fmt.Sprintf("toAdd: %s; toRemove: %s, Multiset-Hash: %s", d.toAdd, d.toRemove, d.diffMultiset.Hash())
+	}
+	return fmt.Sprintf("toAdd: %s; toRemove: %s", d.toAdd, d.toRemove)
+}
+
+// UTXOSet represents a set of unspent transaction outputs
+// Every DAG has exactly one fullUTXOSet.
+// When a new block arrives, it is validated and applied to the fullUTXOSet in the following manner:
+// 1. Get the block's PastUTXO:
+// 2. Add all the block's transactions to the block's PastUTXO
+// 3. For each of the block's parents,
+// 3.1. Rebuild their utxoDiff
+// 3.2. Set the block as their diffChild
+// 4. Create and initialize a new virtual block
+// 5. Get the new virtual's PastUTXO
+// 6. Rebuild the utxoDiff for all the tips
+// 7. Convert (meld) the new virtual's diffUTXOSet into a fullUTXOSet. This updates the DAG's fullUTXOSet
+type UTXOSet interface {
+	fmt.Stringer
+	diffFrom(other UTXOSet) (*UTXODiff, error)
+	WithDiff(utxoDiff *UTXODiff) (UTXOSet, error)
+	diffFromTx(tx *wire.MsgTx, acceptingBlueScore uint64) (*UTXODiff, error)
+	diffFromAcceptedTx(tx *wire.MsgTx, acceptingBlueScore uint64) (*UTXODiff, error)
+	AddTx(tx *wire.MsgTx, blockBlueScore uint64) (ok bool, err error)
+	clone() UTXOSet
+	Get(outpoint wire.Outpoint) (*UTXOEntry, bool)
+	Multiset() *ecc.Multiset
+	WithTransactions(transactions []*wire.MsgTx, blockBlueScore uint64, ignoreDoubleSpends bool) (UTXOSet, error)
+}
+
+// diffFromTx is a common implementation for diffFromTx, that works
+// for both diff-based and full UTXO sets
+// Returns a diff that is equivalent to provided transaction,
+// or an error if provided transaction is not valid in the context of this UTXOSet
+func diffFromTx(u UTXOSet, tx *wire.MsgTx, acceptingBlueScore uint64) (*UTXODiff, error) {
+	diff := NewUTXODiff()
+	isCoinbase := tx.IsCoinBase()
+	if !isCoinbase {
+		for _, txIn := range tx.TxIn {
+			if entry, ok := u.Get(txIn.PreviousOutpoint); ok {
+				err := diff.RemoveEntry(txIn.PreviousOutpoint, entry)
+				if err != nil {
+					return nil, err
+				}
+			} else {
+				return nil, ruleError(ErrMissingTxOut, fmt.Sprintf(
+					"Transaction %s is invalid because spends outpoint %s that is not in utxo set",
+					tx.TxID(), txIn.PreviousOutpoint))
+			}
+		}
+	}
+	for i, txOut := range tx.TxOut {
+		entry := NewUTXOEntry(txOut, isCoinbase, acceptingBlueScore)
+		outpoint := *wire.NewOutpoint(tx.TxID(), uint32(i))
+		err := diff.AddEntry(outpoint, entry)
+		if err != nil {
+			return nil, err
+		}
+	}
+	return diff, nil
+}
+
+// diffFromAcceptedTx is a common implementation for diffFromAcceptedTx, that works
+// for both diff-based and full UTXO sets.
+// Returns a diff that replaces an entry's blockBlueScore with the given acceptingBlueScore.
+// Returns an error if the provided transaction's entry is not valid in the context
+// of this UTXOSet.
+func diffFromAcceptedTx(u UTXOSet, tx *wire.MsgTx, acceptingBlueScore uint64) (*UTXODiff, error) {
+	diff := NewUTXODiff()
+	isCoinbase := tx.IsCoinBase()
+	for i, txOut := range tx.TxOut {
+		// Fetch any unaccepted transaction
+		existingOutpoint := *wire.NewOutpoint(tx.TxID(), uint32(i))
+		existingEntry, ok := u.Get(existingOutpoint)
+		if !ok {
+			return nil, errors.Errorf("cannot accept outpoint %s because it doesn't exist in the given UTXO", existingOutpoint)
+		}
+
+		// Remove unaccepted entries
+		err := diff.RemoveEntry(existingOutpoint, existingEntry)
+		if err != nil {
+			return nil, err
+		}
+
+		// Add new entries with their accepting blue score
+		newEntry := NewUTXOEntry(txOut, isCoinbase, acceptingBlueScore)
+		err = diff.AddEntry(existingOutpoint, newEntry)
+		if err != nil {
+			return nil, err
+		}
+	}
+	return diff, nil
+}
+
+// FullUTXOSet represents a full list of transaction outputs and their values
+type FullUTXOSet struct {
+	utxoCollection
+	UTXOMultiset *ecc.Multiset
+}
+
+// NewFullUTXOSet creates a new utxoSet with full list of transaction outputs and their values
+func NewFullUTXOSet() *FullUTXOSet {
+	return &FullUTXOSet{
+		utxoCollection: utxoCollection{},
+		UTXOMultiset:   ecc.NewMultiset(ecc.S256()),
+	}
+}
+
+// newFullUTXOSetFromUTXOCollection converts a utxoCollection to a FullUTXOSet
+func newFullUTXOSetFromUTXOCollection(collection utxoCollection) (*FullUTXOSet, error) {
+	var err error
+	multiset := ecc.NewMultiset(ecc.S256())
+	for outpoint, utxoEntry := range collection {
+		multiset, err = addUTXOToMultiset(multiset, utxoEntry, &outpoint)
+		if err != nil {
+			return nil, err
+		}
+	}
+	return &FullUTXOSet{
+		utxoCollection: collection,
+		UTXOMultiset:   multiset,
+	}, nil
+}
+
+// diffFrom returns the difference between this utxoSet and another
+// diffFrom can only work when other is a diffUTXOSet, and its base utxoSet is this.
+func (fus *FullUTXOSet) diffFrom(other UTXOSet) (*UTXODiff, error) {
+	otherDiffSet, ok := other.(*DiffUTXOSet)
+	if !ok {
+		return nil, errors.New("can't diffFrom two fullUTXOSets")
+	}
+
+	if otherDiffSet.base != fus {
+		return nil, errors.New("can diffFrom only with diffUTXOSet where this fullUTXOSet is the base")
+	}
+
+	return otherDiffSet.UTXODiff, nil
+}
+
+// WithDiff returns a utxoSet which is a diff between this and another utxoSet
+func (fus *FullUTXOSet) WithDiff(other *UTXODiff) (UTXOSet, error) {
+	return NewDiffUTXOSet(fus, other.clone()), nil
+}
+
+// AddTx adds a transaction to this utxoSet and returns isAccepted=true iff it's valid in this UTXO's context.
+// It returns error if something unexpected happens, such as serialization error (isAccepted=false doesn't
+// necessarily means there's an error).
+//
+// This function MUST be called with the DAG lock held.
+func (fus *FullUTXOSet) AddTx(tx *wire.MsgTx, blueScore uint64) (isAccepted bool, err error) {
+	isCoinbase := tx.IsCoinBase()
+	if !isCoinbase {
+		if !fus.containsInputs(tx) {
+			return false, nil
+		}
+
+		for _, txIn := range tx.TxIn {
+			outpoint := *wire.NewOutpoint(&txIn.PreviousOutpoint.TxID, txIn.PreviousOutpoint.Index)
+			err := fus.removeAndUpdateMultiset(outpoint)
+			if err != nil {
+				return false, err
+			}
+		}
+	}
+
+	for i, txOut := range tx.TxOut {
+		outpoint := *wire.NewOutpoint(tx.TxID(), uint32(i))
+		entry := NewUTXOEntry(txOut, isCoinbase, blueScore)
+
+		err := fus.addAndUpdateMultiset(outpoint, entry)
+		if err != nil {
+			return false, err
+		}
+	}
+
+	return true, nil
+}
+
+// diffFromTx returns a diff that is equivalent to provided transaction,
+// or an error if provided transaction is not valid in the context of this UTXOSet
+func (fus *FullUTXOSet) diffFromTx(tx *wire.MsgTx, acceptingBlueScore uint64) (*UTXODiff, error) {
+	return diffFromTx(fus, tx, acceptingBlueScore)
+}
+
+func (fus *FullUTXOSet) containsInputs(tx *wire.MsgTx) bool {
+	for _, txIn := range tx.TxIn {
+		outpoint := *wire.NewOutpoint(&txIn.PreviousOutpoint.TxID, txIn.PreviousOutpoint.Index)
+		if !fus.contains(outpoint) {
+			return false
+		}
+	}
+
+	return true
+}
+
+func (fus *FullUTXOSet) diffFromAcceptedTx(tx *wire.MsgTx, acceptingBlueScore uint64) (*UTXODiff, error) {
+	return diffFromAcceptedTx(fus, tx, acceptingBlueScore)
+}
+
+// clone returns a clone of this utxoSet
+func (fus *FullUTXOSet) clone() UTXOSet {
+	return &FullUTXOSet{utxoCollection: fus.utxoCollection.clone(), UTXOMultiset: fus.UTXOMultiset.Clone()}
+}
+
+// Get returns the UTXOEntry associated with the given Outpoint, and a boolean indicating if such entry was found
+func (fus *FullUTXOSet) Get(outpoint wire.Outpoint) (*UTXOEntry, bool) {
+	utxoEntry, ok := fus.utxoCollection[outpoint]
+	return utxoEntry, ok
+}
+
+// Multiset returns the ecmh-Multiset of this utxoSet
+func (fus *FullUTXOSet) Multiset() *ecc.Multiset {
+	return fus.UTXOMultiset
+}
+
+// addAndUpdateMultiset adds a UTXOEntry to this utxoSet and updates its multiset accordingly
+func (fus *FullUTXOSet) addAndUpdateMultiset(outpoint wire.Outpoint, entry *UTXOEntry) error {
+	fus.add(outpoint, entry)
+	newMs, err := addUTXOToMultiset(fus.UTXOMultiset, entry, &outpoint)
+	if err != nil {
+		return err
+	}
+	fus.UTXOMultiset = newMs
+	return nil
+}
+
+// removeAndUpdateMultiset removes a UTXOEntry from this utxoSet and updates its multiset accordingly
+func (fus *FullUTXOSet) removeAndUpdateMultiset(outpoint wire.Outpoint) error {
+	entry, ok := fus.Get(outpoint)
+	if !ok {
+		return errors.Errorf("Couldn't find outpoint %s", outpoint)
+	}
+	fus.remove(outpoint)
+	var err error
+	newMs, err := removeUTXOFromMultiset(fus.UTXOMultiset, entry, &outpoint)
+	if err != nil {
+		return err
+	}
+	fus.UTXOMultiset = newMs
+	return nil
+}
+
+// WithTransactions returns a new UTXO Set with the added transactions.
+//
+// This function MUST be called with the DAG lock held.
+func (fus *FullUTXOSet) WithTransactions(transactions []*wire.MsgTx, blockBlueScore uint64, ignoreDoubleSpends bool) (UTXOSet, error) {
+	diffSet := NewDiffUTXOSet(fus, NewUTXODiff())
+	for _, tx := range transactions {
+		isAccepted, err := diffSet.AddTx(tx, blockBlueScore)
+		if err != nil {
+			return nil, err
+		}
+		if !ignoreDoubleSpends && !isAccepted {
+			return nil, errors.Errorf("Transaction %s is not valid with the current UTXO set", tx.TxID())
+		}
+	}
+	return UTXOSet(diffSet), nil
+}
+
+// DiffUTXOSet represents a utxoSet with a base fullUTXOSet and a UTXODiff
+type DiffUTXOSet struct {
+	base     *FullUTXOSet
+	UTXODiff *UTXODiff
+}
+
+// NewDiffUTXOSet Creates a new utxoSet based on a base fullUTXOSet and a UTXODiff
+func NewDiffUTXOSet(base *FullUTXOSet, diff *UTXODiff) *DiffUTXOSet {
+	return &DiffUTXOSet{
+		base:     base,
+		UTXODiff: diff,
+	}
+}
+
+// diffFrom returns the difference between this utxoSet and another.
+// diffFrom can work if other is this's base fullUTXOSet, or a diffUTXOSet with the same base as this
+func (dus *DiffUTXOSet) diffFrom(other UTXOSet) (*UTXODiff, error) {
+	otherDiffSet, ok := other.(*DiffUTXOSet)
+	if !ok {
+		return nil, errors.New("can't diffFrom diffUTXOSet with fullUTXOSet")
+	}
+
+	if otherDiffSet.base != dus.base {
+		return nil, errors.New("can't diffFrom with another diffUTXOSet with a different base")
+	}
+
+	return dus.UTXODiff.diffFrom(otherDiffSet.UTXODiff)
+}
+
+// WithDiff return a new utxoSet which is a diffFrom between this and another utxoSet
+func (dus *DiffUTXOSet) WithDiff(other *UTXODiff) (UTXOSet, error) {
+	diff, err := dus.UTXODiff.WithDiff(other)
+	if err != nil {
+		return nil, err
+	}
+
+	return NewDiffUTXOSet(dus.base, diff), nil
+}
+
+// AddTx adds a transaction to this utxoSet and returns true iff it's valid in this UTXO's context.
+//
+// If dus.UTXODiff.useMultiset is true, this function MUST be
+// called with the DAG lock held.
+func (dus *DiffUTXOSet) AddTx(tx *wire.MsgTx, blockBlueScore uint64) (bool, error) {
+	isCoinbase := tx.IsCoinBase()
+	if !isCoinbase && !dus.containsInputs(tx) {
+		return false, nil
+	}
+
+	err := dus.appendTx(tx, blockBlueScore, isCoinbase)
+	if err != nil {
+		return false, err
+	}
+
+	return true, nil
+}
+
+func (dus *DiffUTXOSet) appendTx(tx *wire.MsgTx, blockBlueScore uint64, isCoinbase bool) error {
+	if !isCoinbase {
+		for _, txIn := range tx.TxIn {
+			outpoint := *wire.NewOutpoint(&txIn.PreviousOutpoint.TxID, txIn.PreviousOutpoint.Index)
+			entry, ok := dus.Get(outpoint)
+			if !ok {
+				return errors.Errorf("Couldn't find entry for outpoint %s", outpoint)
+			}
+			err := dus.UTXODiff.RemoveEntry(outpoint, entry)
+			if err != nil {
+				return err
+			}
+		}
+	}
+
+	for i, txOut := range tx.TxOut {
+		outpoint := *wire.NewOutpoint(tx.TxID(), uint32(i))
+		entry := NewUTXOEntry(txOut, isCoinbase, blockBlueScore)
+
+		err := dus.UTXODiff.AddEntry(outpoint, entry)
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (dus *DiffUTXOSet) containsInputs(tx *wire.MsgTx) bool {
+	for _, txIn := range tx.TxIn {
+		outpoint := *wire.NewOutpoint(&txIn.PreviousOutpoint.TxID, txIn.PreviousOutpoint.Index)
+		isInBase := dus.base.contains(outpoint)
+		isInDiffToAdd := dus.UTXODiff.toAdd.contains(outpoint)
+		isInDiffToRemove := dus.UTXODiff.toRemove.contains(outpoint)
+		if (!isInBase && !isInDiffToAdd) || (isInDiffToRemove && !(isInBase && isInDiffToAdd)) {
+			return false
+		}
+	}
+
+	return true
+}
+
+// meldToBase updates the base fullUTXOSet with all changes in diff
+func (dus *DiffUTXOSet) meldToBase() error {
+	for outpoint := range dus.UTXODiff.toRemove {
+		if _, ok := dus.base.Get(outpoint); ok {
+			dus.base.remove(outpoint)
+		} else {
+			return errors.Errorf("Couldn't remove outpoint %s because it doesn't exist in the DiffUTXOSet base", outpoint)
+		}
+	}
+
+	for outpoint, utxoEntry := range dus.UTXODiff.toAdd {
+		dus.base.add(outpoint, utxoEntry)
+	}
+
+	if dus.UTXODiff.useMultiset {
+		dus.base.UTXOMultiset = dus.base.UTXOMultiset.Union(dus.UTXODiff.diffMultiset)
+	}
+
+	if dus.UTXODiff.useMultiset {
+		dus.UTXODiff = NewUTXODiff()
+	} else {
+		dus.UTXODiff = NewUTXODiffWithoutMultiset()
+	}
+	return nil
+}
+
+// diffFromTx returns a diff that is equivalent to provided transaction,
+// or an error if provided transaction is not valid in the context of this UTXOSet
+func (dus *DiffUTXOSet) diffFromTx(tx *wire.MsgTx, acceptingBlueScore uint64) (*UTXODiff, error) {
+	return diffFromTx(dus, tx, acceptingBlueScore)
+}
+
+func (dus *DiffUTXOSet) diffFromAcceptedTx(tx *wire.MsgTx, acceptingBlueScore uint64) (*UTXODiff, error) {
+	return diffFromAcceptedTx(dus, tx, acceptingBlueScore)
+}
+
+func (dus *DiffUTXOSet) String() string {
+	return fmt.Sprintf("{Base: %s, To Add: %s, To Remove: %s, Multiset-Hash:%s}", dus.base, dus.UTXODiff.toAdd, dus.UTXODiff.toRemove, dus.Multiset().Hash())
+}
+
+// clone returns a clone of this UTXO Set
+func (dus *DiffUTXOSet) clone() UTXOSet {
+	return NewDiffUTXOSet(dus.base.clone().(*FullUTXOSet), dus.UTXODiff.clone())
+}
+
+// Get returns the UTXOEntry associated with provided outpoint in this UTXOSet.
+// Returns false in second output if this UTXOEntry was not found
+func (dus *DiffUTXOSet) Get(outpoint wire.Outpoint) (*UTXOEntry, bool) {
+	if toRemoveEntry, ok := dus.UTXODiff.toRemove.get(outpoint); ok {
+		// An exception is made for entries with unequal blue scores
+		// These are just "updates" to accepted blue score
+		if toAddEntry, ok := dus.UTXODiff.toAdd.get(outpoint); ok && toAddEntry.blockBlueScore != toRemoveEntry.blockBlueScore {
+			return toAddEntry, true
+		}
+		return nil, false
+	}
+	if txOut, ok := dus.base.get(outpoint); ok {
+		return txOut, true
+	}
+	txOut, ok := dus.UTXODiff.toAdd.get(outpoint)
+	return txOut, ok
+}
+
+// Multiset returns the ecmh-Multiset of this utxoSet
+func (dus *DiffUTXOSet) Multiset() *ecc.Multiset {
+	return dus.base.UTXOMultiset.Union(dus.UTXODiff.diffMultiset)
+}
+
+// WithTransactions returns a new UTXO Set with the added transactions.
+//
+// If dus.UTXODiff.useMultiset is true, this function MUST be
+// called with the DAG lock held.
+func (dus *DiffUTXOSet) WithTransactions(transactions []*wire.MsgTx, blockBlueScore uint64, ignoreDoubleSpends bool) (UTXOSet, error) {
+	diffSet := NewDiffUTXOSet(dus.base, dus.UTXODiff.clone())
+	for _, tx := range transactions {
+		isAccepted, err := diffSet.AddTx(tx, blockBlueScore)
+		if err != nil {
+			return nil, err
+		}
+		if !ignoreDoubleSpends && !isAccepted {
+			return nil, errors.Errorf("Transaction %s is not valid with the current UTXO set", tx.TxID())
+		}
+	}
+	return UTXOSet(diffSet), nil
+}
+
+func addUTXOToMultiset(ms *ecc.Multiset, entry *UTXOEntry, outpoint *wire.Outpoint) (*ecc.Multiset, error) {
+	utxoMS, err := utxoMultiset(entry, outpoint)
+	if err != nil {
+		return nil, err
+	}
+	return ms.Union(utxoMS), nil
+}
+
+func removeUTXOFromMultiset(ms *ecc.Multiset, entry *UTXOEntry, outpoint *wire.Outpoint) (*ecc.Multiset, error) {
+	utxoMS, err := utxoMultiset(entry, outpoint)
+	if err != nil {
+		return nil, err
+	}
+	return ms.Subtract(utxoMS), nil
+}

blockdag/validate.go

@@ -0,0 +1,981 @@
+// Copyright (c) 2013-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"fmt"
+	"math"
+	"sort"
+	"time"
+
+	"github.com/pkg/errors"
+
+	"github.com/kaspanet/kaspad/dagconfig"
+	"github.com/kaspanet/kaspad/txscript"
+	"github.com/kaspanet/kaspad/util"
+	"github.com/kaspanet/kaspad/util/daghash"
+	"github.com/kaspanet/kaspad/util/subnetworkid"
+	"github.com/kaspanet/kaspad/wire"
+)
+
+const (
+	// MaxCoinbasePayloadLen is the maximum length a coinbase payload can be.
+	MaxCoinbasePayloadLen = 150
+
+	// baseSubsidy is the starting subsidy amount for mined blocks. This
+	// value is halved every SubsidyHalvingInterval blocks.
+	baseSubsidy = 50 * util.SompiPerKaspa
+
+	// the following are used when calculating a transaction's mass
+
+	// MassPerTxByte is the number of grams that any byte
+	// adds to a transaction.
+	MassPerTxByte = 1
+
+	// MassPerScriptPubKeyByte is the number of grams that any
+	// scriptPubKey byte adds to a transaction.
+	MassPerScriptPubKeyByte = 10
+
+	// MassPerSigOp is the number of grams that any
+	// signature operation adds to a transaction.
+	MassPerSigOp = 10000
+)
+
+// isNullOutpoint determines whether or not a previous transaction outpoint
+// is set.
+func isNullOutpoint(outpoint *wire.Outpoint) bool {
+	if outpoint.Index == math.MaxUint32 && outpoint.TxID == daghash.ZeroTxID {
+		return true
+	}
+	return false
+}
+
+// SequenceLockActive determines if a transaction's sequence locks have been
+// met, meaning that all the inputs of a given transaction have reached a
+// blue score or time sufficient for their relative lock-time maturity.
+func SequenceLockActive(sequenceLock *SequenceLock, blockBlueScore uint64,
+	medianTimePast time.Time) bool {
+
+	// If either the seconds, or blue score relative-lock time has not yet
+	// reached, then the transaction is not yet mature according to its
+	// sequence locks.
+	if sequenceLock.Seconds >= medianTimePast.Unix() ||
+		sequenceLock.BlockBlueScore >= int64(blockBlueScore) {
+		return false
+	}
+
+	return true
+}
+
+// IsFinalizedTransaction determines whether or not a transaction is finalized.
+func IsFinalizedTransaction(tx *util.Tx, blockBlueScore uint64, blockTime time.Time) bool {
+	msgTx := tx.MsgTx()
+
+	// Lock time of zero means the transaction is finalized.
+	lockTime := msgTx.LockTime
+	if lockTime == 0 {
+		return true
+	}
+
+	// The lock time field of a transaction is either a block blue score at
+	// which the transaction is finalized or a timestamp depending on if the
+	// value is before the txscript.LockTimeThreshold. When it is under the
+	// threshold it is a block blue score.
+	blockTimeOrBlueScore := int64(0)
+	if lockTime < txscript.LockTimeThreshold {
+		blockTimeOrBlueScore = int64(blockBlueScore)
+	} else {
+		blockTimeOrBlueScore = blockTime.Unix()
+	}
+	if int64(lockTime) < blockTimeOrBlueScore {
+		return true
+	}
+
+	// At this point, the transaction's lock time hasn't occurred yet, but
+	// the transaction might still be finalized if the sequence number
+	// for all transaction inputs is maxed out.
+	for _, txIn := range msgTx.TxIn {
+		if txIn.Sequence != math.MaxUint64 {
+			return false
+		}
+	}
+	return true
+}
+
+// CalcBlockSubsidy returns the subsidy amount a block at the provided blue score
+// should have. This is mainly used for determining how much the coinbase for
+// newly generated blocks awards as well as validating the coinbase for blocks
+// has the expected value.
+//
+// The subsidy is halved every SubsidyReductionInterval blocks. Mathematically
+// this is: baseSubsidy / 2^(blueScore/SubsidyReductionInterval)
+//
+// At the target block generation rate for the main network, this is
+// approximately every 4 years.
+func CalcBlockSubsidy(blueScore uint64, dagParams *dagconfig.Params) uint64 {
+	if dagParams.SubsidyReductionInterval == 0 {
+		return baseSubsidy
+	}
+
+	// Equivalent to: baseSubsidy / 2^(blueScore/subsidyHalvingInterval)
+	return baseSubsidy >> uint(blueScore/dagParams.SubsidyReductionInterval)
+}
+
+// CheckTransactionSanity performs some preliminary checks on a transaction to
+// ensure it is sane. These checks are context free.
+func CheckTransactionSanity(tx *util.Tx, subnetworkID *subnetworkid.SubnetworkID) error {
+	isCoinbase := tx.IsCoinBase()
+	// A transaction must have at least one input.
+	msgTx := tx.MsgTx()
+	if !isCoinbase && len(msgTx.TxIn) == 0 {
+		return ruleError(ErrNoTxInputs, "transaction has no inputs")
+	}
+
+	// A transaction must not exceed the maximum allowed block mass when
+	// serialized.
+	serializedTxSize := msgTx.SerializeSize()
+	if serializedTxSize*MassPerTxByte > wire.MaxMassPerTx {
+		str := fmt.Sprintf("serialized transaction is too big - got "+
+			"%d, max %d", serializedTxSize, wire.MaxMassPerBlock)
+		return ruleError(ErrTxMassTooHigh, str)
+	}
+
+	// Ensure the transaction amounts are in range. Each transaction
+	// output must not be negative or more than the max allowed per
+	// transaction. Also, the total of all outputs must abide by the same
+	// restrictions. All amounts in a transaction are in a unit value known
+	// as a sompi. One kaspa is a quantity of sompi as defined by the
+	// SompiPerKaspa constant.
+	var totalSompi uint64
+	for _, txOut := range msgTx.TxOut {
+		sompi := txOut.Value
+		if sompi > util.MaxSompi {
+			str := fmt.Sprintf("transaction output value of %d is "+
+				"higher than max allowed value of %d", sompi,
+				util.MaxSompi)
+			return ruleError(ErrBadTxOutValue, str)
+		}
+
+		// Binary arithmetic guarantees that any overflow is detected and reported.
+		// This is impossible for Kaspa, but perhaps possible if an alt increases
+		// the total money supply.
+		newTotalSompi := totalSompi + sompi
+		if newTotalSompi < totalSompi {
+			str := fmt.Sprintf("total value of all transaction "+
+				"outputs exceeds max allowed value of %d",
+				util.MaxSompi)
+			return ruleError(ErrBadTxOutValue, str)
+		}
+		totalSompi = newTotalSompi
+		if totalSompi > util.MaxSompi {
+			str := fmt.Sprintf("total value of all transaction "+
+				"outputs is %d which is higher than max "+
+				"allowed value of %d", totalSompi,
+				util.MaxSompi)
+			return ruleError(ErrBadTxOutValue, str)
+		}
+	}
+
+	// Check for duplicate transaction inputs.
+	existingTxOut := make(map[wire.Outpoint]struct{})
+	for _, txIn := range msgTx.TxIn {
+		if _, exists := existingTxOut[txIn.PreviousOutpoint]; exists {
+			return ruleError(ErrDuplicateTxInputs, "transaction "+
+				"contains duplicate inputs")
+		}
+		existingTxOut[txIn.PreviousOutpoint] = struct{}{}
+	}
+
+	// Coinbase payload length must not exceed the max length.
+	if isCoinbase {
+		payloadLen := len(msgTx.Payload)
+		if payloadLen > MaxCoinbasePayloadLen {
+			str := fmt.Sprintf("coinbase transaction payload length "+
+				"of %d is out of range (max: %d)",
+				payloadLen, MaxCoinbasePayloadLen)
+			return ruleError(ErrBadCoinbasePayloadLen, str)
+		}
+	} else {
+		// Previous transaction outputs referenced by the inputs to this
+		// transaction must not be null.
+		for _, txIn := range msgTx.TxIn {
+			if isNullOutpoint(&txIn.PreviousOutpoint) {
+				return ruleError(ErrBadTxInput, "transaction "+
+					"input refers to previous output that "+
+					"is null")
+			}
+		}
+	}
+
+	// Check payload's hash
+	if !msgTx.SubnetworkID.IsEqual(subnetworkid.SubnetworkIDNative) {
+		payloadHash := daghash.DoubleHashH(msgTx.Payload)
+		if !msgTx.PayloadHash.IsEqual(&payloadHash) {
+			return ruleError(ErrInvalidPayloadHash, "invalid payload hash")
+		}
+	} else if msgTx.PayloadHash != nil {
+		return ruleError(ErrInvalidPayloadHash, "unexpected non-empty payload hash in native subnetwork")
+	}
+
+	// Transactions in native, registry and coinbase subnetworks must have Gas = 0
+	if (msgTx.SubnetworkID.IsEqual(subnetworkid.SubnetworkIDNative) ||
+		msgTx.SubnetworkID.IsBuiltIn()) &&
+		msgTx.Gas > 0 {
+
+		return ruleError(ErrInvalidGas, "transaction in the native or "+
+			"registry subnetworks has gas > 0 ")
+	}
+
+	if msgTx.SubnetworkID.IsEqual(subnetworkid.SubnetworkIDRegistry) {
+		err := validateSubnetworkRegistryTransaction(msgTx)
+		if err != nil {
+			return err
+		}
+	}
+
+	if msgTx.SubnetworkID.IsEqual(subnetworkid.SubnetworkIDNative) &&
+		len(msgTx.Payload) > 0 {
+
+		return ruleError(ErrInvalidPayload,
+			"transaction in the native subnetwork includes a payload")
+	}
+
+	// If we are a partial node, only transactions on built in subnetworks
+	// or our own subnetwork may have a payload
+	isLocalNodeFull := subnetworkID == nil
+	shouldTxBeFull := msgTx.SubnetworkID.IsBuiltIn() ||
+		msgTx.SubnetworkID.IsEqual(subnetworkID)
+	if !isLocalNodeFull && !shouldTxBeFull && len(msgTx.Payload) > 0 {
+		return ruleError(ErrInvalidPayload,
+			"transaction that was expected to be partial has a payload "+
+				"with length > 0")
+	}
+
+	return nil
+}
+
+// checkProofOfWork ensures the block header bits which indicate the target
+// difficulty is in min/max range and that the block hash is less than the
+// target difficulty as claimed.
+//
+// The flags modify the behavior of this function as follows:
+//  - BFNoPoWCheck: The check to ensure the block hash is less than the target
+//    difficulty is not performed.
+func (dag *BlockDAG) checkProofOfWork(header *wire.BlockHeader, flags BehaviorFlags) error {
+	// The target difficulty must be larger than zero.
+	target := util.CompactToBig(header.Bits)
+	if target.Sign() <= 0 {
+		str := fmt.Sprintf("block target difficulty of %064x is too low",
+			target)
+		return ruleError(ErrUnexpectedDifficulty, str)
+	}
+
+	// The target difficulty must be less than the maximum allowed.
+	if target.Cmp(dag.dagParams.PowMax) > 0 {
+		str := fmt.Sprintf("block target difficulty of %064x is "+
+			"higher than max of %064x", target, dag.dagParams.PowMax)
+		return ruleError(ErrUnexpectedDifficulty, str)
+	}
+
+	// The block hash must be less than the claimed target unless the flag
+	// to avoid proof of work checks is set.
+	if flags&BFNoPoWCheck != BFNoPoWCheck {
+		// The block hash must be less than the claimed target.
+		hash := header.BlockHash()
+		hashNum := daghash.HashToBig(hash)
+		if hashNum.Cmp(target) > 0 {
+			str := fmt.Sprintf("block hash of %064x is higher than "+
+				"expected max of %064x", hashNum, target)
+			return ruleError(ErrHighHash, str)
+		}
+	}
+
+	return nil
+}
+
+// ValidateTxMass makes sure that the given transaction's mass does not exceed
+// the maximum allowed limit. Currently, it is equivalent to the block mass limit.
+// See CalcTxMass for further details.
+func ValidateTxMass(tx *util.Tx, utxoSet UTXOSet) error {
+	txMass, err := CalcTxMassFromUTXOSet(tx, utxoSet)
+	if err != nil {
+		return err
+	}
+	if txMass > wire.MaxMassPerBlock {
+		str := fmt.Sprintf("tx %s has mass %d, which is above the "+
+			"allowed limit of %d", tx.ID(), txMass, wire.MaxMassPerBlock)
+		return ruleError(ErrTxMassTooHigh, str)
+	}
+	return nil
+}
+
+func validateBlockMass(pastUTXO UTXOSet, transactions []*util.Tx) error {
+	_, err := CalcBlockMass(pastUTXO, transactions)
+	return err
+}
+
+// CalcBlockMass sums up and returns the "mass" of a block. See CalcTxMass
+// for further details.
+func CalcBlockMass(pastUTXO UTXOSet, transactions []*util.Tx) (uint64, error) {
+	totalMass := uint64(0)
+	for _, tx := range transactions {
+		txMass, err := CalcTxMassFromUTXOSet(tx, pastUTXO)
+		if err != nil {
+			return 0, err
+		}
+		totalMass += txMass
+
+		// We could potentially overflow the accumulator so check for
+		// overflow as well.
+		if totalMass < txMass || totalMass > wire.MaxMassPerBlock {
+			str := fmt.Sprintf("block has total mass %d, which is "+
+				"above the allowed limit of %d", totalMass, wire.MaxMassPerBlock)
+			return 0, ruleError(ErrBlockMassTooHigh, str)
+		}
+	}
+	return totalMass, nil
+}
+
+// CalcTxMassFromUTXOSet calculates the transaction mass based on the
+// UTXO set in its past.
+//
+// See CalcTxMass for more details.
+func CalcTxMassFromUTXOSet(tx *util.Tx, utxoSet UTXOSet) (uint64, error) {
+	if tx.IsCoinBase() {
+		return CalcTxMass(tx, nil), nil
+	}
+	previousScriptPubKeys := make([][]byte, len(tx.MsgTx().TxIn))
+	for txInIndex, txIn := range tx.MsgTx().TxIn {
+		entry, ok := utxoSet.Get(txIn.PreviousOutpoint)
+		if !ok {
+			str := fmt.Sprintf("output %s referenced from "+
+				"transaction %s input %d either does not exist or "+
+				"has already been spent", txIn.PreviousOutpoint,
+				tx.ID(), txInIndex)
+			return 0, ruleError(ErrMissingTxOut, str)
+		}
+		previousScriptPubKeys[txInIndex] = entry.ScriptPubKey()
+	}
+	return CalcTxMass(tx, previousScriptPubKeys), nil
+}
+
+// CalcTxMass sums up and returns the "mass" of a transaction. This number
+// is an approximation of how many resources (CPU, RAM, etc.) it would take
+// to process the transaction.
+// The following properties are considered in the calculation:
+// * The transaction length in bytes
+// * The length of all output scripts in bytes
+// * The count of all input sigOps
+func CalcTxMass(tx *util.Tx, previousScriptPubKeys [][]byte) uint64 {
+	txSize := tx.MsgTx().SerializeSize()
+
+	if tx.IsCoinBase() {
+		return uint64(txSize * MassPerTxByte)
+	}
+
+	scriptPubKeySize := 0
+	for _, txOut := range tx.MsgTx().TxOut {
+		scriptPubKeySize += len(txOut.ScriptPubKey)
+	}
+
+	sigOpsCount := 0
+	for txInIndex, txIn := range tx.MsgTx().TxIn {
+		// Count the precise number of signature operations in the
+		// referenced public key script.
+		sigScript := txIn.SignatureScript
+		isP2SH := txscript.IsPayToScriptHash(previousScriptPubKeys[txInIndex])
+		sigOpsCount += txscript.GetPreciseSigOpCount(sigScript, previousScriptPubKeys[txInIndex], isP2SH)
+	}
+
+	return uint64(txSize*MassPerTxByte +
+		scriptPubKeySize*MassPerScriptPubKeyByte +
+		sigOpsCount*MassPerSigOp)
+}
+
+// checkBlockHeaderSanity performs some preliminary checks on a block header to
+// ensure it is sane before continuing with processing. These checks are
+// context free.
+//
+// The flags do not modify the behavior of this function directly, however they
+// are needed to pass along to checkProofOfWork.
+func (dag *BlockDAG) checkBlockHeaderSanity(header *wire.BlockHeader, flags BehaviorFlags) (delay time.Duration, err error) {
+	// Ensure the proof of work bits in the block header is in min/max range
+	// and the block hash is less than the target value described by the
+	// bits.
+	err = dag.checkProofOfWork(header, flags)
+	if err != nil {
+		return 0, err
+	}
+
+	if len(header.ParentHashes) == 0 {
+		if !header.BlockHash().IsEqual(dag.dagParams.GenesisHash) {
+			return 0, ruleError(ErrNoParents, "block has no parents")
+		}
+	} else {
+		err = checkBlockParentsOrder(header)
+		if err != nil {
+			return 0, err
+		}
+	}
+
+	// A block timestamp must not have a greater precision than one second.
+	// This check is necessary because Go time.Time values support
+	// nanosecond precision whereas the consensus rules only apply to
+	// seconds and it's much nicer to deal with standard Go time values
+	// instead of converting to seconds everywhere.
+	if !header.Timestamp.Equal(time.Unix(header.Timestamp.Unix(), 0)) {
+		str := fmt.Sprintf("block timestamp of %s has a higher "+
+			"precision than one second", header.Timestamp)
+		return 0, ruleError(ErrInvalidTime, str)
+	}
+
+	// Ensure the block time is not too far in the future. If it's too far, return
+	// the duration of time that should be waited before the block becomes valid.
+	// This check needs to be last as it does not return an error but rather marks the
+	// header as delayed (and valid).
+	maxTimestamp := dag.AdjustedTime().Add(time.Second *
+		time.Duration(int64(dag.TimestampDeviationTolerance)*dag.targetTimePerBlock))
+	if header.Timestamp.After(maxTimestamp) {
+		return header.Timestamp.Sub(maxTimestamp), nil
+	}
+
+	return 0, nil
+}
+
+//checkBlockParentsOrder ensures that the block's parents are ordered by hash
+func checkBlockParentsOrder(header *wire.BlockHeader) error {
+	sortedHashes := make([]*daghash.Hash, 0, header.NumParentBlocks())
+	for _, hash := range header.ParentHashes {
+		sortedHashes = append(sortedHashes, hash)
+	}
+	sort.Slice(sortedHashes, func(i, j int) bool {
+		return daghash.Less(sortedHashes[i], sortedHashes[j])
+	})
+	if !daghash.AreEqual(header.ParentHashes, sortedHashes) {
+		return ruleError(ErrWrongParentsOrder, "block parents are not ordered by hash")
+	}
+	return nil
+}
+
+// checkBlockSanity performs some preliminary checks on a block to ensure it is
+// sane before continuing with block processing. These checks are context free.
+//
+// The flags do not modify the behavior of this function directly, however they
+// are needed to pass along to checkBlockHeaderSanity.
+func (dag *BlockDAG) checkBlockSanity(block *util.Block, flags BehaviorFlags) (time.Duration, error) {
+	msgBlock := block.MsgBlock()
+	header := &msgBlock.Header
+	delay, err := dag.checkBlockHeaderSanity(header, flags)
+	if err != nil {
+		return 0, err
+	}
+
+	// A block must have at least one transaction.
+	numTx := len(msgBlock.Transactions)
+	if numTx == 0 {
+		return 0, ruleError(ErrNoTransactions, "block does not contain "+
+			"any transactions")
+	}
+
+	// A block must not have more transactions than the max block mass or
+	// else it is certainly over the block mass limit.
+	if numTx > wire.MaxMassPerBlock {
+		str := fmt.Sprintf("block contains too many transactions - "+
+			"got %d, max %d", numTx, wire.MaxMassPerBlock)
+		return 0, ruleError(ErrBlockMassTooHigh, str)
+	}
+
+	// The first transaction in a block must be a coinbase.
+	transactions := block.Transactions()
+	if !transactions[util.CoinbaseTransactionIndex].IsCoinBase() {
+		return 0, ruleError(ErrFirstTxNotCoinbase, "first transaction in "+
+			"block is not a coinbase")
+	}
+
+	txOffset := util.CoinbaseTransactionIndex + 1
+
+	// A block must not have more than one coinbase. And transactions must be
+	// ordered by subnetwork
+	for i, tx := range transactions[txOffset:] {
+		if tx.IsCoinBase() {
+			str := fmt.Sprintf("block contains second coinbase at "+
+				"index %d", i+2)
+			return 0, ruleError(ErrMultipleCoinbases, str)
+		}
+		if i != 0 && subnetworkid.Less(&tx.MsgTx().SubnetworkID, &transactions[i].MsgTx().SubnetworkID) {
+			return 0, ruleError(ErrTransactionsNotSorted, "transactions must be sorted by subnetwork")
+		}
+	}
+
+	// Do some preliminary checks on each transaction to ensure they are
+	// sane before continuing.
+	for _, tx := range transactions {
+		err := CheckTransactionSanity(tx, dag.subnetworkID)
+		if err != nil {
+			return 0, err
+		}
+	}
+
+	// Build merkle tree and ensure the calculated merkle root matches the
+	// entry in the block header. This also has the effect of caching all
+	// of the transaction hashes in the block to speed up future hash
+	// checks.
+	hashMerkleTree := BuildHashMerkleTreeStore(block.Transactions())
+	calculatedHashMerkleRoot := hashMerkleTree.Root()
+	if !header.HashMerkleRoot.IsEqual(calculatedHashMerkleRoot) {
+		str := fmt.Sprintf("block hash merkle root is invalid - block "+
+			"header indicates %s, but calculated value is %s",
+			header.HashMerkleRoot, calculatedHashMerkleRoot)
+		return 0, ruleError(ErrBadMerkleRoot, str)
+	}
+
+	// Check for duplicate transactions. This check will be fairly quick
+	// since the transaction IDs are already cached due to building the
+	// merkle tree above.
+	existingTxIDs := make(map[daghash.TxID]struct{})
+	for _, tx := range transactions {
+		id := tx.ID()
+		if _, exists := existingTxIDs[*id]; exists {
+			str := fmt.Sprintf("block contains duplicate "+
+				"transaction %s", id)
+			return 0, ruleError(ErrDuplicateTx, str)
+		}
+		existingTxIDs[*id] = struct{}{}
+	}
+
+	return delay, nil
+}
+
+// checkBlockHeaderContext performs several validation checks on the block header
+// which depend on its position within the block dag.
+//
+// The flags modify the behavior of this function as follows:
+//  - BFFastAdd: No checks are performed.
+//
+// This function MUST be called with the dag state lock held (for writes).
+func (dag *BlockDAG) checkBlockHeaderContext(header *wire.BlockHeader, bluestParent *blockNode, fastAdd bool) error {
+	if !fastAdd {
+		if err := dag.validateDifficulty(header, bluestParent); err != nil {
+			return err
+		}
+
+		if err := validateMedianTime(dag, header, bluestParent); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func validateMedianTime(dag *BlockDAG, header *wire.BlockHeader, bluestParent *blockNode) error {
+	if !header.IsGenesis() {
+		// Ensure the timestamp for the block header is not before the
+		// median time of the last several blocks (medianTimeBlocks).
+		medianTime := bluestParent.PastMedianTime(dag)
+		if header.Timestamp.Before(medianTime) {
+			str := fmt.Sprintf("block timestamp of %s is not after expected %s", header.Timestamp, medianTime)
+			return ruleError(ErrTimeTooOld, str)
+		}
+	}
+
+	return nil
+}
+
+func (dag *BlockDAG) validateDifficulty(header *wire.BlockHeader, bluestParent *blockNode) error {
+	// Ensure the difficulty specified in the block header matches
+	// the calculated difficulty based on the previous block and
+	// difficulty retarget rules.
+	expectedDifficulty := dag.requiredDifficulty(bluestParent,
+		header.Timestamp)
+	blockDifficulty := header.Bits
+	if blockDifficulty != expectedDifficulty {
+		str := fmt.Sprintf("block difficulty of %d is not the expected value of %d", blockDifficulty, expectedDifficulty)
+		return ruleError(ErrUnexpectedDifficulty, str)
+	}
+
+	return nil
+}
+
+// validateParents validates that no parent is an ancestor of another parent, and no parent is finalized
+func validateParents(blockHeader *wire.BlockHeader, parents blockSet) error {
+	minBlueScore := uint64(math.MaxUint64)
+	queue := newDownHeap()
+	visited := newSet()
+	for parent := range parents {
+		// isFinalized might be false-negative because node finality status is
+		// updated in a separate goroutine. This is why later the block is
+		// checked more thoroughly on the finality rules in dag.checkFinalityRules.
+		if parent.isFinalized {
+			return ruleError(ErrFinality, fmt.Sprintf("block %s is a finalized parent of block %s", parent.hash, blockHeader.BlockHash()))
+		}
+		if parent.blueScore < minBlueScore {
+			minBlueScore = parent.blueScore
+		}
+		for grandParent := range parent.parents {
+			if !visited.contains(grandParent) {
+				queue.Push(grandParent)
+				visited.add(grandParent)
+			}
+		}
+	}
+	for queue.Len() > 0 {
+		current := queue.pop()
+		if parents.contains(current) {
+			return ruleError(ErrInvalidParentsRelation, fmt.Sprintf("block %s is both a parent of %s and an"+
+				" ancestor of another parent",
+				current.hash,
+				blockHeader.BlockHash()))
+		}
+		if current.blueScore > minBlueScore {
+			for parent := range current.parents {
+				if !visited.contains(parent) {
+					queue.Push(parent)
+					visited.add(parent)
+				}
+			}
+		}
+	}
+	return nil
+}
+
+// checkBlockContext peforms several validation checks on the block which depend
+// on its position within the block DAG.
+//
+// The flags modify the behavior of this function as follows:
+//  - BFFastAdd: The transaction are not checked to see if they are finalized
+//    and the somewhat expensive BIP0034 validation is not performed.
+//
+// The flags are also passed to checkBlockHeaderContext. See its documentation
+// for how the flags modify its behavior.
+//
+// This function MUST be called with the dag state lock held (for writes).
+func (dag *BlockDAG) checkBlockContext(block *util.Block, parents blockSet, flags BehaviorFlags) error {
+	bluestParent := parents.bluest()
+	fastAdd := flags&BFFastAdd == BFFastAdd
+
+	err := validateParents(&block.MsgBlock().Header, parents)
+	if err != nil {
+		return err
+	}
+
+	// Perform all block header related validation checks.
+	header := &block.MsgBlock().Header
+	if err = dag.checkBlockHeaderContext(header, bluestParent, fastAdd); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (dag *BlockDAG) validateAllTxsFinalized(block *util.Block, node *blockNode, bluestParent *blockNode) error {
+	blockTime := block.MsgBlock().Header.Timestamp
+	if !block.IsGenesis() {
+		blockTime = bluestParent.PastMedianTime(dag)
+	}
+
+	// Ensure all transactions in the block are finalized.
+	for _, tx := range block.Transactions() {
+		if !IsFinalizedTransaction(tx, node.blueScore, blockTime) {
+			str := fmt.Sprintf("block contains unfinalized "+
+				"transaction %s", tx.ID())
+			return ruleError(ErrUnfinalizedTx, str)
+		}
+	}
+
+	return nil
+}
+
+// ensureNoDuplicateTx ensures blocks do not contain duplicate transactions which
+// 'overwrite' older transactions that are not fully spent. This prevents an
+// attack where a coinbase and all of its dependent transactions could be
+// duplicated to effectively revert the overwritten transactions to a single
+// confirmation thereby making them vulnerable to a double spend.
+//
+// For more details, see http://r6.ca/blog/20120206T005236Z.html.
+//
+// This function MUST be called with the dag state lock held (for reads).
+func ensureNoDuplicateTx(utxoSet UTXOSet, transactions []*util.Tx) error {
+	// Fetch utxos for all of the transaction ouputs in this block.
+	// Typically, there will not be any utxos for any of the outputs.
+	fetchSet := make(map[wire.Outpoint]struct{})
+	for _, tx := range transactions {
+		prevOut := wire.Outpoint{TxID: *tx.ID()}
+		for txOutIdx := range tx.MsgTx().TxOut {
+			prevOut.Index = uint32(txOutIdx)
+			fetchSet[prevOut] = struct{}{}
+		}
+	}
+
+	// Duplicate transactions are only allowed if the previous transaction
+	// is fully spent.
+	for outpoint := range fetchSet {
+		if _, ok := utxoSet.Get(outpoint); ok {
+			str := fmt.Sprintf("tried to overwrite transaction %s "+
+				"that is not fully spent", outpoint.TxID)
+			return ruleError(ErrOverwriteTx, str)
+		}
+	}
+
+	return nil
+}
+
+// CheckTransactionInputsAndCalulateFee performs a series of checks on the inputs to a
+// transaction to ensure they are valid. An example of some of the checks
+// include verifying all inputs exist, ensuring the block reward seasoning
+// requirements are met, detecting double spends, validating all values and fees
+// are in the legal range and the total output amount doesn't exceed the input
+// amount. As it checks the inputs, it also calculates the total fees for the
+// transaction and returns that value.
+//
+// NOTE: The transaction MUST have already been sanity checked with the
+// CheckTransactionSanity function prior to calling this function.
+func CheckTransactionInputsAndCalulateFee(tx *util.Tx, txBlueScore uint64, utxoSet UTXOSet, dagParams *dagconfig.Params, fastAdd bool) (
+	txFeeInSompi uint64, err error) {
+
+	// Coinbase transactions have no standard inputs to validate.
+	if tx.IsCoinBase() {
+		return 0, nil
+	}
+
+	txID := tx.ID()
+	var totalSompiIn uint64
+	for txInIndex, txIn := range tx.MsgTx().TxIn {
+		// Ensure the referenced input transaction is available.
+		entry, ok := utxoSet.Get(txIn.PreviousOutpoint)
+		if !ok {
+			str := fmt.Sprintf("output %s referenced from "+
+				"transaction %s input %d either does not exist or "+
+				"has already been spent", txIn.PreviousOutpoint,
+				tx.ID(), txInIndex)
+			return 0, ruleError(ErrMissingTxOut, str)
+		}
+
+		if !fastAdd {
+			if err = validateCoinbaseMaturity(dagParams, entry, txBlueScore, txIn); err != nil {
+				return 0, err
+			}
+		}
+
+		// Ensure the transaction amounts are in range. Each of the
+		// output values of the input transactions must not be negative
+		// or more than the max allowed per transaction. All amounts in
+		// a transaction are in a unit value known as a sompi. One
+		// kaspa is a quantity of sompi as defined by the
+		// SompiPerKaspa constant.
+		originTxSompi := entry.Amount()
+		if originTxSompi > util.MaxSompi {
+			str := fmt.Sprintf("transaction output value of %s is "+
+				"higher than max allowed value of %d",
+				util.Amount(originTxSompi),
+				util.MaxSompi)
+			return 0, ruleError(ErrBadTxOutValue, str)
+		}
+
+		// The total of all outputs must not be more than the max
+		// allowed per transaction. Also, we could potentially overflow
+		// the accumulator so check for overflow.
+		lastSompiIn := totalSompiIn
+		totalSompiIn += originTxSompi
+		if totalSompiIn < lastSompiIn ||
+			totalSompiIn > util.MaxSompi {
+			str := fmt.Sprintf("total value of all transaction "+
+				"inputs is %d which is higher than max "+
+				"allowed value of %d", totalSompiIn,
+				util.MaxSompi)
+			return 0, ruleError(ErrBadTxOutValue, str)
+		}
+	}
+
+	// Calculate the total output amount for this transaction. It is safe
+	// to ignore overflow and out of range errors here because those error
+	// conditions would have already been caught by checkTransactionSanity.
+	var totalSompiOut uint64
+	for _, txOut := range tx.MsgTx().TxOut {
+		totalSompiOut += txOut.Value
+	}
+
+	// Ensure the transaction does not spend more than its inputs.
+	if totalSompiIn < totalSompiOut {
+		str := fmt.Sprintf("total value of all transaction inputs for "+
+			"transaction %s is %d which is less than the amount "+
+			"spent of %d", txID, totalSompiIn, totalSompiOut)
+		return 0, ruleError(ErrSpendTooHigh, str)
+	}
+
+	txFeeInSompi = totalSompiIn - totalSompiOut
+	return txFeeInSompi, nil
+}
+
+func validateCoinbaseMaturity(dagParams *dagconfig.Params, entry *UTXOEntry, txBlueScore uint64, txIn *wire.TxIn) error {
+	// Ensure the transaction is not spending coins which have not
+	// yet reached the required coinbase maturity.
+	if entry.IsCoinbase() {
+		originBlueScore := entry.BlockBlueScore()
+		blueScoreSincePrev := txBlueScore - originBlueScore
+		if blueScoreSincePrev < dagParams.BlockCoinbaseMaturity {
+			str := fmt.Sprintf("tried to spend coinbase "+
+				"transaction output %s from blue score %d "+
+				"to blue score %d before required maturity "+
+				"of %d", txIn.PreviousOutpoint,
+				originBlueScore, txBlueScore,
+				dagParams.BlockCoinbaseMaturity)
+			return ruleError(ErrImmatureSpend, str)
+		}
+	}
+	return nil
+}
+
+// checkConnectToPastUTXO performs several checks to confirm connecting the passed
+// block to the DAG represented by the passed view does not violate any rules.
+//
+// An example of some of the checks performed are ensuring connecting the block
+// would not cause any duplicate transaction hashes for old transactions that
+// aren't already fully spent, double spends, exceeding the maximum allowed
+// signature operations per block, invalid values in relation to the expected
+// block subsidy, or fail transaction script validation.
+//
+// It also returns the feeAccumulator for this block.
+//
+// This function MUST be called with the dag state lock held (for writes).
+func (dag *BlockDAG) checkConnectToPastUTXO(block *blockNode, pastUTXO UTXOSet,
+	transactions []*util.Tx, fastAdd bool) (compactFeeData, error) {
+
+	if !fastAdd {
+		err := ensureNoDuplicateTx(pastUTXO, transactions)
+		if err != nil {
+			return nil, err
+		}
+
+		if err := validateBlockMass(pastUTXO, transactions); err != nil {
+			return nil, err
+		}
+	}
+
+	// Perform several checks on the inputs for each transaction. Also
+	// accumulate the total fees. This could technically be combined with
+	// the loop above instead of running another loop over the transactions,
+	// but by separating it we can avoid running the more expensive (though
+	// still relatively cheap as compared to running the scripts) checks
+	// against all the inputs when the signature operations are out of
+	// bounds.
+	// In addition - add all fees into a fee accumulator, to be stored and checked
+	// when validating descendants' coinbase transactions.
+	var totalFees uint64
+	compactFeeFactory := newCompactFeeFactory()
+
+	for _, tx := range transactions {
+		txFee, err := CheckTransactionInputsAndCalulateFee(tx, block.blueScore, pastUTXO,
+			dag.dagParams, fastAdd)
+		if err != nil {
+			return nil, err
+		}
+
+		// Sum the total fees and ensure we don't overflow the
+		// accumulator.
+		lastTotalFees := totalFees
+		totalFees += txFee
+		if totalFees < lastTotalFees {
+			return nil, ruleError(ErrBadFees, "total fees for block "+
+				"overflows accumulator")
+		}
+
+		err = compactFeeFactory.add(txFee)
+		if err != nil {
+			return nil, errors.Errorf("error adding tx %s fee to compactFeeFactory: %s", tx.ID(), err)
+		}
+	}
+	feeData, err := compactFeeFactory.data()
+	if err != nil {
+		return nil, errors.Errorf("error getting bytes of fee data: %s", err)
+	}
+
+	if !fastAdd {
+		scriptFlags := txscript.ScriptNoFlags
+
+		// We obtain the MTP of the *previous* block (unless it's genesis block)
+		// in order to determine if transactions in the current block are final.
+		medianTime := block.Header().Timestamp
+		if !block.isGenesis() {
+			medianTime = block.selectedParent.PastMedianTime(dag)
+		}
+
+		// We also enforce the relative sequence number based
+		// lock-times within the inputs of all transactions in this
+		// candidate block.
+		for _, tx := range transactions {
+			// A transaction can only be included within a block
+			// once the sequence locks of *all* its inputs are
+			// active.
+			sequenceLock, err := dag.calcSequenceLock(block, pastUTXO, tx, false)
+			if err != nil {
+				return nil, err
+			}
+			if !SequenceLockActive(sequenceLock, block.blueScore,
+				medianTime) {
+				str := fmt.Sprintf("block contains " +
+					"transaction whose input sequence " +
+					"locks are not met")
+				return nil, ruleError(ErrUnfinalizedTx, str)
+			}
+		}
+
+		// Now that the inexpensive checks are done and have passed, verify the
+		// transactions are actually allowed to spend the coins by running the
+		// expensive ECDSA signature check scripts. Doing this last helps
+		// prevent CPU exhaustion attacks.
+		err := checkBlockScripts(block, pastUTXO, transactions, scriptFlags, dag.sigCache)
+		if err != nil {
+			return nil, err
+		}
+	}
+	return feeData, nil
+}
+
+// CheckConnectBlockTemplate fully validates that connecting the passed block to
+// the DAG does not violate any consensus rules, aside from the proof of
+// work requirement.
+//
+// This function is safe for concurrent access.
+func (dag *BlockDAG) CheckConnectBlockTemplate(block *util.Block) error {
+	dag.dagLock.RLock()
+	defer dag.dagLock.RUnlock()
+	return dag.CheckConnectBlockTemplateNoLock(block)
+}
+
+// CheckConnectBlockTemplateNoLock fully validates that connecting the passed block to
+// the DAG does not violate any consensus rules, aside from the proof of
+// work requirement. The block must connect to the current tip of the main dag.
+func (dag *BlockDAG) CheckConnectBlockTemplateNoLock(block *util.Block) error {
+
+	// Skip the proof of work check as this is just a block template.
+	flags := BFNoPoWCheck
+
+	header := block.MsgBlock().Header
+
+	delay, err := dag.checkBlockSanity(block, flags)
+	if err != nil {
+		return err
+	}
+
+	if delay != 0 {
+		return errors.Errorf("Block timestamp is too far in the future")
+	}
+
+	parents, err := lookupParentNodes(block, dag)
+	if err != nil {
+		return err
+	}
+
+	err = dag.checkBlockContext(block, parents, flags)
+	if err != nil {
+		return err
+	}
+
+	templateNode, _ := dag.newBlockNode(&header, dag.virtual.tips())
+
+	_, err = dag.checkConnectToPastUTXO(templateNode,
+		dag.UTXOSet(), block.Transactions(), false)
+
+	return err
+}

blockdag/versionbits.go

@@ -0,0 +1,294 @@
+// Copyright (c) 2016-2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"math"
+
+	"github.com/kaspanet/kaspad/dagconfig"
+)
+
+const (
+	// vbTopBits defines the bits to set in the version to signal that the
+	// version bits scheme is being used.
+	vbTopBits = 0x10000000
+
+	// vbTopMask is the bitmask to use to determine whether or not the
+	// version bits scheme is in use.
+	vbTopMask = 0xe0000000
+
+	// vbNumBits is the total number of bits available for use with the
+	// version bits scheme.
+	vbNumBits = 29
+
+	// unknownVerNumToCheck is the number of previous blocks to consider
+	// when checking for a threshold of unknown block versions for the
+	// purposes of warning the user.
+	unknownVerNumToCheck = 100
+
+	// unknownVerWarnNum is the threshold of previous blocks that have an
+	// unknown version to use for the purposes of warning the user.
+	unknownVerWarnNum = unknownVerNumToCheck / 2
+)
+
+// bitConditionChecker provides a thresholdConditionChecker which can be used to
+// test whether or not a specific bit is set when it's not supposed to be
+// according to the expected version based on the known deployments and the
+// current state of the DAG. This is useful for detecting and warning about
+// unknown rule activations.
+type bitConditionChecker struct {
+	bit uint32
+	dag *BlockDAG
+}
+
+// Ensure the bitConditionChecker type implements the thresholdConditionChecker
+// interface.
+var _ thresholdConditionChecker = bitConditionChecker{}
+
+// BeginTime returns the unix timestamp for the median block time after which
+// voting on a rule change starts (at the next window).
+//
+// Since this implementation checks for unknown rules, it returns 0 so the rule
+// is always treated as active.
+//
+// This is part of the thresholdConditionChecker interface implementation.
+func (c bitConditionChecker) BeginTime() uint64 {
+	return 0
+}
+
+// EndTime returns the unix timestamp for the median block time after which an
+// attempted rule change fails if it has not already been locked in or
+// activated.
+//
+// Since this implementation checks for unknown rules, it returns the maximum
+// possible timestamp so the rule is always treated as active.
+//
+// This is part of the thresholdConditionChecker interface implementation.
+func (c bitConditionChecker) EndTime() uint64 {
+	return math.MaxUint64
+}
+
+// RuleChangeActivationThreshold is the number of blocks for which the condition
+// must be true in order to lock in a rule change.
+//
+// This implementation returns the value defined by the DAG params the checker
+// is associated with.
+//
+// This is part of the thresholdConditionChecker interface implementation.
+func (c bitConditionChecker) RuleChangeActivationThreshold() uint64 {
+	return c.dag.dagParams.RuleChangeActivationThreshold
+}
+
+// MinerConfirmationWindow is the number of blocks in each threshold state
+// retarget window.
+//
+// This implementation returns the value defined by the DAG params the checker
+// is associated with.
+//
+// This is part of the thresholdConditionChecker interface implementation.
+func (c bitConditionChecker) MinerConfirmationWindow() uint64 {
+	return c.dag.dagParams.MinerConfirmationWindow
+}
+
+// Condition returns true when the specific bit associated with the checker is
+// set and it's not supposed to be according to the expected version based on
+// the known deployments and the current state of the DAG.
+//
+// This function MUST be called with the DAG state lock held (for writes).
+//
+// This is part of the thresholdConditionChecker interface implementation.
+func (c bitConditionChecker) Condition(node *blockNode) (bool, error) {
+	conditionMask := uint32(1) << c.bit
+	version := uint32(node.version)
+	if version&vbTopMask != vbTopBits {
+		return false, nil
+	}
+	if version&conditionMask == 0 {
+		return false, nil
+	}
+
+	expectedVersion, err := c.dag.calcNextBlockVersion(node.selectedParent)
+	if err != nil {
+		return false, err
+	}
+	return uint32(expectedVersion)&conditionMask == 0, nil
+}
+
+// deploymentChecker provides a thresholdConditionChecker which can be used to
+// test a specific deployment rule. This is required for properly detecting
+// and activating consensus rule changes.
+type deploymentChecker struct {
+	deployment *dagconfig.ConsensusDeployment
+	dag        *BlockDAG
+}
+
+// Ensure the deploymentChecker type implements the thresholdConditionChecker
+// interface.
+var _ thresholdConditionChecker = deploymentChecker{}
+
+// BeginTime returns the unix timestamp for the median block time after which
+// voting on a rule change starts (at the next window).
+//
+// This implementation returns the value defined by the specific deployment the
+// checker is associated with.
+//
+// This is part of the thresholdConditionChecker interface implementation.
+func (c deploymentChecker) BeginTime() uint64 {
+	return c.deployment.StartTime
+}
+
+// EndTime returns the unix timestamp for the median block time after which an
+// attempted rule change fails if it has not already been locked in or
+// activated.
+//
+// This implementation returns the value defined by the specific deployment the
+// checker is associated with.
+//
+// This is part of the thresholdConditionChecker interface implementation.
+func (c deploymentChecker) EndTime() uint64 {
+	return c.deployment.ExpireTime
+}
+
+// RuleChangeActivationThreshold is the number of blocks for which the condition
+// must be true in order to lock in a rule change.
+//
+// This implementation returns the value defined by the DAG params the checker
+// is associated with.
+//
+// This is part of the thresholdConditionChecker interface implementation.
+func (c deploymentChecker) RuleChangeActivationThreshold() uint64 {
+	return c.dag.dagParams.RuleChangeActivationThreshold
+}
+
+// MinerConfirmationWindow is the number of blocks in each threshold state
+// retarget window.
+//
+// This implementation returns the value defined by the DAG params the checker
+// is associated with.
+//
+// This is part of the thresholdConditionChecker interface implementation.
+func (c deploymentChecker) MinerConfirmationWindow() uint64 {
+	return c.dag.dagParams.MinerConfirmationWindow
+}
+
+// Condition returns true when the specific bit defined by the deployment
+// associated with the checker is set.
+//
+// This is part of the thresholdConditionChecker interface implementation.
+func (c deploymentChecker) Condition(node *blockNode) (bool, error) {
+	conditionMask := uint32(1) << c.deployment.BitNumber
+	version := uint32(node.version)
+	return (version&vbTopMask == vbTopBits) && (version&conditionMask != 0),
+		nil
+}
+
+// calcNextBlockVersion calculates the expected version of the block after the
+// passed previous block node based on the state of started and locked in
+// rule change deployments.
+//
+// This function differs from the exported CalcNextBlockVersion in that the
+// exported version uses the selected tip as the previous block node
+// while this function accepts any block node.
+//
+// This function MUST be called with the DAG state lock held (for writes).
+func (dag *BlockDAG) calcNextBlockVersion(prevNode *blockNode) (int32, error) {
+	// Set the appropriate bits for each actively defined rule deployment
+	// that is either in the process of being voted on, or locked in for the
+	// activation at the next threshold window change.
+	expectedVersion := uint32(vbTopBits)
+	for id := 0; id < len(dag.dagParams.Deployments); id++ {
+		deployment := &dag.dagParams.Deployments[id]
+		cache := &dag.deploymentCaches[id]
+		checker := deploymentChecker{deployment: deployment, dag: dag}
+		state, err := dag.thresholdState(prevNode, checker, cache)
+		if err != nil {
+			return 0, err
+		}
+		if state == ThresholdStarted || state == ThresholdLockedIn {
+			expectedVersion |= uint32(1) << deployment.BitNumber
+		}
+	}
+	return int32(expectedVersion), nil
+}
+
+// CalcNextBlockVersion calculates the expected version of the block after the
+// end of the current selected tip based on the state of started and locked in
+// rule change deployments.
+//
+// This function is safe for concurrent access.
+func (dag *BlockDAG) CalcNextBlockVersion() (int32, error) {
+	version, err := dag.calcNextBlockVersion(dag.selectedTip())
+	return version, err
+}
+
+// warnUnknownRuleActivations displays a warning when any unknown new rules are
+// either about to activate or have been activated. This will only happen once
+// when new rules have been activated and every block for those about to be
+// activated.
+//
+// This function MUST be called with the DAG state lock held (for writes)
+func (dag *BlockDAG) warnUnknownRuleActivations(node *blockNode) error {
+	// Warn if any unknown new rules are either about to activate or have
+	// already been activated.
+	for bit := uint32(0); bit < vbNumBits; bit++ {
+		checker := bitConditionChecker{bit: bit, dag: dag}
+		cache := &dag.warningCaches[bit]
+		state, err := dag.thresholdState(node.selectedParent, checker, cache)
+		if err != nil {
+			return err
+		}
+
+		switch state {
+		case ThresholdActive:
+			if !dag.unknownRulesWarned {
+				log.Warnf("Unknown new rules activated (bit %d)",
+					bit)
+				dag.unknownRulesWarned = true
+			}
+
+		case ThresholdLockedIn:
+			window := checker.MinerConfirmationWindow()
+			activationBlueScore := window - (node.blueScore % window)
+			log.Warnf("Unknown new rules are about to activate in "+
+				"%d blueScore (bit %d)", activationBlueScore, bit)
+		}
+	}
+
+	return nil
+}
+
+// warnUnknownVersions logs a warning if a high enough percentage of the last
+// blocks have unexpected versions.
+//
+// This function MUST be called with the DAG state lock held (for writes)
+func (dag *BlockDAG) warnUnknownVersions(node *blockNode) error {
+	// Nothing to do if already warned.
+	if dag.unknownVersionsWarned {
+		return nil
+	}
+
+	// Warn if enough previous blocks have unexpected versions.
+	numUpgraded := uint32(0)
+	for i := uint32(0); i < unknownVerNumToCheck && node != nil; i++ {
+		expectedVersion, err := dag.calcNextBlockVersion(node.selectedParent)
+		if err != nil {
+			return err
+		}
+		if (node.version & ^expectedVersion) != 0 {
+
+			numUpgraded++
+		}
+
+		node = node.selectedParent
+	}
+	if numUpgraded > unknownVerWarnNum {
+		log.Warn("Unknown block versions are being mined, so new " +
+			"rules might be in effect. Are you running the " +
+			"latest version of the software?")
+		dag.unknownVersionsWarned = true
+	}
+
+	return nil
+}

blockdag/virtualblock.go

@@ -0,0 +1,162 @@
+// Copyright (c) 2017 The btcsuite developers
+// Use of this source code is governed by an ISC
+// license that can be found in the LICENSE file.
+
+package blockdag
+
+import (
+	"github.com/kaspanet/kaspad/util/daghash"
+	"sync"
+)
+
+// virtualBlock is a virtual block whose parents are the tips of the DAG.
+type virtualBlock struct {
+	mtx     sync.Mutex
+	dag     *BlockDAG
+	utxoSet *FullUTXOSet
+	blockNode
+
+	// selectedParentChainSet is a block set that includes all the blocks
+	// that belong to the chain of selected parents from the virtual block.
+	selectedParentChainSet blockSet
+
+	// selectedParentChainSlice is an ordered slice that includes all the
+	// blocks that belong the the chain of selected parents from the
+	// virtual block.
+	selectedParentChainSlice []*blockNode
+}
+
+// newVirtualBlock creates and returns a new VirtualBlock.
+func newVirtualBlock(dag *BlockDAG, tips blockSet) *virtualBlock {
+	// The mutex is intentionally not held since this is a constructor.
+	var virtual virtualBlock
+	virtual.dag = dag
+	virtual.utxoSet = NewFullUTXOSet()
+	virtual.selectedParentChainSet = newSet()
+	virtual.selectedParentChainSlice = nil
+	virtual.setTips(tips)
+
+	return &virtual
+}
+
+// clone creates and returns a clone of the virtual block.
+func (v *virtualBlock) clone() *virtualBlock {
+	return &virtualBlock{
+		utxoSet:                v.utxoSet,
+		blockNode:              v.blockNode,
+		selectedParentChainSet: v.selectedParentChainSet,
+	}
+}
+
+// setTips replaces the tips of the virtual block with the blocks in the
+// given blockSet. This only differs from the exported version in that it
+// is up to the caller to ensure the lock is held.
+//
+// This function MUST be called with the view mutex locked (for writes).
+func (v *virtualBlock) setTips(tips blockSet) *chainUpdates {
+	oldSelectedParent := v.selectedParent
+	node, _ := v.dag.newBlockNode(nil, tips)
+	v.blockNode = *node
+	return v.updateSelectedParentSet(oldSelectedParent)
+}
+
+// updateSelectedParentSet updates the selectedParentSet to match the
+// new selected parent of the virtual block.
+// Every time the new selected parent is not a child of
+// the old one, it updates the selected path by removing from
+// the path blocks that are selected ancestors of the old selected
+// parent and are not selected ancestors of the new one, and adding
+// blocks that are selected ancestors of the new selected parent
+// and aren't selected ancestors of the old one.
+func (v *virtualBlock) updateSelectedParentSet(oldSelectedParent *blockNode) *chainUpdates {
+	var intersectionNode *blockNode
+	nodesToAdd := make([]*blockNode, 0)
+	for node := v.blockNode.selectedParent; intersectionNode == nil && node != nil; node = node.selectedParent {
+		if v.selectedParentChainSet.contains(node) {
+			intersectionNode = node
+		} else {
+			nodesToAdd = append(nodesToAdd, node)
+		}
+	}
+
+	if intersectionNode == nil && oldSelectedParent != nil {
+		panic("updateSelectedParentSet: Cannot find intersection node. The block index may be corrupted.")
+	}
+
+	// Remove the nodes in the set from the oldSelectedParent down to the intersectionNode
+	// Also, save the hashes of the removed blocks to removedChainBlockHashes
+	removeCount := 0
+	var removedChainBlockHashes []*daghash.Hash
+	if intersectionNode != nil {
+		for node := oldSelectedParent; !node.hash.IsEqual(intersectionNode.hash); node = node.selectedParent {
+			v.selectedParentChainSet.remove(node)
+			removedChainBlockHashes = append(removedChainBlockHashes, node.hash)
+			removeCount++
+		}
+	}
+	// Remove the last removeCount nodes from the slice
+	v.selectedParentChainSlice = v.selectedParentChainSlice[:len(v.selectedParentChainSlice)-removeCount]
+
+	// Reverse nodesToAdd, since we collected them in reverse order
+	for left, right := 0, len(nodesToAdd)-1; left < right; left, right = left+1, right-1 {
+		nodesToAdd[left], nodesToAdd[right] = nodesToAdd[right], nodesToAdd[left]
+	}
+	// Add the nodes to the set and to the slice
+	// Also, save the hashes of the added blocks to addedChainBlockHashes
+	var addedChainBlockHashes []*daghash.Hash
+	for _, node := range nodesToAdd {
+		v.selectedParentChainSet.add(node)
+		addedChainBlockHashes = append(addedChainBlockHashes, node.hash)
+	}
+	v.selectedParentChainSlice = append(v.selectedParentChainSlice, nodesToAdd...)
+
+	return &chainUpdates{
+		removedChainBlockHashes: removedChainBlockHashes,
+		addedChainBlockHashes:   addedChainBlockHashes,
+	}
+}
+
+// SetTips replaces the tips of the virtual block with the blocks in the
+// given blockSet.
+//
+// This function is safe for concurrent access.
+func (v *virtualBlock) SetTips(tips blockSet) {
+	v.mtx.Lock()
+	v.setTips(tips)
+	v.mtx.Unlock()
+}
+
+// addTip adds the given tip to the set of tips in the virtual block.
+// All former tips that happen to be the given tips parents are removed
+// from the set. This only differs from the exported version in that it
+// is up to the caller to ensure the lock is held.
+//
+// This function MUST be called with the view mutex locked (for writes).
+func (v *virtualBlock) addTip(newTip *blockNode) *chainUpdates {
+	updatedTips := v.tips().clone()
+	for parent := range newTip.parents {
+		updatedTips.remove(parent)
+	}
+
+	updatedTips.add(newTip)
+	return v.setTips(updatedTips)
+}
+
+// AddTip adds the given tip to the set of tips in the virtual block.
+// All former tips that happen to be the given tip's parents are removed
+// from the set.
+//
+// This function is safe for concurrent access.
+func (v *virtualBlock) AddTip(newTip *blockNode) *chainUpdates {
+	v.mtx.Lock()
+	defer v.mtx.Unlock()
+	return v.addTip(newTip)
+}
+
+// tips returns the current tip block nodes for the DAG. It will return
+// an empty blockSet if there is no tip.
+//
+// This function is safe for concurrent access.
+func (v *virtualBlock) tips() blockSet {
+	return v.parents
+}