121 Commits

Author SHA1 Message Date
Ori Newman
2adb4f5d0f
Fix UTXO index (#1548)
* Add VirtualUTXODiff and VirtualParents to block insertion result

* Add GetVirtualUTXOs

* Add OnPruningPointUTXOSetOverrideHandler

* Add recovery to UTXO index

* Add UTXO set override notification

* Fix compilation error

* Fix iterators in UTXO index and fix TestUTXOIndex

* Change Dialing to DEBUG

* Change LogBlock location

* Rename StopNotify to StopNotifying

* Add sanity check

* Add comment

* Remove receiver from serialization functions

Co-authored-by: Elichai Turkel <elichai.turkel@gmail.com>
2021-02-23 16:51:51 +02:00
Elichai Turkel
a581dea127
Remove unused utils and structures (#1526)
* Remove unused utils

* Remove unneeded randomness from tests

* Remove more unused functions

* Remove unused protobuf structures

* Fix small errors
2021-02-14 18:13:20 +02:00
Elichai Turkel
2edf6bfd07
Minimize memory usage in tests (#1495)
* Make leveldb cache configurable

* Fix leveldb tests

* Add a preallocate option to all caches and disable in tests

Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-02-08 18:37:02 +02:00
Elichai Turkel
3d0a2a47b2
Move testGHOSTDagSorter to testutils, and build a boilerplate for overriding specific managers (#1486)
* Move testGHOSTDagSorter to testutils

* Allow overriding managers in consensus, starting with ghostdag

* Add test prefix to SetDataDir and SetGHOSTDAGManager

Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-02-08 15:24:26 +02:00
Ori Newman
d30f05b250
Remove IsPushOnlyScript from mempool validation (#1492)
* Remove IsPushOnlyScript from mempool validation

* Fix TestCheckTransactionStandard
2021-02-08 10:04:19 +02:00
Ori Newman
dbababb978
Limit mempool size to million transactions and remove the least profitable transactions (#1483)
* Limit mempool size to million transactions and remove the least profitable transactions

* Simplify insert

* Fix typo

* Improve findTxIndexInOrderedTransactionsByFeeRate readability
2021-02-03 19:45:39 +02:00
stasatdaglabs
756f40c59a
Sync pruning point UTXO sets incrementally instead of all at once (#1431)
* Replaced the content of MsgIBDRootUTXOSetChunk with pairs of outpoint-utxo entry pairs.

* Rename utxoIter to utxoIterator.

* Add a big stinky TODO on an assert.

* Replace pruningStore staging with a UTXO set iterator.

* Reimplement receiveAndInsertIBDRootUTXOSet.

* Extract OutpointAndUTXOEntryPairsToDomainOutpointAndUTXOEntryPairs into domainconverters.go.

* Pass the outpoint and utxy entry pairs to the pruning store.

* Implement InsertCandidatePruningPointUTXOs.

* Implement ClearCandidatePruningPointUTXOs.

* Implement UpdateCandidatePruningPointMultiset.

* Use the candidate pruning point multiset in updatePruningPoint.

* Implement CandidatePruningPointUTXOIterator.

* Use the pruning point utxo set iterator for StageVirtualUTXOSet.

* Defer ClearCandidatePruningPointUTXOs.

* Implement OverwriteVirtualUTXOSet.

* Implement CommitCandidatePruningPointUTXOSet.

* Implement BeginOverwritingVirtualUTXOSet and FinishOverwritingVirtualUTXOSet.

* Implement overwriteVirtualUTXOSetAndCommitPruningPointUTXOSet.

* Rename ClearCandidatePruningPointUTXOs to ClearCandidatePruningPointData.

* Add missing methods to dbManager.

* Implement PruningPointUTXOs.

* Implement RecoverUTXOIfRequired.

* Delete the utxoserialization package.

* Fix compilation errors in TestValidateAndInsertPruningPoint.

* Switch order of operations in the if statements in PruningPointUTXOs so that Next() wouldn't be unnecessarily called.

* Fix missing pruning point utxo set staging and bad slice length.

* Fix no default multiset in InsertCandidatePruningPointUTXOs.

* Make go vet happy.

* Rename candidateXXX to importedXXX.

* Do some more renaming.

* Rename some more.

* Fix bad MsgIBDRootNotFound logic.

* Fix an error message.

* Simplify receiveIBDRootBlock.

* Fix error message in receiveAndInsertIBDRootUTXOSet.

* Do some more renaming.

* Fix merge errors.

* Fix a bug caused by calling iterator.First() unnecessarily.

* Remove databaseContext from stores and don't use a transaction in ClearXXX functions.

* Simplify receiveAndInsertIBDRootUTXOSet.

* Fix offset count in PruningPointUTXOs().

* Fix readOnlyUTXOIteratorWithDiff.First().

* Split handleRequestIBDRootUTXOSetAndBlockFlow into smaller methods.

* Rename IbdRootNotFound to UnexpectedPruningPoint.

* Rename requestIBDRootHash to requestPruningPointHash.

* Rename IBDRootHash to PruningPointHash.

* Rename RequestIBDRootUTXOSetAndBlock to RequestPruningPointUTXOSetAndBlock.

* Rename IBDRootUTXOSetChunk to PruningPointUTXOSetChunk.

* Rename RequestNextIBDRootUTXOSetChunk to RequestNextPruningPointUTXOSetChunk.

* Rename DoneIBDRootUTXOSetChunks to DonePruningPointUTXOSetChunks.

* Rename remaining references to IBD root.

* Fix an error message.

* Add a check for HadStartedImportingPruningPointUTXOSet in commitVirtualUTXODiff.

* Add a check for HadStartedImportingPruningPointUTXOSet in ImportPruningPointUTXOSetIntoVirtualUTXOSet.

* Move FinishImportingPruningPointUTXOSet closer to HadStartedImportingPruningPointUTXOSet.

* Remove reference to pruningStore in utxoSetIterator.

* Pointerify utxoSetIterator receivers.

* Fix bad insert in CommitImportedPruningPointUTXOSet.

* Rename commitImportedPruningPointUTXOSetAll to applyImportedPruningPointUTXOSet.

* Simplify PruningPointUTXOs.

* Add populateTransactionWithUTXOEntriesFromUTXOSet.

* Fix a TODO comment.

* Rename InsertImportedPruningPointUTXOs to AppendImportedPruningPointUTXOs.

* Extract handleRequestPruningPointUTXOSetAndBlockMessage to a separate method.

* Rename stuff in readOnlyUTXOIteratorWithDiff.First().

* Address toAddIterator in readOnlyUTXOIteratorWithDiff.First().

* Call First() before any full iteration on ReadOnlyUTXOSetIterator.

* Call First() before any full iteration on a database Cursor.

* Put StartImportingPruningPointUTXOSet inside the pruning point transaction.

* Make serializeOutpoint and serializeUTXOEntry free functions in pruningStore.

* Fix readOnlyUTXOIteratorWithDiff.First().

* Fix bad validations in importPruningPoint.

* Remove superfluous call to validateBlockTransactionsAgainstPastUTXO.
2021-01-21 17:24:52 +02:00
Svarog
ad9c213a06
Restructure database to prevent double-slashes in keys, causing bugs in cursors (#1432)
* Add TestValidateAndInsertPruningPointWithSideBlocks

* Optimize infrastracture bucket paths

* Update infrastracture tests

* Refactor the consensus/database layer

* Remove utils/dbkeys

* Use consensus/database in consensus instead of infrastructure

* Fix a bug in dbBucketToDatabaseBucket and MakeBucket combination

Co-authored-by: Elichai Turkel <elichai.turkel@gmail.com>
Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-01-19 14:19:08 +02:00
Ori Newman
dd57e6abe6
Fix checkParentHeadersExist and cover pruning_violation_proof_of_work_and_difficulty.go with tests (#1418)
* Fix checkParentHeadersExist and cover pruning_violation_proof_of_work_and_difficulty.go with tests

* Remove unused variable

* Change consensus violation

* Change condition order

* Get rid of irrelevant error codes in extractRejectCode

* Fix wrong test db names

* Fix checkParentHeadersExist
2021-01-17 11:27:04 +02:00
Ori Newman
6101e6bdb6
Fix UTXO serialization, its test, and the static check that missed it (#1396)
* Fix UTXO serialization, its test, and the static check that missed it

* Remove duplicate case

* Use one line for static check

Co-authored-by: Elichai Turkel <elichai.turkel@gmail.com>
2021-01-11 17:45:17 +02:00
Ori Newman
b8ca33d91d
Add selected chain store and optimize block locator with it (#1394)
* Add selected chain store and optimize block locator with it

* Fix build error

* Fix comments

* Fix IsStaged

* Rename CalculateSelectedParentChainChanges to CalculateChainPath and SelectedParentChainChanges->SelectedChainPath

* Use binary.LittleEndian directly to allow compiler optimizations

* Remove boolean from HeadersSelectedChainStore interface

* Prevent endless loop in block locator
2021-01-11 15:51:45 +02:00
Svarog
c7deda41c6
Fix deserialization of script version in UTXOSet deserialization (#1395)
* Initalize protoUTXOSetIterator with index = -1

* Handle error when failed to deserialize Script version

* Add support for (de)serialization of (u)int16

* Log the error when converting it into ErrMalformedUTXO
2021-01-11 15:23:27 +02:00
Svarog
9ea4c0fa38
Add sanity check that makes sure that the utxoset that we save fits the pruningPoints's commitment. (#1366)
Co-authored-by: Elichai Turkel <elichai.turkel@gmail.com>
2021-01-06 13:27:02 +02:00
Elichai Turkel
4577023e44
Add script pubkey version to signature hash (#1360)
* Replace 0xffff with math.MaxUint16 on version checks

* Add script version to the signature hash

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-01-06 11:51:12 +02:00
talelbaz
8a309a7d2a
Upgradability mechanisms script version (#1313)
* ''

* ''

* ''

* Changes genesis block version to 0.

* a

* a

* All tests are done.

* All tests passed for changed block version from int32 to uint16

* Adds validation of rejecting blocks with unknown versions.

* Changes txn version from int32 to uint16.

* .

* Adds comments to exported functions.

* Change functions name from ConvertFromRpcScriptPubKeyToRPCScriptPubKey to ConvertFromAppMsgRPCScriptPubKeyToRPCScriptPubKey and from ConvertFromRPCScriptPubKeyToRpcScriptPubKey to ConvertFromRPCScriptPubKeyToAppMsgRPCScriptPubKey

* change comment to "ScriptPublicKey represents a Kaspad ScriptPublicKey"

* delete part (tx.Version < 0) that cannot be exist on the if statement.

* Revert protobuf version.

* Fix a comment.

* Fix a comment.

* Rename a variable.

* Rename a variable.

* Remove a const.

* Rename a type.

* Rename a field.

* Rename a field.

* Remove commented-out code.

* Remove dangerous nil case in DomainTransactionOutput.Clone().

* Remove a constant.

* Fix a string.

* Fix wrong totalScriptPubKeySize in transactionMassStandalonePart.

* Remove a constant.

* Remove an unused error.

* Fix a serialization error.

* Specify version types to be uint16 explicitly.

* Use constants.ScriptPublicKeyVersion.

* Fix a bad test.

* Remove some whitespace.

* Add a case to utxoEntry.Equal().

* Rename scriptPubKey to scriptPublicKey.

* Remove a TODO.

* Rename constants.

* Rename a variable.

* Add version to parseShortForm.

Co-authored-by: tal <tal@daglabs.com>
Co-authored-by: stasatdaglabs <stas@daglabs.com>
2021-01-05 17:50:09 +02:00
Elichai Turkel
e509cb1597
Smal performance improvements to BlueWindow (#1343)
* Convert BlockGHOSTDAGData from an interface to a public struct with getters

* Move hashes.Less to externalapi so it can access the hashes directly without copying

* Reduce calls to ghostdagstore.Get in blueWindow

* Simplify the logic in RequiredDifficulty and reuse big.Int

* Remove bigintpool as its no longer used

* Use ChooseSelectedParent in RequiredDifficulty instead of looping over the parents

* Remove comment
2021-01-05 12:13:02 +02:00
Ori Newman
23304a4977
Add UTXO set cache (#1333)
* Add UTXO set cache

* Add clear method to cache
2020-12-31 16:34:46 +02:00
Svarog
9b12b9c58a
Make ReachabilityData a read-only interface with a writable variant, to prevent cloning (#1316)
* Rename reachabilityManager.data to dataForInsertion, and use it only during insertions

* Make reachabilityData an interface

* Fix db serialization of reachability data

* Fix reachabilityDataStore

* Fix all tests

* Cleanup debugging code

* Fix insertToFutureCoveringSet

* Add comments

* Rename to ReachabilityData and MutableReachabilityData
2020-12-30 09:48:38 +02:00
Ori Newman
49b6cc6038
Add mutable and immutable header interfaces (#1305)
* Add mutable and immutable header interfaces

* Fix ShouldMine()

* Remove false comment

* Fix Equal signature

* Fix Equal implementation
2020-12-29 13:55:17 +02:00
Svarog
05941a76e7
Make DomainHash and TransactionID read-only structs (#1282)
* Increase size of reachability cache

* Change DomainHash to struct with unexported hashArray

* Fixing compilation errors stemming from new DomainHash structure

* Remove obsolete Read/WriteElement methods in appmessage

* Fix all tests

* Fix all tests

* Add comments

* A few renamings

* go mod tidy
2020-12-24 16:15:23 +02:00
Elichai Turkel
273c271771
Remove hash reversal (#1270)
* Remove hash reversing

* Move toBig to pow.go

* Update some tests
2020-12-23 12:42:37 +02:00
Ori Newman
cb9d7e313d
Implement Clone and Equal for all model types (#1155)
* [NOD-1575] Implement Clone and Equal for all model types

* [NOD-1575] Add assertion for transaction ID equality

* [NOD-1575] Use DomainTransaction.Equal to compare to expected coinbase transaction

* [NOD-1575] Add TestDomainBlockHeader_Clone

* [NOD-1575] Don't clone nil values

* [NOD-1575] Add type assertions

* [NOD-1575] Don't clone nil values

* [NOD-1575] Add missing Equals

* [NOD-1575] Add length checks

* [NOD-1575] Update comment

* [NOD-1575] Check length for TransactionAcceptanceData

* [NOD-1575] Explicitly clone nils where needed

* [NOD-1575] Clone tx id

* [NOD-1575] Flip condition

* Nod 1576 make coverage tests for equal clone inside model externalapi (#1177)

* [NOD-1576] Make coverage tests for equal and clone inside model and externalapi

* Some formatting and naming fixes

* Made transactionToCompare type exported

* Added some tests and made some changes to the tests code

* No changes made

* Some formatting and naming changes made

* Made better test coverage for externalapi clone and equal functions

* Changed expected result for two cases

* Added equal and clone functions tests for ghostdag and utxodiff

* Added tests

* [NOD-1576] Implement reachabilitydata equal/clone unit tests

* [NOD-1576]  Full coverage of reachabilitydata equal/clone unit tests

* Made changes and handling panic to transaction_equal_clone_test.go and formating of utxodiff_equal_clone_test.go

* Added recoverForEqual2 for handling panic to transaction_equal_clone_test.go

* [NOD-1576]  Full coverage of transaction equal unit test

* [NOD-1576] Add expects panic

* [NOD-1576] Allow composites in go vet

* [NOD-1576] Code review fixes (#1223)

* [NOD-1576] Code review fixes

* [NOD-1576] Code review fixes part 2

* [NOD-1576] Fix wrong name

Co-authored-by: karim1king <karimkaspersky@yahoo.com>
Co-authored-by: Ori Newman <orinewman1@gmail.com>
Co-authored-by: Karim <karim1king@users.noreply.github.com>

* Fix merge errors

* Use Equal where possible

* Use Equal where possible

* Use Equal where possible

Co-authored-by: andrey-hash <74914043+andrey-hash@users.noreply.github.com>
Co-authored-by: karim1king <karimkaspersky@yahoo.com>
Co-authored-by: Karim <karim1king@users.noreply.github.com>
2020-12-22 17:38:54 +02:00
Elichai Turkel
45edacfbfa
Replace Double-SHA256 with blake2b and implement domain seperation (#1245)
* Replace default hasher (Double-SHA256) with domain seperated blake2b

* Replace all hashes with domain seperated blake2b

* Update the genesis blocks

* Replace OP_HASH256 with OP_BLAKE2B

* Fix the merkle tree by appending zeros instead of duplicating the hash when there is 1 branch left

* Update tests

* Add a payloadHash function

* Update gitignore to ignore binaries

* Fix a bug in the blake2b opcode
2020-12-21 12:51:45 +02:00
Ori Newman
12379bedb6
Fix UTXO serialization errors (#1233) 2020-12-16 11:27:43 +02:00
Elichai Turkel
3354ac67c8
Parallelize all the tests in ForAllNets (#1199) 2020-12-09 17:43:36 +02:00
Svarog
0d8f7bba40
[NOD-1595] Implement all fields of GetBlockDAGInfo (#1200)
* [NOD-1595] Implement all fields of GetBlockDAGInfo

* [NOD-1595]

* [NOD-1595] Don't swallow errors in GetDifficultyRatio

* [NOD-1595] Change roundingPrecision in GetDifficultyRatio to 2 decimal places
2020-12-09 12:14:15 +02:00
Elichai Turkel
82fa8e6831
Abstract CheckProofOfWork as a public function and change PoW structure (#1198)
* Expose CheckProofOfWork from model/pow

* Update blockvalidator to call the new CheckProofOfWork

* Update genesis blocks

* Update tools to use the new CheckProofOfWork

* Update tests with new PoW
2020-12-08 17:17:30 +02:00
Ori Newman
32a04d1811
Allow to configure consensus (closes #1067)
* Allow to configure consensus with a JSON file

* Define everywhere maxBlockParents as KType

* Move consensus default to consensus_defaults.go
2020-12-03 18:30:01 +02:00
Svarog
a585f32763
[NOD-1551] Make UTXODiff immutable + skip cloning it in datastore (#1167)
* [NOD-1551] Make UTXO-Diff implemented fully in utils/utxo

* [NOD-1551] Fixes everywhere except database

* [NOD-1551] Fix database

* [NOD-1551] Add comments

* [NOD-1551] Partial commit

* [NOD-1551] Comlete making UTXOEntry immutable + don't clone it in UTXOCollectionClone

* [NOD-1551] Rename ToUnmutable -> ToImmutable

* [NOD-1551] Track immutable references generated from mutable UTXODiff, and invalidate them if the mutable one changed

* [NOD-1551] Clone scriptPubKey in NewUTXOEntry

* [NOD-1551] Remove redundant code

* [NOD-1551] Remove redundant call for .CloneMutable and then .ToImmutable

* [NOD-1551] Make utxoEntry pointert-receiver + clone ScriptPubKey in getter
2020-12-03 13:24:24 +02:00
Mike Zak
9866abb75a [NOD-1583] Split consensusserialization to consensushashing and serialization 2020-12-02 13:18:50 +02:00
Elichai Turkel
f1c6df48c9
[#1028] Replace oldschnorr with the BIP340 schnorr variant (#1165)
* Update go-secp256k1 to v0.0.3

* Update the txscript engine to support only 32 bytes pubkeys

* Update the txscript engine tests

* Update txscript/sign.go to use the new Schnorr KeyPair API

* Update txscript sign_test to use the new schnorr

* Update sigcache tests to use new schnorr pubkey

* Update integration tests to use the new txscript and new schnorr pubkey
2020-12-01 08:48:23 +02:00
Svarog
80c445c78b
[NOD-1551] Optimize binary writes + remove redundant VarInt code (#1163)
* [NOD-1551] Optimize binary writes + remove redundant VarInt code

* [NOD-1551] Remove varInt tests

* [NOD-1551] Fix TestBech32 for Go1.15
2020-11-30 13:45:06 +02:00
Svarog
3b6eb73e53
[NOD-1551] Remove lock in SigCache (#1161) 2020-11-30 11:34:19 +02:00
Svarog
048caebda3
[NOD-1551] Add SigCache to TransactionValidator + Option to manipulate it in TestConsensus (#1159)
* [NOD-1551] Add SigCache

* [NOD-1551] Add option to edit SigCache in TestConsensus

* [NOD-1551] Fix comments and make SetSigCache pointer-receiver
2020-11-29 10:18:00 +02:00
Svarog
f7fa823f17
[NOD-1551] Requirements for performance tests (#1154)
* [NOD-1551] Add NewTestConsensusWithDataDir to factory

* [NOD-1551] Cache transaction ID

* [NOD-1551] Should return err if err != nil

* [NOD-1551] BuildBlockWithParents returns the blocks pastUTXOData

* [NOD-1551] Set BlockCoinbaseMaturity to 0 in TestDoubleSpends

* [NOD-1551] Fix comments

* --amend

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2020-11-26 12:12:01 +02:00
Svarog
546ea83123
[NOD-1570] Fix the way UTXO iterators work (#1153)
* [NOD-1570] Implement utxo.IteratorWithDiff

* [NOD-1570] Utilize utxo.ITeratorWithDiff in RestorePastUTXOSetIterator and VirtualUTXOSetIterator

* [NOD-1570] Fix comment
2020-11-25 18:28:42 +02:00
stasatdaglabs
0fa13357c3
[NOD-1566] Add caching to all stores (#1152)
* [NOD-1566] Add a dependency to golang-lru.

* [NOD-1566] Add caching to blockstore.go.

* [NOD-1566] Add LRUCache to all store objects and initialize them.

* [NOD-1566] Add caching to acceptanceDataStore.

* [NOD-1566] Add caching to blockHeaderStore.

* [NOD-1566] Implement a simpler LRU cache.

* [NOD-1566] Use the simpler cache implementation everywhere.

* [NOD-1566] Remove dependency in golang-lru.

* [NOD-1566] Fix object reuse issues in store Get functions.

* [NOD-1566] Add caching to blockRelationStore.

* [NOD-1566] Add caching to blockStatusStore.

* [NOD-1566] Add caching to ghostdagDataStore.

* [NOD-1566] Add caching to multisetStore.

* [NOD-1566] Add caching to reachabilityDataStore.

* [NOD-1566] Add caching to utxoDiffStore.

* [NOD-1566] Add caching to reachabilityReindexRoot.

* [NOD-1566] Add caching to pruningStore.

* [NOD-1566] Add caching to headerTipsStore.

* [NOD-1566] Add caching to consensusStateStore.

* [NOD-1566] Add comments explaining why we don't discard staging at the normal location in consensusStateStore.

* [NOD-1566] Make go vet happy.

* [NOD-1566] Fix merge errors.

* [NOD-1566] Add a missing break statement.

* [NOD-1566] Run go mod tidy.

* [NOD-1566] Remove serializedUTXOSetCache.
2020-11-25 13:41:13 +02:00
Ori Newman
45d9b63572
[NOD-1567] Add clone methods to data stores types (#1149)
* [NOD-1567] Add clone methods to data stores types

* [NOD-1567] Fix comments

* [NOD-1567] Fix test
2020-11-24 17:56:18 +02:00
Ori Newman
bb2d7f72ac
[NOD-1560] Add TestValidateTransactionInIsolation (#1140)
* [NOD-1560] Add TestValidateTransactionInIsolation

* [NOD-1560] Make ForAllNets copy the params before mutating them

* [NOD-1560] Remove redundant continue

* [NOD-1560] Don't change finality duration
2020-11-23 16:28:59 +02:00
Ori Newman
8500acd86b
[NOD-1548] Remove PoW check from tests (#1105)
* [NOD-1548] Add TestDifficulty and remove PoW check from tests

* [NOD-1548] Add TestSkipProofOfWork

* [NOD-1548] Remove TestDifficulty
2020-11-18 10:27:29 +02:00
Elichai Turkel
c785ca0e52 Add more compactBits tests (#1096) 2020-11-17 16:00:16 +02:00
Ori Newman
14d7ab5fc6 Add TestBigToCompact and TestCompactToBig (#1092)
* Add TestBigToCompact and TestCompactToBig

* Add tests
2020-11-17 16:00:16 +02:00
Mike Zak
7224d58940 [NOD-1532] Add comments 2020-11-17 16:00:16 +02:00
Mike Zak
dc80a39c54 [NOD-1532] OpTrueScript should also return the redeem script 2020-11-17 16:00:16 +02:00
Ori Newman
b50421beee [NOD-1535] Don't reuse pointers on loop (#1069)
* [NOD-1535] Don't reuse pointers on loop

* [NOD-1535] Don't reuse pointers on loop
2020-11-17 16:00:16 +02:00
Ori Newman
fc5e39f6cc [NOD-1535] fix reachability test (#1061)
* Revert "[NOD-1500] Delete integration tests"

This reverts commit fcb57a206690a884fa6afb69d5d493282954a8bf.

* [NOD-1518] hashserialization -> consenusserialization

* [NOD-1518] Fix add genesis to virtual

* [NOD-1518] Fix a bug in SerializeCoinbasePayload.

* [NOD-1518] Fix a loop error and make pastMedianTime behave correctly everywhere on genesis.

* [NOD-1518] Fix another bug and an infinite loop.

* [NOD-1518] Fix uninitialized slice.

* [NOD-1518] Fix bad should-commit checks and another infinite loop.

* [NOD-1518] Fix nil serialization.

* [NOD-1518] Rename blockHash to currentBlockHash.

* [NOD-1518] Move the check whether stagedVirtualUTXOSet != nil to the top of commitVirtualUTXODiff.

* [NOD-1518] Simplify utxoDiffStore.Commit.

* [NOD-1518] Unextract resolveBlockStatusAndCheckFinality.

* [NOD-1518] Move no-transactions logic into CalculateIDMerkleRoot.

* [NOD-1518] Remove redundant is-staged check.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Don't write anything if utxoDiffChild is nil.

* [NOD-1518] Stage virtualAcceptanceData and virtualMultiset.

* [NOD-1518] Fix bugs in getBlockTemplate and submitBlock.

* [NOD-1518] Fix bad validation order in validateHeaderInContext.

* [NOD-1518] Fix bug in Next().

* [NOD-1518] Fix nil dereference of subnetworks in AddressCache.

* [NOD-1518] Fix multisetStore.Get returning a pointer to a multiset that is changed in place.

* [NOD-1518] Break on genesis in countSubtrees.

* [NOD-1518] Fix createBlockLocator.

* [NOD-1518] Fix MsgTxToDomainTransaction.

* [NOD-1518] Set MaxTxVersion to 1.

* [NOD-1518] Fix missing error handling, bug in MsgTxToDomainTransaction, and bad subnetwork equality check.

* [NOD-1518] Fix bug in hasUTXOByOutpointFromStagedVirtualUTXODiff.

* [NOD-1518] Remove irrelevant comments.

* [NOD-1518] Generate transactions with sufficient fee in tx_relay_test.

* [NOD-1518] Fix broken RPC handlers.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Fix bad exists check in restorePastUTXO and missing genesis check in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Add a comment.

* [NOD-1518] Use a regular mutex instead of a read-write mutex in consensus to avoid dealing with sneaky not-actually-read functions.

* [NOD-1518] Fix a deadlock in GetVirtualSelectedParent.

* [NOD-1518] Fix missing handler registration for CmdHeader.

* [NOD-1518] Fix processHeader calling OnNewBlock and LogBlock. Also fix conversion errors in IBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad Command() in MsgIBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad SyncStateMissingUTXOSet logic in resolveSyncState.

* [NOD-1518] Rename mode to syncState.

* [NOD-1518] Fix headers-only blocks coming in after the consensus thinks it's synced.

* [NOD-1518] Fix selectedChildIterator.Next not ignoring virtual, infinite loop in HashSet.Length().

* [NOD-1518] Fix not-properly wrapped IBD blocks.

* [NOD-1518] Fix bad conversion in RequestIBDBlocks.

* [NOD-1518] Fix bad string for CmdRequestHeaders.

* [NOD-1518] Fix bad string for CmdDoneHeaders.

* [NOD-1518] Fix bad Command() for MsgIBDRootNotFound.

* [NOD-1518] Fix bad areHeaderTipsSyncedMaxTimeDifference value.

* [NOD-1518] Add missing string for CmdRequestIBDBlocks.

* [NOD-1518] Fix bad check for SyncStateMissingBlockBodies.

* [NOD-1518] Fix bad timeout durations in tests.

* [NOD-1518] Fix IBD blocks not calling OnNewBlock.

* [NOD-1518] Change when IBD finishes.

* [NOD-1518] Properly clone utxoDiffChild.

* [NOD-1535] Fix reachability tests

* [NOD-1518] Fix merge errors.

* [NOD-1518] Move call to LogBlock to into OnNewBlock.

* [NOD-1518] Return "not implemented" in unimplemented RPC handlers.

* [NOD-1518] Extract cloning of hashes to a method over DomainHash.

* [NOD-1518] Use isHeaderOnlyBlock.

* [NOD-1518] Use constants.TransactionVersion.

* [NOD-1518] Break immediately if we reached the virtual in SelectedChildIterator.

* [NOD-1518] Don't stage nil utxoDiffChild.

* [NOD-1518] Properly check the genesis hash in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Explain why we break on current == nil in countSubtrees.

* [NOD-1518] Add a comment explaining why we check against StatusValid in resolveSyncState.

* [NOD-1535] Add external reachability tests

* [NOD-1535] Fix reachability tests and fix related bugs

* [NOD-1535] Add setters fox reindex slack and window

* [NOD-1535] Remove redundant line

* [NOD-1535] Add comment

* [NOD-1535] Fix comments

* [NOD-1535] Rename DBReader->DatabaseContext

* [NOD-1535] Check that reindex root is changed

* [NOD-1535] Fix calculateNewTips

Co-authored-by: Mike Zak <feanorr@gmail.com>
Co-authored-by: stasatdaglabs <stas@daglabs.com>
2020-11-17 16:00:16 +02:00
Svarog
8ccf381fc7 [NOD-1532] csm unit tests (#1059)
* Revert "[NOD-1500] Delete integration tests"

This reverts commit fcb57a206690a884fa6afb69d5d493282954a8bf.

* [NOD-1518] hashserialization -> consenusserialization

* [NOD-1518] Fix add genesis to virtual

* [NOD-1518] Fix a bug in SerializeCoinbasePayload.

* [NOD-1518] Fix a loop error and make pastMedianTime behave correctly everywhere on genesis.

* [NOD-1518] Fix another bug and an infinite loop.

* [NOD-1518] Fix uninitialized slice.

* [NOD-1518] Fix bad should-commit checks and another infinite loop.

* [NOD-1518] Fix nil serialization.

* [NOD-1518] Rename blockHash to currentBlockHash.

* [NOD-1518] Move the check whether stagedVirtualUTXOSet != nil to the top of commitVirtualUTXODiff.

* [NOD-1518] Simplify utxoDiffStore.Commit.

* [NOD-1518] Unextract resolveBlockStatusAndCheckFinality.

* [NOD-1518] Move no-transactions logic into CalculateIDMerkleRoot.

* [NOD-1518] Remove redundant is-staged check.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Don't write anything if utxoDiffChild is nil.

* [NOD-1518] Stage virtualAcceptanceData and virtualMultiset.

* [NOD-1518] Fix bugs in getBlockTemplate and submitBlock.

* [NOD-1518] Fix bad validation order in validateHeaderInContext.

* [NOD-1518] Fix bug in Next().

* [NOD-1518] Fix nil dereference of subnetworks in AddressCache.

* [NOD-1518] Fix multisetStore.Get returning a pointer to a multiset that is changed in place.

* [NOD-1518] Break on genesis in countSubtrees.

* [NOD-1518] Fix createBlockLocator.

* [NOD-1518] Fix MsgTxToDomainTransaction.

* [NOD-1518] Set MaxTxVersion to 1.

* [NOD-1518] Fix missing error handling, bug in MsgTxToDomainTransaction, and bad subnetwork equality check.

* [NOD-1518] Fix bug in hasUTXOByOutpointFromStagedVirtualUTXODiff.

* [NOD-1518] Remove irrelevant comments.

* [NOD-1518] Generate transactions with sufficient fee in tx_relay_test.

* [NOD-1518] Fix broken RPC handlers.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Fix bad exists check in restorePastUTXO and missing genesis check in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Add a comment.

* [NOD-1518] Use a regular mutex instead of a read-write mutex in consensus to avoid dealing with sneaky not-actually-read functions.

* [NOD-1518] Fix a deadlock in GetVirtualSelectedParent.

* [NOD-1518] Fix missing handler registration for CmdHeader.

* [NOD-1518] Fix processHeader calling OnNewBlock and LogBlock. Also fix conversion errors in IBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad Command() in MsgIBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad SyncStateMissingUTXOSet logic in resolveSyncState.

* [NOD-1518] Rename mode to syncState.

* [NOD-1518] Fix headers-only blocks coming in after the consensus thinks it's synced.

* [NOD-1518] Fix selectedChildIterator.Next not ignoring virtual, infinite loop in HashSet.Length().

* [NOD-1518] Fix not-properly wrapped IBD blocks.

* [NOD-1532] Add TestMultiset

* [NOD-1518] Fix bad conversion in RequestIBDBlocks.

* [NOD-1518] Fix bad string for CmdRequestHeaders.

* [NOD-1518] Fix bad string for CmdDoneHeaders.

* [NOD-1518] Fix bad Command() for MsgIBDRootNotFound.

* [NOD-1532] Add TestPastUTXOMultiset

* [NOD-1518] Fix bad areHeaderTipsSyncedMaxTimeDifference value.

* [NOD-1532] Added TestDoubleSpends

* [NOD-1518] Add missing string for CmdRequestIBDBlocks.

* [NOD-1518] Fix bad check for SyncStateMissingBlockBodies.

* [NOD-1518] Fix bad timeout durations in tests.

* [NOD-1518] Fix IBD blocks not calling OnNewBlock.

* [NOD-1518] Change when IBD finishes.

* [NOD-1518] Properly clone utxoDiffChild.

* [NOD-1532] Update hashes of blocks

* [NOD-1532] Fix genesis blocks and a few more bugs

* [NOD-1532] Bugfix: incorrect key passed to dbTx.Put

* [NOD-1532] Make sure there's no nil payloads

* [NOD-1532] Fix AddBlockToVirtual

* [NOD-1532] Update tips and virtualDiffParents properly

* [NOD-1532] Allow nil payload

* [NOD-1532] Check for actual error and not just some RuleError

* [NOD-1532] Get rid of SimpleCoinbaseData and make OpTrueScript P2SH

* [NOD-1532] If coinbaseData is nil - fill in with generic coinbaseData

Co-authored-by: Ori Newman <orinewman1@gmail.com>
Co-authored-by: stasatdaglabs <stas@daglabs.com>
2020-11-17 16:00:16 +02:00
stasatdaglabs
eef5e3768c
[NOD-1518] Fix genesis block insertion and integration tests (#1013)
* Revert "[NOD-1500] Delete integration tests"

This reverts commit fcb57a206690a884fa6afb69d5d493282954a8bf.

* [NOD-1518] hashserialization -> consenusserialization

* [NOD-1518] Fix add genesis to virtual

* [NOD-1518] Fix a bug in SerializeCoinbasePayload.

* [NOD-1518] Fix a loop error and make pastMedianTime behave correctly everywhere on genesis.

* [NOD-1518] Fix another bug and an infinite loop.

* [NOD-1518] Fix uninitialized slice.

* [NOD-1518] Fix bad should-commit checks and another infinite loop.

* [NOD-1518] Fix nil serialization.

* [NOD-1518] Rename blockHash to currentBlockHash.

* [NOD-1518] Move the check whether stagedVirtualUTXOSet != nil to the top of commitVirtualUTXODiff.

* [NOD-1518] Simplify utxoDiffStore.Commit.

* [NOD-1518] Unextract resolveBlockStatusAndCheckFinality.

* [NOD-1518] Move no-transactions logic into CalculateIDMerkleRoot.

* [NOD-1518] Remove redundant is-staged check.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Don't write anything if utxoDiffChild is nil.

* [NOD-1518] Stage virtualAcceptanceData and virtualMultiset.

* [NOD-1518] Fix bugs in getBlockTemplate and submitBlock.

* [NOD-1518] Fix bad validation order in validateHeaderInContext.

* [NOD-1518] Fix bug in Next().

* [NOD-1518] Fix nil dereference of subnetworks in AddressCache.

* [NOD-1518] Fix multisetStore.Get returning a pointer to a multiset that is changed in place.

* [NOD-1518] Break on genesis in countSubtrees.

* [NOD-1518] Fix createBlockLocator.

* [NOD-1518] Fix MsgTxToDomainTransaction.

* [NOD-1518] Set MaxTxVersion to 1.

* [NOD-1518] Fix missing error handling, bug in MsgTxToDomainTransaction, and bad subnetwork equality check.

* [NOD-1518] Fix bug in hasUTXOByOutpointFromStagedVirtualUTXODiff.

* [NOD-1518] Remove irrelevant comments.

* [NOD-1518] Generate transactions with sufficient fee in tx_relay_test.

* [NOD-1518] Fix broken RPC handlers.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Fix bad exists check in restorePastUTXO and missing genesis check in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Add a comment.

* [NOD-1518] Use a regular mutex instead of a read-write mutex in consensus to avoid dealing with sneaky not-actually-read functions.

* [NOD-1518] Fix a deadlock in GetVirtualSelectedParent.

* [NOD-1518] Fix missing handler registration for CmdHeader.

* [NOD-1518] Fix processHeader calling OnNewBlock and LogBlock. Also fix conversion errors in IBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad Command() in MsgIBDRootUTXOSetAndBlock.

* [NOD-1518] Fix bad SyncStateMissingUTXOSet logic in resolveSyncState.

* [NOD-1518] Rename mode to syncState.

* [NOD-1518] Fix headers-only blocks coming in after the consensus thinks it's synced.

* [NOD-1518] Fix selectedChildIterator.Next not ignoring virtual, infinite loop in HashSet.Length().

* [NOD-1518] Fix not-properly wrapped IBD blocks.

* [NOD-1518] Fix bad conversion in RequestIBDBlocks.

* [NOD-1518] Fix bad string for CmdRequestHeaders.

* [NOD-1518] Fix bad string for CmdDoneHeaders.

* [NOD-1518] Fix bad Command() for MsgIBDRootNotFound.

* [NOD-1518] Fix bad areHeaderTipsSyncedMaxTimeDifference value.

* [NOD-1518] Add missing string for CmdRequestIBDBlocks.

* [NOD-1518] Fix bad check for SyncStateMissingBlockBodies.

* [NOD-1518] Fix bad timeout durations in tests.

* [NOD-1518] Fix IBD blocks not calling OnNewBlock.

* [NOD-1518] Change when IBD finishes.

* [NOD-1518] Properly clone utxoDiffChild.

* [NOD-1518] Fix merge errors.

* [NOD-1518] Move call to LogBlock to into OnNewBlock.

* [NOD-1518] Return "not implemented" in unimplemented RPC handlers.

* [NOD-1518] Extract cloning of hashes to a method over DomainHash.

* [NOD-1518] Use isHeaderOnlyBlock.

* [NOD-1518] Use constants.TransactionVersion.

* [NOD-1518] Break immediately if we reached the virtual in SelectedChildIterator.

* [NOD-1518] Don't stage nil utxoDiffChild.

* [NOD-1518] Properly check the genesis hash in CalculatePastUTXOAndAcceptanceData.

* [NOD-1518] Explain why we break on current == nil in countSubtrees.

* [NOD-1518] Add a comment explaining why we check against StatusValid in resolveSyncState.

Co-authored-by: Mike Zak <feanorr@gmail.com>
Co-authored-by: Ori Newman <orinewman1@gmail.com>
2020-11-12 15:19:39 +02:00
Svarog
37fbdcb453
[NOD-1526] Restore txscript tests (#1019)
* [NOD-1526] Fix compilation errors

* [NOD-1526] Make MsgTx.PayloadHash non-pointer

* [NOD-1526] Fixed many tests

* [NOD-1526] Fix reference_test.go

* [NOD-1526] Removed last instances of appmessage in consensus

* [NOD-1526] No need to check for subnetwork
2020-11-12 10:22:17 +02:00
Svarog
135ffbd4f2
[NOD-1529] Add getters + AddBlock to TestConsensus (#1025)
* [NOD-1529] Add all stores and processes to consensus, and add access to TestConsensus

* [NOD-1529] Move the getters of TestConsensus to separate file

* [NOD-1529] Add AddBlock to TestConsensus

* [NOD-1529] Update NewTestConsensus to be more all-encompassing

* [NOD-1529] Remove test directory in teardown

* [NOD-1529] Add ForAllNets function

* [NOD-1529] Add comment
2020-11-11 12:31:13 +02:00