19 Commits

Author SHA1 Message Date
Ori Newman
afaac28da1
Validate each level parents (#1827)
* Create BlockParentBuilder.

* Implement BuildParents.

* Explictly set level 0 blocks to be the same as direct parents.

* Add checkIndirectParents to validateBlockHeaderInContext.

* Fix test_block_builder.go and BlockLevelParents::Equal.

* Don't check indirect parents for blocks with trusted data.

* Handle pruned blocks when building block level parents.

* Fix bad deletions from unprocessedXxxParents.

* Fix merge errors.

* Fix bad pruning point parent replaces.

* Fix duplicates in newBlockLevelParents.

* Skip checkIndirectParents

* Get rid of staging constant IDs

* Fix BuildParents

* Fix tests

* Add comments

* Change order of directParentHashes

* Get rid of maybeAddDirectParentParents

* Add comments

* Add blockToReferences type

* Use ParentsAtLevel

Co-authored-by: stasatdaglabs <stas@daglabs.com>
2021-09-13 14:22:00 +03:00
Ori Newman
4207c82f5a
Add database prefix (#1750)
* Add prefix to stores

* Add prefix to forgotten stores

* Add a special type for prefix

* Rename transaction->dbTx

* Change error message

* Use countKeyName

* Rename Temporary Consesnsus to Staging

* Add DeleteStagingConsensus to Domain interface

* Add lock to staging consensus

* Make prefix type-safer

* Use ioutil.TempDir instead of t.TempDir
2021-06-15 17:47:17 +03:00
Svarog
f1451406f7
Add support for multiple staging areas (#1633)
* Add StagingArea struct

* Implemented staging areas in blockStore

* Move blockStagingShard to separate folder

* Apply staging shard to acceptanceDataStore

* Update blockHeaderStore with StagingArea

* Add StagingArea to BlockRelationStore

* Add StagingArea to blockStatusStore

* Add StagingArea to consensusStateStore

* Add StagingArea to daaBlocksStore

* Add StagingArea to finalityStore

* Add StagingArea to ghostdagDataStore

* Add StagingArea to headersSelectedChainStore and headersSelectedTipStore

* Add StagingArea to multisetStore

* Add StagingArea to pruningStore

* Add StagingArea to reachabilityDataStore

* Add StagingArea to utxoDiffStore

* Fix forgotten compilation error

* Update reachability manager and some more things with StagingArea

* Add StagingArea to dagTopologyManager, and some more

* Add StagingArea to GHOSTDAGManager, and some more

* Add StagingArea to difficultyManager, and some more

* Add StagingArea to dagTraversalManager, and some more

* Add StagingArea to headerTipsManager, and some more

* Add StagingArea to constnsusStateManager, pastMedianTimeManager

* Add StagingArea to transactionValidator

* Add StagingArea to finalityManager

* Add StagingArea to mergeDepthManager

* Add StagingArea to pruningManager

* Add StagingArea to rest of ValidateAndInsertBlock

* Add StagingArea to blockValidator

* Add StagingArea to coinbaseManager

* Add StagingArea to syncManager

* Add StagingArea to blockBuilder

* Update consensus with StagingArea

* Add StagingArea to ghostdag2

* Fix remaining compilation errors

* Update names of stagingShards

* Fix forgotten stagingArea passing

* Mark stagingShard.isCommited = true once commited

* Move isStaged to stagingShard, so that it's available without going through store

* Make blockHeaderStore count be avilable from stagingShard

* Fix remaining forgotten stagingArea passing

* commitAllChanges should call dbTx.Commit in the end

* Fix all tests tests in blockValidator

* Fix all tests in consensusStateManager and some more

* Fix all tests in pruningManager

* Add many missing stagingAreas in tests

* Fix many tests

* Fix most of all other tests

* Fix ghostdag_test.go

* Add comment to StagingArea

* Make list of StagingShards an array

* Add comment to StagingShardID

* Make sure all staging shards are pointer-receiver

* Undo bucket rename in block_store

* Typo: isCommited -> isCommitted

* Add comment explaining why stagingArea.shards is an array
2021-03-29 10:34:11 +03:00
Elichai Turkel
2edf6bfd07
Minimize memory usage in tests (#1495)
* Make leveldb cache configurable

* Fix leveldb tests

* Add a preallocate option to all caches and disable in tests

Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-02-08 18:37:02 +02:00
Svarog
ad9c213a06
Restructure database to prevent double-slashes in keys, causing bugs in cursors (#1432)
* Add TestValidateAndInsertPruningPointWithSideBlocks

* Optimize infrastracture bucket paths

* Update infrastracture tests

* Refactor the consensus/database layer

* Remove utils/dbkeys

* Use consensus/database in consensus instead of infrastructure

* Fix a bug in dbBucketToDatabaseBucket and MakeBucket combination

Co-authored-by: Elichai Turkel <elichai.turkel@gmail.com>
Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-01-19 14:19:08 +02:00
Ori Newman
49b6cc6038
Add mutable and immutable header interfaces (#1305)
* Add mutable and immutable header interfaces

* Fix ShouldMine()

* Remove false comment

* Fix Equal signature

* Fix Equal implementation
2020-12-29 13:55:17 +02:00
Svarog
05941a76e7
Make DomainHash and TransactionID read-only structs (#1282)
* Increase size of reachability cache

* Change DomainHash to struct with unexported hashArray

* Fixing compilation errors stemming from new DomainHash structure

* Remove obsolete Read/WriteElement methods in appmessage

* Fix all tests

* Fix all tests

* Add comments

* A few renamings

* go mod tidy
2020-12-24 16:15:23 +02:00
stasatdaglabs
0fa13357c3
[NOD-1566] Add caching to all stores (#1152)
* [NOD-1566] Add a dependency to golang-lru.

* [NOD-1566] Add caching to blockstore.go.

* [NOD-1566] Add LRUCache to all store objects and initialize them.

* [NOD-1566] Add caching to acceptanceDataStore.

* [NOD-1566] Add caching to blockHeaderStore.

* [NOD-1566] Implement a simpler LRU cache.

* [NOD-1566] Use the simpler cache implementation everywhere.

* [NOD-1566] Remove dependency in golang-lru.

* [NOD-1566] Fix object reuse issues in store Get functions.

* [NOD-1566] Add caching to blockRelationStore.

* [NOD-1566] Add caching to blockStatusStore.

* [NOD-1566] Add caching to ghostdagDataStore.

* [NOD-1566] Add caching to multisetStore.

* [NOD-1566] Add caching to reachabilityDataStore.

* [NOD-1566] Add caching to utxoDiffStore.

* [NOD-1566] Add caching to reachabilityReindexRoot.

* [NOD-1566] Add caching to pruningStore.

* [NOD-1566] Add caching to headerTipsStore.

* [NOD-1566] Add caching to consensusStateStore.

* [NOD-1566] Add comments explaining why we don't discard staging at the normal location in consensusStateStore.

* [NOD-1566] Make go vet happy.

* [NOD-1566] Fix merge errors.

* [NOD-1566] Add a missing break statement.

* [NOD-1566] Run go mod tidy.

* [NOD-1566] Remove serializedUTXOSetCache.
2020-11-25 13:41:13 +02:00
Ori Newman
45d9b63572
[NOD-1567] Add clone methods to data stores types (#1149)
* [NOD-1567] Add clone methods to data stores types

* [NOD-1567] Fix comments

* [NOD-1567] Fix test
2020-11-24 17:56:18 +02:00
stasatdaglabs
bb244706ea
[NOD-1543] Optimize the performance of Count() in BlockHeaderStore and BlockStore (#1109)
* [NOD-1543] Optimize Count() in BlockHeaderStore.

* [NOD-1543] Optimize Count() in BlockStore.

* [NOD-1543] Fix commitCount.

* [NOD-1543] Explicitly initialize count to 0.
2020-11-18 16:35:32 +02:00
stasatdaglabs
48d8137604 [NOD-1538] Implement GetBlockCount. 2020-11-17 16:00:16 +02:00
Ori Newman
4736213ba4
[NOD-1528] Make data stores copy data on stage (#1020)
* [NOD-1528] Make data stores copy data on stage

* [NOD-1528] Add proto objects to serialize consensus state objects

* [NOD-1528] Fix receiver names

* [NOD-1528] Add copy to block store and utxo diff staging

* [NOD-1528] Return errors where needed
2020-11-10 18:32:42 +02:00
Ori Newman
c59adaa4db
[NOD-1494] Remove transactions from data store delete (#984)
* [NOD-1494] Remove transactions from data store delete

* [NOD-1494] Remove redundant underscores
2020-10-29 18:15:11 +02:00
stasatdaglabs
01c7c67aed
[NOD-1493] Implement serialization in AcceptanceDataStore, BlockRelationStore, BlockStatusStore, and BlockStore (#982)
* [NOD-1493] Add DbHashToDomainHash and DomainHashToDbHash.

* [NOD-1493] Use DbHashToDomainHash and DomainHashToDbHash.

* [NOD-1493] Begin implementing serializeAcceptanceData.

* [NOD-1493] Extract serialization blockHeader logic to serialization.

* [NOD-1493] Extract serialization acceptance data logic to serialization.

* [NOD-1493] Implement acceptance data serialization/deserialization.

* [NOD-1493] Implement transaction serialization/deserialization.

* [NOD-1493] Implement outpoint serialization/deserialization.

* [NOD-1493] Implement transaction ID serialization/deserialization.

* [NOD-1493] Implement subnetwork ID serialization/deserialization.

* [NOD-1493] Implement block relation serialization/deserialization.

* [NOD-1493] Implement block status serialization/deserialization.

* [NOD-1493] Implement block serialization/deserialization.

* [NOD-1493] Implement serialization/deserialization in BlockRelationStore.

* [NOD-1493] Implement serialization/deserialization in BlockStatusStore.

* [NOD-1493] Implement serialization/deserialization in BlockStore.

* [NOD-1493] Make go vet happy.

* [NOD-1493] Use DomainHashesToDbHashes.
2020-10-29 16:49:05 +02:00
stasatdaglabs
9cf1557c37
[NOD-1493] Implement types for serialization (#980)
* [NOD-1493] Add DbAcceptanceData.

* [NOD-1493] Add DbBlockRelations.

* [NOD-1493] Add DbBlockStatus.

* [NOD-1493] Add DbBlockGhostdagData.

* [NOD-1493] Add DbMultiset.

* [NOD-1493] Add DbPruningPoint.

* [NOD-1493] Add DbUtxoSet.

* [NOD-1493] Add DbReachabilityData.

* [NOD-1493] Add DbReachabilityReindexRoot.

* [NOD-1493] Add DbUtxoDiff.

* [NOD-1493] Add DbUtxoDiffChild.

* [NOD-1493] Make sure everything is lowercase.

* [NOD-1493] Add DbHash.

* [NOD-1493] Fix BlockHeaderStore.
2020-10-29 12:18:18 +02:00
stasatdaglabs
126e2e49bb
[NOD-1493] Implement serialization/deserialization inside BlockHeaderStore (#979)
* [NOD-1492] Rename dbmanager to database.

* [NOD-1492] Write messages.proto for DbBlock and DbTransaction.

* [NOD-1492] Implement serializeHeader.

* [NOD-1492] Implement deserializeHeader.
2020-10-29 11:15:14 +02:00
Ori Newman
eae8bce941
[NOD-1491] Implement block headers store (#976)
* [NOD-1491] Implement block headers store

* [NOD-1491] Don't commit transaction and delete from staging too
2020-10-28 14:34:00 +02:00
Ori Newman
be56fb7e8b
[NOD-1488] Get rid of dbaccess (#973)
* [NOD-1488] Get rid of dbaccess

* [NOD-1488] Rename dbwrapper to dbmanager

* [NOD-1488] Create DBWriter interface

* [NOD-1488] Fix block header store

* [NOD-1488] Rename dbwrapper.go to dbmanager.go
2020-10-28 11:34:06 +02:00
stasatdaglabs
4fbe130592
[NOD-1489] Add BlockHeaderStore (#974)
* [NOD-1489] Add BlockHeaderStore.

* [NOD-1489] Use BlockHeaderStore.
2020-10-28 10:35:18 +02:00