18 Commits

Author SHA1 Message Date
Svarog
7c1cddff11
Address Manager: Don't seed if we have enough Addresses + Remove Address that failed connection too many times (#1899)
* Seed from DNS only if we ran out of addresses to connect to

* Remove address if it has failed 10 connections

* Change connectionFailedCountForRemove to 3
2021-12-29 20:34:32 +02:00
Ori Newman
765dd170e4
Optimizations and header size reduce hardfork (#1853)
* Modify DefaultTimeout to 120 seconds

A temporary workaround for nodes having trouble to sync (currently the download of pruning point related data during IBD takes more than 30 seconds)

* Cache existence in reachability store

* Cache block level in the header

* Fix IBD indication on submit block

* Add hardForkOmitGenesisFromParentsDAAScore logic

* Fix NumThreads bug in the wallet

* Get rid of ParentsAtLevel header method

* Fix a bug in BuildPruningPointProof

* Increase race detector timeout

* Add cache to BuildPruningPointProof

* Add comments and temp comment out go vet

* Fix ParentsAtLevel

* Dont fill empty parents

* Change HardForkOmitGenesisFromParentsDAAScore in fast netsync test

* Add --allow-submit-block-when-not-synced in stability tests

* Fix TestPruning

* Return fast tests

* Fix off by one error on kaspawallet

* Fetch only one block with trusted data at a time

* Update fork DAA score

* Don't ban for unexpected message type

* Fix tests

Co-authored-by: Michael Sutton <mikisiton2@gmail.com>
Co-authored-by: Ori Newman <>
2021-11-22 09:00:39 +02:00
Ori Newman
5dbb1da84b
Implement pruning point proof (#1832)
* Calculate GHOSTDAG, reachability etc for each level

* Don't preallocate cache for dag stores except level 0 and reduce the number of connections in the integration test to 32

* Reduce the number of connections in the integration test to 16

* Increase page file

* BuildPruningPointProof

* BuildPruningPointProof

* Add PruningProofManager

* Implement ApplyPruningPointProof

* Add prefix and fix blockAtDepth and fill headersByLevel

* Some bug fixes

* Include all relevant blocks for each level in the proof

* Fix syncAndValidatePruningPointProof to return the right block hash

* Fix block window

* Fix isAncestorOfPruningPoint

* Ban for rule errors on pruning proof

* Find common ancestor for blockAtDepthMAtNextLevel

* Use pruning proof in TestValidateAndInsertImportedPruningPoint

* stage status and finality point for proof blocks

* Uncomment golint

* Change test timeouts

* Calculate merge set for ApplyPruningPointProof

* Increase test timeout

* Add better caching for daa window store

* Return to default timeout

* Add ErrPruningProofMissesBlocksBelowPruningPoint

* Add errDAAWindowBlockNotFound

* Force connection loop next iteration on connection manager stop

* Revert to Test64IncomingConnections

* Remove BlockAtDepth from DAGTraversalManager

* numBullies->16

* Set page file size to 8gb

* Increase p2p max message size

* Test64IncomingConnections->Test16IncomingConnections

* Add comment for PruningProofM

* Add comment in `func (c *ConnectionManager) Stop()`

* Rename isAncestorOfPruningPoint->isAncestorOfSelectedTip

* Revert page file to 16gb

* Improve ExpectedHeaderPruningPoint perf

* Fix comment

* Revert "Improve ExpectedHeaderPruningPoint perf"

This reverts commit bca1080e7140c78d510f51bbea858ae280c2f38e.

* Don't test windows
2021-10-26 09:48:27 +03:00
Constantine Bitensky
79c74c482b
Get connections from Seeder when no connections left (#1742)
Co-authored-by: Constantine Bitensky <constantinebitensky@gmail.com>
2021-06-08 17:42:13 +03:00
stasatdaglabs
a795a9e619
Add a size limit to the address manager (#1652)
* Remove a random address from the address manager if it's full.

* Implement TestOverfillAddressManager.

* Add connectionFailedCount to addresses.

* Mark connection failures.

* Mark connection successes.

* Implement removing by most connection failures.

* Expand TestOverfillAddressManager.

* Add comments.

* Use a better method for finding the address with the greatest connectionFailedCount.

* Fix a comment.

* Compare addresses by IP in TestOverfillAddressManager.

* Add a comment for updateNotBanned.

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-04-05 17:56:13 +03:00
stasatdaglabs
cbd0bb6d14
Remove the Services field from NetAddress. (#1610) 2021-03-16 14:22:52 +02:00
Elichai Turkel
1f69f9eed9
Cleanup the logger and make it asynchronous (#1524)
* Remove Subsystems map and replace with RegisterSubSystem

* Clean up the logger

* Fix LOGFLAGS and make LongFile work correctly

* Parallelize the logger backend

* More logger cleanup

* Initialize and close the logger backend wherever it's needed

* Move the location where the backend is closed, also print the log if it panics while writing

* Add TestMain to reachability manager tests to preserve the same log level

* Fix review comments

Co-authored-by: Svarog <feanorr@gmail.com>
2021-03-01 14:04:40 +02:00
stasatdaglabs
d835f72e74
Make AddressManager persistent (#1525)
* Move existing address/bannedAddress functionality to a new addressStore object.

* Implement TestAddressManager.

* Implement serializeAddressKey and deserializeAddressKey.

* Implement serializeNetAddress and deserializeNetAddress.

* Store addresses and banned addresses to disk.

* Implement restoreNotBannedAddresses.

* Fix bannedDatabaseKey.

* Implement restoreBannedAddresses.

* Implement TestRestoreAddressManager.

* Defer closing the database in TestRestoreAddressManager.

* Defer closing the database in TestRestoreAddressManager.

* Add a log.

* Return errors where appropriate.

Co-authored-by: Elichai Turkel <elichai.turkel@gmail.com>
2021-02-14 19:08:06 +02:00
Ori Newman
b636ae234e
Add ban and unban RPC commands (#1478)
* Add ban and unban RPC commands

* Fix names

* Fix commands strings

* Update RPC documentation

* Rename functions

* Simplify return

* Use IP strings in app messages

* Add parse IP error

* Fix wrong condition
2021-02-01 17:34:43 +02:00
Elichai Turkel
12c438d389
Fix data races in ConnectionManager and flow tests (#1474)
* Reuse the ticker in ConnectionManager.waitTillNextIteration

* Fix a data race in ConnectionManager by locking the mutex

* Add a mutex to fakeRelayInvsContext in block relay flow test

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-02-01 15:03:31 +02:00
Elichai Turkel
280fa3de46
Prevent infinite ticker leaks in kaspaminer (#1476)
* Prevent infinite tickers leaks in kaspaminer

* Reset ticker in ConnectionManager instead of allocating a new one

Co-authored-by: Ori Newman <orinewman1@gmail.com>
2021-02-01 14:52:17 +02:00
Ori Newman
669a9ab4c3
Ban by IP (#1471)
* Ban by IP

* Fix panic

* Fix error format

* Remove failed addresses

Co-authored-by: stasatdaglabs <39559713+stasatdaglabs@users.noreply.github.com>
2021-02-01 10:51:18 +02:00
Ori Newman
7a7821e1c8
[NOD-1313] Refactor AddressManager (#918) (#1049)
* [NOD-1313] Refactor AddressManager (#918)

* [NOD-1313] Refactor AddressManager.

* [NOD-1313]Remove old tests.Minor improvements,fixes.

* [NOD-1313] After merge fixes. Fix import cycle.

* [NOD-1313] Integration tests fixes.

* [NOD-1313] Allocate new slice for the returned key.

* [NOD-1313] AddressManager improvements and fixes.

* Move local and banned addresses to separate lists.
* Move AddressManager config to the separate file.
* Add LocalAddressManager.
* Remove redundant KnownAddress structure.
* Restore local addresses functionality.
* Call initListeners from the LocalAddressManager.
* AddressManager minor improvements and fixes.

* [NOD-1313] Minor fixes.

* [NOD-1313] Implement HandleGetPeerAddresses. Refactoring.

* [NOD-1313] After-merge fixes.

* [NOD-1313] Minor improvements.

* AddressManager: added BannedAddresses() method.
* AddressManager: HandleGetPeerAddresses() add banned addresses
  separately.
* AddressManager: remove addressEntry redundant struct.
* ConnectionManager: checkOutgoingConnections() minor improvements and
  fixes.
* Minor refactoring.
* Minor fixes.

* [NOD-1313] GetPeerAddresses RPC message update

* GetPeerAddresses RPC: add BannedAddresses in the separate field.
* Update protobuf.

* [NOD-1534] Update messages.pb.go

Co-authored-by: Kirill <gammerxpower@gmail.com>
2020-11-12 14:40:41 +02:00
stasatdaglabs
4c915f12b7
[NOD-1319] Reimplement kaspad's RPC in gRPC (#914)
* [NOD-1319] Create the protorpc package.

* [NOD-1319] Use a general ClientStream in closeSend.

* [NOD-1319] Decouple p2pServer from gRPCServer.

* [NOD-1319] Begin implementing rpcServer.

* [NOD-1319] Move grpcStream to grpc_connection.go.

* [NOD-1319] Fold the rpc messages.proto into a common message.proto.

* [NOD-1319] Remove code duplication in MessageStream.

* [NOD-1319] Rename methods in netadapter.

* [NOD-1319] Rename message_xxx to p2p_xxx.

* [NOD-1319] Section off p2p messages from rpc messages in messages.proto.

* [NOD-1319] Split toPayload to a p2p part and and rpc part.

* [NOD-1319] Rename msgxxx.go to p2p_msgxx.go in the appmessage package.

* [NOD-1319] Implement GetCurrentVersionRequestMessage and GetCurrentVersionResponseMessage.

* [NOD-1319] Implement toAppMessage and fromAppMessage for getCurrentNetwork

* [NOD-1319] Make a temporary workaround so that tests pass.

* [NOD-1319] Begin implementing the rpc manager.

* [NOD-1319] Implement an initial routerInitializer for rpc.

* [NOD-1319] Rename the spawn in routerInitializer.

* [NOD-1319] Implement an RPC context.

* [NOD-1319] Move the actual handlers to a separate package.

* [NOD-1319] Use the correct value for the GetCurrentNetwork response.

* [NOD-1319] Fix some names.

* [NOD-1319] Begin implementing a cli rpc client.

* [NOD-1319] Implement connecting to the RPC server.

* [NOD-1319] Make sure that connecting to the server and sending/receiving messages works.

* [NOD-1319] Make kaspactl2 speak in json strings.

* [NOD-1319] Finish implementing kaspactl2.

* [NOD-1319] Remove debug messages.

* [NOD-1319] Properly handle errors in rpc.go.

* [NOD-1319] Move the grpc client to a separate package.

* [NOD-1319] Extract Post out of PostString.

* [NOD-1319] Implement PostAppMessage.

* [NOD-1319] Stub out submitBlock.

* [NOD-1319] Stub out getBlockTemplate.

* [NOD-1319] Combine request and reponse files.

* [NOD-1319] Implement submitBlock.

* [NOD-1319] Implement returning errors from RPC.

* [NOD-1319] Begin implementing getBlockTemplate.

* [NOD-1319] Add missing field in GetBlockTemplateRequestMessage.

* [NOD-1319] Implement a minimal getBlockTemplate.

* [NOD-1319] Add getBlockTemplate stuff to grpc.

* [NOD-1319] Implement the rest of getBlockTemplate.

* [NOD-1319] Add block/transaction added handlers to the protocol manager.

* [NOD-1319] Implement NotifyTransactionAddedToMempool.

* [NOD-1319] Implement NotifyBlockAddedToDAG.

* [NOD-1319] Connect block/transaction added handlers.

* [NOD-1319] Add notifyBlockAdded.

* [NOD-1319] Add a notification system.

* [NOD-1319] Improve the notification system.

* [NOD-1319] Add a block added listener stub.

* [NOD-1319] Add BlockAddedNotificationMessage.

* [NOD-1319] Finish implementing HandleNotifyBlockAdded.

* [NOD-1319] Println instead of Print in kaspactl2.

* [NOD-1319] Remove unused flags in kaspactl2.

* [NOD-1319] Make kaspaminer work with the new RPC.

* [NOD-1319] Fix a bad log.

* [NOD-1319] Make kaspaminer work.

* [NOD-1319] Disconnect the old RPC.

* [NOD-1319] Move grpcclient.go.

* [NOD-1319] Begin generalizing the rpcClient.

* [NOD-1319] Move errors to the side of the payload.

* [NOD-1319] Add errors to appmessage.

* [NOD-1319] Move AttachRouter to grpcclient.

* [NOD-1319] Fix kaspaminer not handling responses.

* [NOD-1319] Properly handle blockAddedNotifications.

* [NOD-1319] Move errors into individual response objects.

* [NOD-1319] Begin replacing the RPC client in the integration tests.

* [NOD-1319] Implement GetPeerAddresses.

* [NOD-1319] Implement GetPeerAddresses.

* [NOD-1319] Fix setOnBlockAddedHandler.

* [NOD-1319] Remove the old kaspactl.

* [NOD-1319] Move ConvertGetBlockTemplateResultToBlock to the mining package.

* [NOD-1319] Implement getSelectedTipHash.

* [NOD-1319] Simplify testRPCRouter.

* [NOD-1319] Write stubs for the required test RPC commands.

* [NOD-1319] Implement a minimal getMempoolEntry.

* [NOD-1319] Implement a minimal getMempoolEntry.

* [NOD-1319] Implement getConnectedPeerInfo.

* [NOD-1319] Delete the old RPC.

* [NOD-1319] Fix a fromAppMessage.

* [NOD-1319] Implement connectToPeer.

* [NOD-1319] Fix a bug in registerForBlockAddedNotifications.

* [NOD-1319] Fix a deadlock in closing notification listeners.

* [NOD-1319] Fix merge errors.

* [NOD-1319] Fix an import.

* [NOD-1319] Properly handle errors in grpcclient.

* [NOD-1319] Fix TestIBD.

* [NOD-1319] Prevent kaspaminer from running when not connected.

* [NOD-1319] Implement sendRawTransaction.

* [NOD-1319] Implement sendRawTransaction in the client.

* [NOD-1319] Extract a general RPC client from the integration test RPC client.

* [NOD-1319] Use the general RPC client for the miner.

* [NOD-1319] Move the rpcclient package out of netadapter.

* [NOD-1319] Normalize onBlockAdded.

* [NOD-1319] Begin implementing notifyChainChanged.

* [NOD-1319] Implement the model for notifyChainChanged.

* [NOD-1319] Implement conversions for notifyChainChanged.

* [NOD-1319] Implement HandleNotifyChainChanged.

* [NOD-1319] Normalize notifications.

* [NOD-1319] Implement RegisterForChainChangedNotifications.

* [NOD-1319] Begin connecting blockdag's chain-changed notification with the RPC.

* [NOD-1319] Finish implementing notifyChainChanged.

* [NOD-1319] Implement getBlockHex.

* [NOD-1319] Rename getBlockHex to getBlock.

* [NOD-1319] Implement the verbose model for getBlock.

* [NOD-1319] Implement buildBlockVerboseData.

* [NOD-1319] Implement buildTransactionVerboseData.

* [NOD-1319] Move verboseData stuff to verbosedata.go.

* [NOD-1319] Add includeTransactionVerboseData.

* [NOD-1319] Begin implementing getSubnetwork.

* [NOD-1319] Finish implementing getSubnetwork.

* [NOD-1319] Begin implementing getChainFromBlock.

* [NOD-1319] Finish implementing getChainFromBlock.

* [NOD-1319] Begin implementing getBlocks.

* [NOD-1319] Finish implementing getBlocks.

* [NOD-1319] Fix bad responses in HandleNotifyChainChanged.

* [NOD-1319] Fix bugs in verbosedata.go.

* [NOD-1319] Fix more bugs in verbosedata.go.

* [NOD-1319] Make go vet happy.

* [NOD-1319] Extract handleBlockDAGNotifications to a method.

* [NOD-1319] Add a newline.

* [NOD-1319] Use peers instead of connections to check if connected.

* [NOD-1319] Add a comment.

* [NOD-1319] Lock the dag lock in getBlock.

* [NOD-1319] Rename netAdapter.connections to p2pConnections.

* [NOD-1319] In protowire, rename wireXXX to protoXXX.

* [NOD-1319] Rename PostString to PostJSON.

* [NOD-1319] Disallow empty transactions in SendRawTransaction.

* [NOD-1319] Disallow empty blocks in SubmitBlocks.

* [NOD-1319] Add SetLogger.

* [NOD-1319] Fix an error message.

* [NOD-1319] Fix an error message.

* [NOD-1319] Rename testTimeout to rpcTimeout.

* [NOD-1319] Rename SendRawTransaction to SubmitTransaction.

* [NOD-1319] Rename ConnectToPeer to AddPeer.

* [NOD-1319] Add missing longPollID to request.

* [NOD-1319] Rename ChainChangedChainBlock to ChainBlock.

* [NOD-1319] Rename Vin and Vout.

* [NOD-1319] Implement RPCErrorf.

* [NOD-1319] Fix RPCErrorf's comment.

* [NOD-1319] Remove unused flags in kaspaminer.
2020-09-07 14:35:40 +03:00
Ori Newman
c331293a2e
[NOD-1289] Check if connection exists before establishing another one with the same address (#883) 2020-08-20 11:50:29 +03:00
Ori Newman
fcae491e6d
[NOD-1286] Close router from netConnection.Disconnect (#881)
* [NOD-1286] Close router from netConnection.Disconnect

* [NOD-1286] Close router in grpc errors as well

* [NOD-1286] Fix typo

* [NOD-1286] Rename isConnected->isRouterClosed
2020-08-19 17:28:01 +03:00
Ori Newman
8dae378bd9
[NOD-1285] Fix deadlock on connection manager (#880) 2020-08-19 13:24:20 +03:00
stasatdaglabs
d14809694f
[NOD-1223] Reorganize directory structure (#874)
* [NOD-1223] Delete unused files/packages.

* [NOD-1223] Move signal and limits to the os package.

* [NOD-1223] Put database and dbaccess into the db package.

* [NOD-1223] Fold the logs package into the logger package.

* [NOD-1223] Rename domainmessage to appmessage.

* [NOD-1223] Rename to/from DomainMessage to AppMessage.

* [NOD-1223] Move appmessage to the app packge.

* [NOD-1223] Move protocol to the app packge.

* [NOD-1223] Move the network package to the infrastructure packge.

* [NOD-1223] Rename cmd to executables.

* [NOD-1223] Fix go.doc in the logger package.
2020-08-18 10:26:39 +03:00