Commit graph

478 commits

Author SHA1 Message Date
Hennadii Stepanov
02ccf69dd6
refactor: Move port mapping code to its own module
This commit does not change behavior.
2021-01-07 18:06:58 +02:00
Wladimir J. van der Laan
f1dbf92ff0
Merge #20434: contrib: Parse ELF directly for symbol and security checks
a0a771843f contrib: Changes to checks for PowerPC64 (Luke Dashjr)
634f6ec4eb contrib: Parse ELF directly for symbol and security checks (Wladimir J. van der Laan)

Pull request description:

  Instead of the ever-messier text parsing of the output of the readelf tool (which is clearly meant for human consumption not to be machine parseable), parse the ELF binaries directly.

  Add a small dependency-less ELF parser specific to the checks.

  This is slightly more secure, too, because it removes potential ambiguity due to misparsing and changes in the output format of `elfread`. It also allows for stricter and more specific ELF format checks in the future.

  This removes the build-time dependency for `readelf`.

  It passes the test-security-check for me locally, ~~though I haven't checked on all platforms~~. I've checked that this works on the cross-compile output for all ELF platforms supported by Bitcoin Core at the moment, as well as PPC64 LE and BE.

Top commit has no ACKs.

Tree-SHA512: 7f9241fec83ee512642fecf5afd90546964561efd8c8c0f99826dcf6660604a4db2b7255e1afb1e9bb0211fd06f5dbad18a6175dfc03e39761a40025118e7bfc
2020-12-18 12:16:40 +01:00
Andrew Chow
e1e7a90d5f wallettool: Add dump command
Adds a new dump command to bitcoin-wallet which prints out all of the
wallet's records in hex.
2020-12-16 12:32:47 -05:00
MarcoFalke
24e4857b29
Merge #20494: refactor: Move node and wallet code out of src/interfaces
629a9299b2 Move WalletImpl from interfaces/wallet.cpp to wallet/interfaces.cpp (Russell Yanofsky)
2a26771d81 Move ChainImpl from interfaces/chain.cpp to node/interfaces.cpp (Russell Yanofsky)
12bd0fc9d7 Move NodeImpl from interfaces/node.cpp to node/interfaces.cpp (Russell Yanofsky)

Pull request description:

  This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10).

  ---

  Move `NodeImpl` from `interfaces/node.cpp` to `node/interfaces.cpp`
  Move `ChainImpl` from `interfaces/chain.cpp` to `node/interfaces.cpp`
  Move `WalletImpl` from `interfaces/wallet.cpp` to `wallet/interfaces.cpp`

  No changes to any classes (can review with `git diff --color-moved=dimmed_zebra`)

  Motivation for this change is to move node and wallet code to respective directories where it might fit in better than `src/interfaces/`, but also to remove all unnecessary code from `src/interfaces/` to unblock #19160 review, which has been hung up partially because of code organization. Building on top of this PR, #19160 should now be able to organize interface implementations more understandably in `src/node/` `src/wallet/` `src/ipc/` and `src/init/` directories instead of having so much functionality all in `src/interfaces/`

ACKs for top commit:
  promag:
    Code review ACK 629a9299b2.
  MarcoFalke:
    review ACK 629a9299b2 🔺

Tree-SHA512: 87c2b8fd51519bbd4e5ad3539a79debcf88c3bf021eb28c63f3f555186538b62a0c4cc1a3f07cfb4ff13aea8b0b2fdde505d81f22a5e5fd12a6e375b55a92ab8
2020-12-01 09:39:56 +01:00
Russell Yanofsky
629a9299b2 Move WalletImpl from interfaces/wallet.cpp to wallet/interfaces.cpp 2020-11-24 10:20:16 -05:00
Russell Yanofsky
2a26771d81 Move ChainImpl from interfaces/chain.cpp to node/interfaces.cpp
No changes to ChainImpl or any related classes (review with `git diff --color-moved=dimmed_zebra`)
2020-11-24 10:13:23 -05:00
Russell Yanofsky
12bd0fc9d7 Move NodeImpl from interfaces/node.cpp to node/interfaces.cpp 2020-11-24 10:13:23 -05:00
Niklas Gögge
e95aaefe25 build: Avoid secp256k1.h include from system 2020-11-23 23:33:53 +01:00
Wladimir J. van der Laan
634f6ec4eb contrib: Parse ELF directly for symbol and security checks
Instead of the ever-messier text parsing of the output of the readelf
tool (which is clearly meant for human consumption not to be machine
parseable), parse the ELF binaries directly.

Add a small dependency-less ELF parser specific to the checks.

This is slightly more secure, too, because it removes potential
ambiguity due to misparsing and changes in the output format of `elfread`. It
also allows for stricter and more specific ELF format checks in the future.

This removes the build-time dependency for `readelf`.

It passes the test-security-check for me locally, though I haven't
checked on all platforms.
2020-11-22 11:11:32 +01:00
Andrew Chow
a58b719cf7 Do not compile BDB things when USE_BDB is defined 2020-11-18 11:56:08 -05:00
Luke Dashjr
7b54d768e1 Make sqlite support optional (compile-time) 2020-10-20 13:44:43 +00:00
Samuel Dobson
8ed37f6c84
Merge #19077: wallet: Add sqlite as an alternative wallet database and use it for new descriptor wallets
c4a29d0a90 Update wallet_multiwallet.py for descriptor and sqlite wallets (Russell Yanofsky)
310b0fde04 Run dumpwallet for legacy wallets only in  wallet_backup.py (Andrew Chow)
6c6639ac9f Include sqlite3 in documentation (Andrew Chow)
f023b7cac0 wallet: Enforce sqlite serialized threading mode (Andrew Chow)
6173269866 Set and check the sqlite user version (Andrew Chow)
9d3d2d263c Use network magic as sqlite wallet application ID (Andrew Chow)
9af5de3798 Use SQLite for descriptor wallets (Andrew Chow)
9b78f3ce8e walletutil: Wallets can also be sqlite (Andrew Chow)
ac38a87225 Determine wallet file type based on file magic (Andrew Chow)
6045f77003 Implement SQLiteDatabase::MakeBatch (Andrew Chow)
727e6b2a4e Implement SQLiteDatabase::Verify (Andrew Chow)
b4df8fdb19 Implement SQLiteDatabase::Rewrite (Andrew Chow)
010e365906 Implement SQLiteDatabase::TxnBegin, TxnCommit, and TxnAbort (Andrew Chow)
ac5c1617e7 Implement SQLiteDatabase::Backup (Andrew Chow)
f6f9cd6a64 Implement SQLiteBatch::StartCursor, ReadAtCursor, and CloseCursor (Andrew Chow)
bf90e033f4 Implement SQLiteBatch::ReadKey, WriteKey, EraseKey, and HasKey (Andrew Chow)
7aa45620e2 Add SetupSQLStatements (Andrew Chow)
6636a2608a Implement SQLiteBatch::Close (Andrew Chow)
93825352a3 Implement SQLiteDatabase::Close (Andrew Chow)
a0de83372b Implement SQLiteDatabase::Open (Andrew Chow)
3bfa0fe125 Initialize and Shutdown sqlite3 globals (Andrew Chow)
5a488b3d77 Constructors, destructors, and relevant private fields for SQLiteDatabase/Batch (Andrew Chow)
ca8b7e04ab Implement SQLiteDatabaseVersion (Andrew Chow)
7577b6e1c8 Add SQLiteDatabase and SQLiteBatch dummy classes (Andrew Chow)
e87df82580 Add sqlite to travis and depends (Andrew Chow)
54729f3f4e Add libsqlite3 (Andrew Chow)

Pull request description:

  This PR adds a new class `SQLiteDatabase` which is a subclass of `WalletDatabase`. This provides access to a SQLite database that is used to store the wallet records. To keep compatibility with BDB and to complexity of the change down, we don't make use of many SQLite's features. We use it strictly as a key-value store. We create a table `main` which has two columns, `key` and `value` both with the type `blob`.

  For new descriptor wallets, we will create a `SQLiteDatabase` instead of a `BerkeleyDatabase`. There is no requirement that all SQLite wallets are descriptor wallets, nor is there a requirement that all descriptor wallets be SQLite wallets. This allows for existing descriptor wallets to work as well as keeping open the option to migrate existing wallets to SQLite.

  We keep the name `wallet.dat` for SQLite wallets. We are able to determine which database type to use by searching for specific magic bytes in the `wallet.dat` file. SQLite begins it's files with a null terminated string `SQLite format 3`. BDB has `0x00053162` at byte 12 (note that the byte order of this integer depends on the system endianness). So when we see that there is a `wallet.dat` file that we want to open, we check for the magic bytes to determine which database system to use.

  I decided to keep the `wallet.dat` naming to keep things like backup script to continue to function as they won't need to be modified to look for a different file name. It also simplifies a couple of things in the implementation and the tests as `wallet.dat` is something that is specifically being looked for. If we don't want this behavior, then I do have another branch which creates `wallet.sqlite` files instead, but I find that this direction is easier.

ACKs for top commit:
  Sjors:
    re-utACK c4a29d0a90
  promag:
    Tested ACK c4a29d0a90.
  fjahr:
    reACK c4a29d0a90
  S3RK:
    Re-review ACK c4a29d0a90
  meshcollider:
    re-utACK c4a29d0a90
  hebasto:
    re-ACK c4a29d0a90, only rebased since my [previous](https://github.com/bitcoin/bitcoin/pull/19077#pullrequestreview-507743699) review, verified with `git range-diff master d18892dcc c4a29d0a9`.
  ryanofsky:
    Code review ACK c4a29d0a90. I am honestly confused about reasons for locking into `wallet.dat` again when it's so easy now to use a clean format. I assume I'm just very dense, or there's some unstated reason, because the only thing that's been brought up are unrealistic compatibility scenarios (all require actively creating a wallet with non-default descriptor+sqlite option, then trying to using the descriptor+sqlite wallets with old software or scripts and ignoring the results) that we didn't pay attention to with previous PRs like #11687, which did not require any active interfaction.
  jonatack:
    ACK c4a29d0a90, debug builds and test runs after rebase to latest master @ c2c4dbaebd, some manual testing creating, using, unloading and reloading a few different new sqlite descriptor wallets over several node restarts/shutdowns.

Tree-SHA512: 19145732e5001484947352d3175a660b5102bc6e833f227a55bd41b9b2f4d92737bbed7cead64b75b509decf9e1408cd81c185ab1fb4b90561aee427c4f9751c
2020-10-15 20:12:29 +13:00
Andrew Chow
7577b6e1c8 Add SQLiteDatabase and SQLiteBatch dummy classes 2020-10-14 11:27:37 -04:00
Andrew Chow
54729f3f4e Add libsqlite3 2020-10-14 11:18:12 -04:00
Pieter Wuille
86f50ed10f Delete limitedmap as it is unused now 2020-10-12 12:14:53 -07:00
Pieter Wuille
da3b8fde03 Add txrequest module
This adds a new module (unused for now) which defines TxRequestTracker, a data
structure that maintains all information about transaction requests, and coordinates
requests.
2020-10-12 11:01:16 -07:00
Wladimir J. van der Laan
8c5f68118c
Merge #18267: BIP-325: Signet [consensus]
8258c4c007 test: some sanity checks for consensus logic (Anthony Towns)
e47ad375bf test: basic signet tests (Karl-Johan Alm)
4c189abdc4 test: add small signet fuzzer (practicalswift)
ec9b25d046 test: signet network selection tests (Karl-Johan Alm)
3efe298dcc signet: hard-coded parameters for Signet Global Network VI (2020-09-07) (Karl-Johan Alm)
c7898bca4e qt: update QT to support signet network (Karl-Johan Alm)
a8de47a1c9 consensus: add signet validation (Karl-Johan Alm)
e8990f1214 add signet chain and accompanying parameters (Karl-Johan Alm)
404682b7cd add signet basic support (signet.cpp) (Karl-Johan Alm)
a2147d7dad validation: move GetWitnessCommitmentIndex to consensus/validation (Karl-Johan Alm)

Pull request description:

  This PR is a part of BIP-325 (https://github.com/bitcoin/bips/blob/master/bip-0325.mediawiki), and is a sub-PR of #16411.

  * Signet consensus (this)
  * Signet RPC tools (pending)
  * Signet utility scripts (contrib/signet) (pending)

ACKs for top commit:
  jonatack:
    re-ACK 8258c4c007 per `git diff dbeea65 8258c4c`, only change since last review is updated `-signet*` config option naming.
  fjahr:
    re-ACK 8258c4c
  laanwj:
    ACK 8258c4c007
  MarcoFalke:
    Approach ACK 8258c4c007 🌵

Tree-SHA512: 5d158add96755910837feafa8214e13695b769a6aec3a2da753cf672618bef377fac43b0f4b772a87b25dd9f0c1c9b29f2789785d7a7d47a155cdcf48f7c975d
2020-09-21 22:33:00 +02:00
fanquake
831b0ecea9
Merge #13686: ZMQ: Small cleanups in the ZMQ code
6fe2ef2acb scripted-diff: Rename SendMessage to SendZmqMessage. (Daniel Kraft)
a3ffb6ebeb Replace zmqconfig.h by a simple zmqutil. (Daniel Kraft)
7f2ad1b9ac Use std::unique_ptr for CZMQNotifierFactory. (Daniel Kraft)
b93b9d5456 Simplify and fix notifier removal on error. (Daniel Kraft)
e15b1cfc31 Various cleanups in zmqnotificationinterface. (Daniel Kraft)

Pull request description:

  This contains various small code cleanups that make the ZMQ code easier to read and maintain (at least in my opinion).  The only functional change is that a potential memory leak is fixed that would have occured when a notifier is removed from the `notifiers` list after its callback function returned `false` (which is likely not relevant in practice but still a bug).

ACKs for top commit:
  instagibbs:
    utACK 6fe2ef2acb
  hebasto:
    re-ACK 6fe2ef2acb, only the latest commit got a scripted-diff since my [previous](https://github.com/bitcoin/bitcoin/pull/13686#pullrequestreview-487649808) review.

Tree-SHA512: 8206f8713bf3698d7cd4cb235f6657dc1c4dd920f50a8c5f371a559dd17ce5ab6d94d6281165eef860a22fc844a6bb25489ada12c83ebc780efd7ccdc0860f70
2020-09-19 17:13:28 +08:00
fanquake
15c27c4441
build: split PTHREAD_* flags out of AM_LDFLAGS
Note that with this change we are no-longer including PTHREAD_* flags
when building libbitcoinconsensus.

Also note that we are including PTHREAD_LIBS in AM_PTHREAD_FLAGS
2020-09-14 16:35:09 +08:00
Karl-Johan Alm
404682b7cd
add signet basic support (signet.cpp)
Co-authored-by: Anthony Towns <aj@erisian.com.au>
2020-09-10 10:47:40 +09:00
Daniel Kraft
a3ffb6ebeb Replace zmqconfig.h by a simple zmqutil.
zmqconfig.h is currently not really needed anywhere, except that
it declares zmqError (which is then defined in
zmqnotificationinterface.cpp).  Note in particular that there is
no need to conditionally include zmq.h only if ZMQ is enabled, because
the place in the core code where the ZMQ library itself is included
(init.cpp) is conditional already on that.

This commit removes zmqconfig.h and replaces it by a much simpler
zmqutil.h library for zmqError.  The definition of the function is
moved to the matching (newly created) zmqutil.cpp.
2020-09-07 10:56:22 +02:00
Pieter Wuille
2ac8bf9583 Implement keccak-f[1600] and SHA3-256 2020-09-06 18:35:18 -07:00
Marcin Jachymiak
7668db3b08 Move only: Move CDiskTxPos to its own file 2020-08-15 18:23:55 -04:00
Hennadii Stepanov
6cb8771173
build: Add missed gcov files to 'make clean' 2020-08-14 14:38:28 +03:00
MarcoFalke
fa4695da4c
build: Sort Makefile.am after renaming file 2020-06-27 11:49:35 -04:00
MarcoFalke
cccc2784a3
scripted-diff: Move ui_interface to the node lib
-BEGIN VERIFY SCRIPT-

 # Move files
 git mv src/ui_interface.h                                          src/node/ui_interface.h
 git mv src/ui_interface.cpp                                        src/node/ui_interface.cpp
 sed -i -e 's/BITCOIN_UI_INTERFACE_H/BITCOIN_NODE_UI_INTERFACE_H/g' src/node/ui_interface.h

 # Adjust includes and makefile
 sed -i -e 's|ui_interface|node/ui_interface|g' $(git grep -l ui_interface)

 # Sort includes
 git diff -U0 | clang-format-diff -p1 -i -v

-END VERIFY SCRIPT-
2020-06-27 11:49:28 -04:00
MarcoFalke
fac96e6450
wallet: Do not include server symbols
ui_interface is in libbitcoin_server and can not be included in the
wallet because the wallet does not link with server symbols.
2020-06-27 11:39:09 -04:00
Samuel Dobson
47a30ef0c6
Merge #19133: rpc, cli, test: add bitcoin-cli -generate command
22cb303cf0 rpc: add missing space in JSON parsing error message, update test (Jon Atack)
bf53ebef06 test: add multiwallet tests for bitcoin-cli -generate (Jon Atack)
4b859cfff9 cli: add multiwallet capability to GetNewAddress and -generate (Jon Atack)
18f93545a1 test: add tests for bitcoin-cli -generate (Jon Atack)
4818124137 cli: create bitcoin-cli -generate command (Jon Atack)
ff41a36900 cli: extract ParseResult() and ParseError() (Jon Atack)
f4185b26d9 cli: create GenerateToAddressRequestHandler class (Harris)
f7c65a3350 cli: create GetNewAddress() (Jon Atack)
9be7fd35c5 rpc: make generatetoaddress locals const (Jon Atack)
cb00510dba rpc: create rpc/mining.h, hoist default max tries values to constant (Jon Atack)

Pull request description:

  This PR continues and completes the work begun in #17700 working on issue #16000 to create a client-side version of RPC `generate`.

  Basically, `bitcoin-cli -generate` wraps calling `generatenewaddress` followed by `generatetoaddress [nblocks] [maxtries]` and prints the following:

  ```
  $ bitcoin-cli -generate
  {
    "address": "bcrt1qn4aszr2y2xvpa70y675a76wsu70wlkwvdyyln6"
    "blocks": [
      "01d2ebcddf663da90b28da7f6805115e2ba7818f16fe747258836646a43a0bb5",
    ]
  }

  $ bitcoin-cli -rpcwallet=wallet-name -generate 3 100
  {
    "address": "bcrt1q4cunfw0gnsj7g7e6mk0v0uuvvau9mwr09dj45l",
    "blocks": [
      "7a6650ca5e0c614992ee64fb148a7e5e022af842e4b6003f81abd8baf1e75136",
      "01d2ebcddf663da90b28da7f6805115e2ba7818f16fe747258836646a43a0bb5",
      "3f8795ec40b1ad812b818c177680841be319a3f6753d4e32dc7dfb5bafe5d00e"
    ]
  }
  ```

  Help doc:
  ```
  $ bitcoin-cli -h | grep -A5 "\-generate"
    -generate
         Generate blocks immediately, equivalent to RPC generatenewaddress
         followed by RPC generatetoaddress. Optional positional arguments
         are number of blocks to generate (default: 1) and maximum
         iterations to try (default: 1000000), equivalent to RPC
         generatetoaddress nblocks and maxtries arguments. Example:
         bitcoin-cli -generate 4 1000
  ```

  Quite a bit of test coverage turned out to be needed to cover the change and the different cases (arguments, multiwallet mode) and error-handling.

  This PR also improves some things that working on these changes brought to light.

  Credit to Harris Brakmić for the initial work in #17700.

ACKs for top commit:
  adamjonas:
    utACK 22cb303cf0
  meshcollider:
    utACK 22cb303cf0

Tree-SHA512: 94f67f632fe093d076f614e0ecff09ce7342ac6e424579200d5211a6615260e438d857861767fb788950ec6da0b26ef56dc8268c430012a3b3d4822b24ca6fbf
2020-06-21 22:24:07 +12:00
Andrew Chow
61c16339da walletdb: Move BDB specific things into bdb.{cpp/h}
Leave wallet/db.{cpp/h} for generic WalletDatabase stuff. The BDB
specific stuff goes into bdb.{cpp/h}
2020-06-15 20:41:05 -04:00
Jon Atack
cb00510dba
rpc: create rpc/mining.h, hoist default max tries values to constant 2020-06-01 15:08:36 +02:00
Russell Yanofsky
4a7253ab6c Remove g_rpc_chain global
Replace with RPC request reference to new WalletContext struct similar to the
existing NodeContext struct and reference.

This PR is a followup to 25ad2c623a
https://github.com/bitcoin/bitcoin/pull/18740 removing the g_rpc_node global.

Some later PRs will follow this up and move more wallet globals to the
WalletContext struct.

Co-authored-by: João Barbosa <joao.paulo.barbosa@gmail.com>
2020-05-28 02:13:19 -04:00
Samuel Dobson
520e435b5e
Merge #18918: wallet: Move salvagewallet into wallettool
84ae0578b6 Add release notes about salvage changes (Andrew Chow)
ea337f2d03 Move RecoverKeysOnlyFilter into RecoverDataBaseFile (Andrew Chow)
9ea2d258b4 Move RecoverDatabaseFile and RecoverKeysOnlyFilter into salvage.{cpp/h} (Andrew Chow)
b426c7764d Make BerkeleyBatch::Recover and WalletBatch::RecoverKeysOnlyFilter standalone (Andrew Chow)
2741774214 Expose a version of ReadKeyValue and use it in RecoverKeysOnlyFilter (Andrew Chow)
ced95d0e43 Move BerkeleyEnvironment::Salvage into BerkeleyBatch::Recover (Andrew Chow)
07250b8dce walletdb: remove fAggressive from Salvage (Andrew Chow)
8ebcbc85c6 walletdb: don't automatically salvage when corruption is detected (Andrew Chow)
d321046f4b wallet: remove -salvagewallet (Andrew Chow)
cdd955e580 Add basic test for bitcoin-wallet salvage (Andrew Chow)
c87770915b wallettool: Add a salvage command (Andrew Chow)

Pull request description:

  Removes the `-salvagewallet` startup option and adds a `salvage` command to the `bitcoin-wallet` tool. As such, `-salvagewallet` is removed. Additionally, the automatic salvage that is done if the wallet file fails to load is removed.

  Lastly the salvage code entirely is moved out entirely into `bitcoin-wallet` from `walletdb.{cpp/h}` and `db.{cpp/h}`.

ACKs for top commit:
  jonatack:
    ACK 84ae0578b6 feedback taken, and compared to my previous review, the bitcoin-wallet salvage command now seems to run and it exits without raising. The new test passes at both 9454105 and 84ae057 so as a sanity check I'd agree there is room for improvement, if possible.
  MarcoFalke:
    re-ACK 84ae0578b6 🏉
  Empact:
    Code Review ACK 84ae0578b6
  ryanofsky:
    Code review ACK 84ae0578b6. Lot of small changes since previous review: added verify step before salvage, added basic test in new commit, removed unused scanstate variable and warnings parameter, tweaked various comments and strings, moved fsuccess variable declaration
  meshcollider:
    Concept / light code review ACK 84ae0578b6

Tree-SHA512: 05be116b56ecade1c58faca1728c8fe4b78f0a082dbc2544a3f7507dd155f1f4f39070bd1fe90053444384337bc48b97149df5c1010230d78f8ecc08e69d93af
2020-05-27 14:51:49 +12:00
Andrew Chow
9ea2d258b4 Move RecoverDatabaseFile and RecoverKeysOnlyFilter into salvage.{cpp/h} 2020-05-25 12:59:29 -04:00
MarcoFalke
25ad2c623a
Merge #18740: Remove g_rpc_node global
b3f7f375ef refactor: Remove g_rpc_node global (Russell Yanofsky)
ccb5059ee8 scripted-diff: Remove g_rpc_node references (Russell Yanofsky)
6fca33b2ed refactor: Pass NodeContext to RPC and REST methods through util::Ref (Russell Yanofsky)
691c817b34 Add util::Ref class as temporary alternative for c++17 std::any (Russell Yanofsky)

Pull request description:

  This PR removes the `g_rpc_node` global, to get same benefits we see removing other globals and make RPC code more testable, modular, and reusable.

  This uses a hybrid of the approaches suggested in #17548. Instead of using `std::any`, which isn't available in c++11, or `void*`, which isn't type safe, it uses a small new `util::Ref` helper class, which acts like a simplified `std::any` that only holds references, not values.

  Motivation for writing this was to provide an simpler alternative to #18647 by Harris Brakmić (brakmic) which avoids some shortcomings of that PR (https://github.com/bitcoin/bitcoin/pull/18647#issuecomment-617878826)

ACKs for top commit:
  MarcoFalke:
    re-ACK b3f7f375ef, only change is adding back const and more tests 🚾
  ajtowns:
    ACK b3f7f375ef

Tree-SHA512: 56292268a001bdbe34d641db1180c215351503966ff451e55cc96c9137f1d262225d7d7733de9c9da7ce7d7a4b34213a98c2476266b58c89dbbb0f3cb5aa5d70
2020-05-21 06:53:39 -04:00
fanquake
97b21b302a
Merge #18677: Multiprocess build support
e2bab2aa16 multiprocess: add multiprocess travis configuration (Russell Yanofsky)
603fd6a2e7 depends: add MULTIPROCESS depends option (Russell Yanofsky)
5d1377b52b build: multiprocess autotools changes (Russell Yanofsky)

Pull request description:

  This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10).

  ---

  This PR consists of build changes only. It adds an `--enable-multiprocess` autoconf option (off by default and marked experimental), that builds new `bitcoin-node` and `bitcoin-gui` binaries. These currently function the same as existing `bitcoind` and `bitcoin-qt` binaries, but are extended in #10102 with IPC features to execute node, wallet, and gui functions in separate processes.

  In addition to adding the `--enable-multiprocess` config flag, it also adds a depends package and autoconf rules to build with the [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) library, and it adds new travis configuration to exercise the build code and run functional tests with the new binaries.

  The changes in this PR were originally part of #10102 but were moved into #16367 to be able to develop and review the multiprocess build changes independently of the code changes. #16367 was briefly merged and then reverted in #18588. Only change since #16367 has been dropping the `native_boost.mk` depends package which was pointed out to be no longer necessary in https://github.com/bitcoin/bitcoin/pull/16367#issuecomment-596484337 and https://github.com/bitcoin/bitcoin/pull/18588#pullrequestreview-391765649

ACKs for top commit:
  practicalswift:
    ACK e2bab2aa16
  Sjors:
    tACK e2bab2aa16 on macOS 10.15.4
  hebasto:
    ACK e2bab2aa16, tested on Linux Mint 19.3 (x86_64):

Tree-SHA512: b5a76eab5abf63d9d8b6d628cbdff4cc1888eef15cafa0a5d56369e2f9d02595fed623f4b74b2cf2830c42c05a774f0943e700f9c768a82d9d348cad199e135c
2020-05-21 15:34:25 +08:00
Russell Yanofsky
691c817b34 Add util::Ref class as temporary alternative for c++17 std::any
This commit does not change behavior
2020-05-13 16:20:13 -04:00
Russell Yanofsky
5d1377b52b build: multiprocess autotools changes
autoconf and automake changes to support multiprocess gui/node/wallet execution.

This adds a new --enable-multiprocess flag, and build configuration code to
detect libraries needed for multiprocess support. The --enable-multiprocess
flag builds new bitcoin-node and bitcoin-gui executables, which are updated in
https://github.com/bitcoin/bitcoin/pull/10102 to communicate across processes.
But for now they are functionally equivalent to existing bitcoind and
bitcoin-qt executables.
2020-05-12 09:47:06 -04:00
fanquake
df6bde031b
test: remove glibc fdelt sanity check
As is, this sanity check doesn't seem to be testing fdelt_chk, because
passing a value of "0" to FD_SET wont cause the compiler to insert any
calls to fdelt_chk().

The documentation is a little misleading. If we actually triggered fdelt_chk
at runtime, bitcoind would abort. I think this check would be better replaced
(if possible) by additional checks in security-check.py.

The compiler may insert a call to fdelt_warn() (aliased with fdelt_chk
in glibc) at compile time if it can determine that an invalid value is
being passed to FD_SET.

These checks are essentially; value < 0 or value >= FD_SETSIZE along
with a check for wether the value is a compile time constant.

If the compiler can determine an invalid value is being passed, a call
to fdelt_warn will be inserted. Passing 0 should never cause a call to
be inserted.

You can check this after compiling:
```bash
objdump -dC bitcoind | grep sanity_fdelt
...
0000000000399d20 <sanity_test_fdelt()>:
  399d20:	48 81 ec 98 00 00 00 	sub    $0x98,%rsp
  399d27:	b9 10 00 00 00       	mov    $0x10,%ecx
  399d2c:	64 48 8b 04 25 28 00 	mov    %fs:0x28,%rax
  399d33:	00 00
  399d35:	48 89 84 24 88 00 00 	mov    %rax,0x88(%rsp)
  399d3c:	00
  399d3d:	31 c0                	xor    %eax,%eax
  399d3f:	48 89 e7             	mov    %rsp,%rdi
  399d42:	fc                   	cld
  399d43:	f3 48 ab             	rep stos %rax,%es:(%rdi)
  399d46:	48 8b 84 24 88 00 00 	mov    0x88(%rsp),%rax
  399d4d:	00
  399d4e:	64 48 33 04 25 28 00 	xor    %fs:0x28,%rax
  399d55:	00 00
  399d57:	75 0d                	jne    399d66 <sanity_test_fdelt()+0x46>
  399d59:	b8 01 00 00 00       	mov    $0x1,%eax
  399d5e:	48 81 c4 98 00 00 00 	add    $0x98,%rsp
  399d65:	c3                   	retq
  399d66:	e8 85 df c8 ff       	callq  27cf0 <__stack_chk_fail@plt>
  399d6b:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)

```

To test, you could modify this test to pass -1 to FD_SET, and check
that a call to fdelt_warn() is inserted, and that running bitcoind
fails. i.e:

```bash
0000000000399d20 <sanity_test_fdelt()>:
  399d20:	48 81 ec 98 00 00 00 	sub    $0x98,%rsp
  399d27:	b9 10 00 00 00       	mov    $0x10,%ecx
  399d2c:	64 48 8b 04 25 28 00 	mov    %fs:0x28,%rax
  399d33:	00 00
  399d35:	48 89 84 24 88 00 00 	mov    %rax,0x88(%rsp)
  399d3c:	00
  399d3d:	31 c0                	xor    %eax,%eax
  399d3f:	48 89 e7             	mov    %rsp,%rdi
  399d42:	fc                   	cld
  399d43:	f3 48 ab             	rep stos %rax,%es:(%rdi)
  399d46:	48 c7 c7 ff ff ff ff 	mov    $0xffffffffffffffff,%rdi
  399d4d:	e8 3e ff ff ff       	callq  399c90 <__fdelt_warn>
  399d52:	0f b6 04 24          	movzbl (%rsp),%eax
  399d56:	83 e0 01             	and    $0x1,%eax
  399d59:	48 8b 94 24 88 00 00 	mov    0x88(%rsp),%rdx
  399d60:	00
  399d61:	64 48 33 14 25 28 00 	xor    %fs:0x28,%rdx
  399d68:	00 00
  399d6a:	75 08                	jne    399d74 <sanity_test_fdelt()+0x54>
  399d6c:	48 81 c4 98 00 00 00 	add    $0x98,%rsp
  399d73:	c3                   	retq
  399d74:	e8 77 df c8 ff       	callq  27cf0 <__stack_chk_fail@plt>
  399d79:	0f 1f 80 00 00 00 00 	nopl   0x0(%rax)

```

```bash
./src/bitcoind
*** buffer overflow detected ***: src/bitcoind terminated
Aborted
```
2020-05-07 15:45:09 +08:00
MarcoFalke
c4c3f110eb
Merge #18190: tests: Add fuzzing harness for Golomb-Rice coding (GolombRiceEncode/GolombRiceDecode)
69749fbe6a tests: Add fuzzing harness for Golomb-Rice coding (GolombRiceEncode/GolombRiceDecode) (practicalswift)

Pull request description:

  Add fuzzing harness for Golomb-Rice coding (`GolombRiceEncode`/`GolombRiceDecode`).

  Test this PR using:

  ```
  $ make distclean
  $ ./autogen.sh
  $ CC=clang CXX=clang++ ./configure --enable-fuzz \
        --with-sanitizers=address,fuzzer,undefined
  $ make
  $ src/test/fuzz/golomb_rice
  …
  ```

Top commit has no ACKs.

Tree-SHA512: 1b26512301b8c22ab3b804d9b9e4baf933f26f8c05e462d583863badcec7e694548a34849a0d7c4ff7d58b19f6338b51819976ecf642bc4659b04ef71182d748
2020-04-20 15:32:41 -04:00
practicalswift
69749fbe6a tests: Add fuzzing harness for Golomb-Rice coding (GolombRiceEncode/GolombRiceDecode) 2020-04-20 14:57:48 +00:00
MarcoFalke
f29bd546ec Revert "Merge #16367: Multiprocess build support"
This reverts the changes made in merge commit
1b30761360:

This reverts commit b919efadff.
This reverts commit d54f64c6c7.
This reverts commit 787f40668d.
This reverts commit d630646662.
This reverts commit e6e44eedd5.
2020-04-10 19:38:21 -04:00
MarcoFalke
1b30761360
Merge #16367: Multiprocess build support
b919efadff depends: Use default macos clang compiler (Russell Yanofsky)
d54f64c6c7 Add multiprocess travis configuration (Russell Yanofsky)
787f40668d Set LD_LIBRARY_PATH consistently in travis tests (Russell Yanofsky)
d630646662 libmultiprocess depends build (Russell Yanofsky)
e6e44eedd5 Multiprocess build changes (Russell Yanofsky)

Pull request description:

  This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10).

  This splits autotools, depends build, and travis changes out of #10102, so code changes and build system changes can be reviewed separately.

ACKs for top commit:
  hebasto:
    re-ACK b919efadff, rebased only since my [previous](https://github.com/bitcoin/bitcoin/pull/16367#issuecomment-605514556) review.

Tree-SHA512: ebc5e403cc99a0d9629ed7fe1595e01d57e6d1255cbf03968a3196ff6f528f734c78060fdc065724ee1f923bcc5aa2b29470fcb36a7f15957eb57c76d58178a4
2020-04-10 12:55:12 -04:00
MarcoFalke
4eb1eeb02c
Merge #18504: build: Drop bitcoin-tx and bitcoin-wallet dependencies on libevent
01a3392b1b Drop bitcoin-wallet dependency on libevent (Russell Yanofsky)
0660119ac3 Drop unintended bitcoin-tx dependency on libevent (Russell Yanofsky)

Pull request description:

  This fixes compile errors trying to build bitcoin-tx and bitcoin-wallet without libevent, which were reported by Luke Dashjr in https://github.com/bitcoin/bitcoin/issues/18465

  The fix avoiding `bitcoin-tx` dependency on libevent just adds a conditional build rule. This is implemented in the first commit (more details in commit description).

  The fix avoiding `bitcoin-wallet` dependency on libevent requires minor code changes, because `bitcoin-wallet` (unlike `bitcoin-tx`) links against code that calls `urlDecode` / `evhttp_uridecode`. This fix is implemented in the second commit (again details in the commit description).

ACKs for top commit:
  jonasschnelli:
    utACK 01a3392b1b.

Tree-SHA512: d2245e912ab494cccceeb427a1eca8e55b01a0006ff93eebcfb5461ae7cecd1083ac2de443d9db036b18bdc6f0fb615546caaa20c585046f66d234937f74870a
2020-04-10 12:52:37 -04:00
Russell Yanofsky
e6e44eedd5 Multiprocess build changes
autotools and automake changes to support multiprocess execution.

This adds a new --enable-multiprocess flag, and build configuration code to
detect libraries needed for multiprocess support. The --enable-multiprocess
flag builds new bitcoin-node and bitcoin-gui executables, which are updated in
https://github.com/bitcoin/bitcoin/pull/10102 to communicate across processes.
But for now they are functionally equivalent to existing bitcoind and
bitcoin-qt executables.
2020-04-05 21:48:21 -04:00
Harris
691e2a7af7
build: create test_fuzz library from src/test/fuzz/fuzz.cpp 2020-04-05 01:01:13 +02:00
Russell Yanofsky
0660119ac3 Drop unintended bitcoin-tx dependency on libevent
Don't include util/url.cpp to libbitcoin_util.a when libevent isn't available.
This fixes a compile error trying to build bitcoin-tx without libevent reported
by Luke Dashjr in https://github.com/bitcoin/bitcoin/issues/18465

Fixes #18465
2020-04-02 08:28:11 -04:00
fanquake
1a0993ae35
scripts: add PE dylib checking to symbol-check.py 2020-03-22 10:47:38 +08:00
Andrew Chow
3d70dd99f9 Move FillPSBT to be a member of CWallet 2020-03-09 11:16:17 -04:00
Anthony Towns
cea19f6859 Drop unused reverselock.h 2020-03-06 23:14:10 +10:00
Jeffrey Czyz
0aed17ef28 Refactor FormatStateMessage into ValidationState 2020-02-27 17:59:07 -08:00