Nodes are restarted and reconnected as part of the test. Afterwards
`sync_blocks` is called immediately on the nodes. `sync_blocks`
first checks that all the included nodes have at least one
connection. Since adding a connection is usually happening in a
thread, sometimes nodes could run into this check before the
connection was fully established so that it would fail the entire
test.
This fix uses the `connect_nodes` helper to make the connection the
nodes. `connect_nodes` has a wait for the connection built into it.
In order to deserialize received or read messages via lookup in
MESSAGEMAP (e.g.: `t = MESSAGEMAP[msgtype]()`), the messages must have a
default constructor, i.e. there needs to be the possibility to
initialize them with zero arguments.
faa5a7a573 test: Check msg type in msg capture is followed by zeros (MacroFake)
Pull request description:
Checking that they are not printable is an odd (and wrong) way to check that all chars are zero.
ACKs for top commit:
theStack:
Code-review ACK faa5a7a573
Tree-SHA512: 63e001bd25298dcf47606f8ab11ddfb704ca963304149b0f6e188eb7dcf45c41f92d39f26bda32bceb03384720c9bdddb2673dba513cd9242dc9663d498b3f29
This change improves the usability of the `dumptxoutset` RPC in two ways,
in the case that an invalid path is passed:
1. return from the RPC immediately, rather then when the file is first
tried to be written (which is _after_ calculating the UTXO set hash)
2. return a proper return code and error message instead of the cryptic
"CAutoFile::operator<<: file handle is nullptr: unspecified
iostream_category error" (-1)
3258bad996 changes color of skipped functional tests (Jacob P. Fickes)
Pull request description:
changes the color of skipped functional tests (currently grey and can be hard to read/invisible on dark backgrounds) to yellow.
resolves#24791
ACKs for top commit:
theStack:
Tested ACK 3258bad996
jarolrod:
Tested ACK 3258bad996
Tree-SHA512: 3fe5ae0d3b4902b2b6bda6e89ab780feb8bf4b7cb1ce7e8467057b94a1e0a26ddeaf3cac0bc19b06ef10d8bccaac9c495029d42740fbedab8fb0d5fdd7d02eaf
dba1231672 test: previous releases: add v23.0 (Sjors Provoost)
Pull request description:
Follows the same pattern as d8b705f1ca (v22.0) and 8a57a06a50 (v0.21.0).
Starting from v23.0 there is a separate macOS release for x86_64 and aarch64.
ACKs for top commit:
prusnak:
Approach ACK dba1231672
Tree-SHA512: 249aeddd5e80e163578581e5c8e9b6579f3694abc3d1fb68dddb7b42d75021ad85266688ec4a365a6631d82a65a19873aff7ba61c0ea59d21f8adbe4b772dc16
We add an RPC to fetch the mempool transactions spending given outpoints.
Without this RPC, application developers would need to first call
`getrawmempool` which returns a long list of `txid`, then fetch each of
these txs individually to check whether they spend the given outpoint(s).
This RPC can later be enriched to also find confirmed transactions instead
of being restricted to mempool transactions.
027aab663a test, contrib, refactor: use `with` when opening a file (brunoerg)
Pull request description:
When manipulating a file in Python without using `with()`, you have to close the file manually, so this PR does it in `get_block_hashes` (`contrib/linearize/linearize-data.py`).
Edit: this PR does it for all occurances that previously weren't using `with`.
ACKs for top commit:
laanwj:
Code review ACK 027aab663a
Tree-SHA512: 879400968e0013e8678ec16f1fe5d0963a73c1e0d442ca34802d885214f0783d2e9a9b500fc6be7c3b93560a367b6a3d685eee24d2f9ce53fddf064ea6feecf8
d1bfe5ebdb test: add coverage for invalid requests for `blockfilterheaders` (brunoerg)
Pull request description:
This PR adds test coverage for invalid requests (`Invalid hash` and `Unknown filtertype`) for `/blockfilterheaders` in REST functional test.
ACKs for top commit:
jonatack:
ACK d1bfe5ebdb
vincenzopalazzo:
ACK d1bfe5ebdb
Tree-SHA512: 9ab7efe7131296577c60642f95921799cf1dbae9c2aaea6752d2ac9f35a1bcc72b9d742a146c314f82fe1848190a80c88836ab78fc28773ed12e97fa327828e7
a498acce45 test: MiniWallet: skip mempool check if `mempool_valid=False` (Sebastian Falbesoner)
01552e8f67 test: MiniWallet: always rehash after signing (P2PK mode) (Sebastian Falbesoner)
Pull request description:
MiniWallet's core method for creating txs (`create_self_transfer`) right now always executes the `testmempoolaccept` RPC to check for mempool validity or invalidity. In some test cases where we use MiniWallet to create a huge number of transactions this can lead to performance issues, in particular feature_fee_estimation.py where the execution time after MiniWallet usage (PR #24817) doubled, see https://github.com/bitcoin/bitcoin/issues/24828#issuecomment-1100058100, https://github.com/bitcoin/bitcoin/issues/24828#issuecomment-1100301980. This PR mitigates this by skipping the mempool check if the parameter `mempool_valid` is set to `False`.
As a preparatory commit, the test feature_csv_activation.py has to be adapted w.r.t. to rehashing of transactions, as we now hash all transactions immediately in `create_self_transfer` in order to get the txid (before we relied on the result of `testmempoolaccept`).
On my machine, this decreases the execution time quite noticably:
master branch:
```
$ time ./test/functional/feature_fee_estimation.py
real 3m20.771s
user 2m52.360s
sys 0m39.340s
```
PR branch:
```
$ time ./test/functional/feature_fee_estimation.py
real 2m1.386s
user 1m42.510s
sys 0m22.980s
```
Partly fixes#24828 (hopefully).
ACKs for top commit:
danielabrozzoni:
tACK a498acce45
Tree-SHA512: f20c358ba42b2ded86175f46ff3ff9eaefb84175cbd1c2624f44904c8d8888e67ce64d6dcbb26aabbf07906e6f5bdea40353eba9ae668618cadcfc517ef7201b
5f213213cb tests: add tests for cross-chain wallet use prevention (Seibart Nedor)
968765973b wallet: ensure wallet files are not reused across chains (Seibart Nedor)
Pull request description:
This implements a proposal in #12805 and is a rebase of #14533.
This seems to be a working approach, but I'm not sure why the `p2p_segwit.py` functional test needed a change, so I'll look into it more.
ACKs for top commit:
achow101:
ACK 5f213213cb
dongcarl:
Code Review ACK 5f213213cb
[deleted]:
tACK 5f213213cb
Tree-SHA512: 2c934300f113e772fc31c16ef5588526300bbc36e4dcef7d77bd0760c5c8f0ec77f766b1bed5503eb0157fa26dc900ed54d2ad1b41863c1f736ce5c1f3b67bec
fafd67479a test: Remove previous release check (MarcoFalke)
Pull request description:
Now that the commit (7c08d81e11) which changes taproot to be enforced for all blocks is sufficiently buried by other commits, and thus less likely to be reverted, it seems a good time to remove no longer needed test code.
The `feature_taproot` functional test is cleaned up to no longer run against a previous release. Since previous releases are static and impossible to change, it is sufficient to run the test once against the release. Now that this is done, the check can be removed without decreasing test coverage.
ACKs for top commit:
laanwj:
Concept and code review ACK fafd67479a
vincenzopalazzo:
ACK fafd67479a
Tree-SHA512: fcb1a93f3bf9deb5f5c7327a7cd23be10ba09c9f4cbfa73ee2764a93c6ce7d6fa98ca32f2cf4023c20ab624aee601beec949fd02a57a3a658fdbd4be1a9ff338
786b3a7c44 tests: Do not always create a descriptor wallet in wallet_createwallet (Andrew Chow)
Pull request description:
The createwallet test for some invalid parameters incorrectly always creates a descriptor wallet. This is unnecessary and also breaks the test when bdb is not compiled in.
Fixes#25007
ACKs for top commit:
jacobpfickes:
ACK 786b3a7c44
Tree-SHA512: 97b0953a08adf83d5ea84cac2651253d790b43d606a2f746dd45d3ccd1fb576bab63e3835e3de592715ef8a5cb133e6f19a3ab810fedf4684072143c3cb578d4
The createwallet teswt for some invalid parameters incorrectly always
creates a descriptor wallet. This is unnecessary and also breaks the
test when bdb is not compiled in.
2ff8f4dd81 Add tests for addr destination rotation (Gleb Naumenko)
77ccb7fce1 Use std::chrono for salting when randomizing ADDR destination (Gleb Naumenko)
Pull request description:
We currently assign a destination peer for relaying particular addresses of nodes every 24 hours, and then rotate. This is done for rate-limiting (ultimately for privacy leak reduction I think?).
Before this change, 24 hours was defined as uint. I replaced it with std::chrono, which is mockable and type-safe.
Also added couple tests for this behavior.
ACKs for top commit:
jonatack:
ACK 2ff8f4dd81
Tree-SHA512: 16f703ef3ffee13ce3afa82ca7b4baa27308af18cd2eececdce5565badfb68656a2ad9c4594b73772e4bfa99b3fb15f8e4089c1cb4be98c0bae6730a9d2f8a25
fa1f6df21e test: Fix intermittent test failure in wallet_listreceivedby.py (MarcoFalke)
Pull request description:
* Remove not needed "Generate block to get out of IBD"
* Sync blocks where possible to avoid incoming blocks on the p2p `msghand` thread while blocks are mined in the RPC thread. See https://github.com/bitcoin/bitcoin/issues/24730 for discussion.
Top commit has no ACKs.
Tree-SHA512: eca0242e7793886535555fec62f7acd4c0955bf26fab78725b4fe53f84f0b118cb12c9ee35627503fc68b83c3a228842e861fab89aab1226e08e18596357aaae
71c3f0356c move-only: Rename index + pruning functional test (Fabian Jahr)
de08932efa test: Update test for indices on pruned nodes (Fabian Jahr)
825d19839b Index: Allow coinstatsindex with pruning enabled (Fabian Jahr)
f08c9fb0c6 Index: Use prune locks for blockfilterindex (Fabian Jahr)
2561823531 blockstorage: Add prune locks to BlockManager (Fabian Jahr)
231fc7b035 refactor: Introduce GetFirstStoredBlock helper function (Fabian Jahr)
Pull request description:
# Motivation
The main motivation of this change and only behavior change noticeable by user is to allow running `coinstatsindex` on pruned nodes as has been requested [here for example](https://twitter.com/benthecarman/status/1388170854140452870?s=20).
# Background
`coinstatsindex` on pruned nodes can be enabled in a much simpler than it is done here but it comes with downside. The ability to run `blockfilterindex`on pruned nodes was added in #15946 but it also added the `blockfilterindex` as a dependency to `validation` and it introduced two new circular dependencies. Enabling `coinstatsindex` on pruned nodes in a similar way would add it as a dependency as well and introduce another circular dependency.
Instead, this PR introduces a `m_prune_blockers` map to `BlockManager` as a flexible approach to block pruning. Entities like `blockfilterindex`, for example, can add a key and a height to block pruning over that height. These entities need to update that value to allow more pruning when they are ready.
# Alternative approach
Upon completing the first draft of this PR I found #19463 as an alternative that follows the same but follows a very different approach. I am listing the main differences here as I see them:
- Usage of globals
- Blocks pruning with a start and a stop height
- Can persist blockers across restarts
- Blockers can be set/unset via RPCs
Personally, I don't think any of these are necessary to be added here but if the general approach or specific features are more appealing to reviewers I am happy to change to a solution based on that PR or port over specific parts of it here.
ACKs for top commit:
mzumsande:
Code review ACK 71c3f0356c
ryanofsky:
Code review ACK 71c3f0356c. Changes since last review: just tweaking comments and asserts, and rebasing
w0xlt:
tACK 71c3f0356c on signet.
Tree-SHA512: de7efda08b44aa31013fbebc47a02cd2de32db170b570f9643e1f013fee0e8e7ca3068952d1acc6e5e74a70910735c5f263437981ad73df841ad945b52d36b71
ab5af9ca72 test: Add test for coinselection tracepoints (Andrew Chow)
ca02b68e8a doc: document coin selection tracepoints (Andrew Chow)
8e3f39e4fa wallet: Add some tracepoints for coin selection (Andrew Chow)
15b58383d0 wallet: compute waste for SelectionResults of preset inputs (Andrew Chow)
912f1ed181 wallet: track which coin selection algorithm produced a SelectionResult (Andrew Chow)
Pull request description:
Tracepoints can be useful for coin selection as they would allow us to observe what is being selected, selection parameters, and calculation results. So this PR adds 4 new tracepoints:
1. After `SelectCoins` returns in order to observe the `SelectionResult`
2. After the first `CreateTransactionInternal` to observe the created transaction
3. Prior to the second `CreateTransactionInternal` to notify that the optimistic avoid partial spends selection is occurring
4. After the second `CreateTransactionInternal` to observe the created transaction and inform which solution is being used.
This PR also adds an algorithm enum to `SelectionResult` so that the first tracepoint will be able to report which algorithm was used to produce that result.
The primary use case for these tracepoints is in running coin selection simulations. The script I use to run these simulations use these tracepoints in order to gather data on the algorithm used and the calculated waste.
ACKs for top commit:
jb55:
crACK ab5af9ca72
josibake:
crACK ab5af9ca72
0xB10C:
ACK ab5af9ca72. Code reviewed, ran the `interface_usdt_coinselection.py` test, and tested with the above bpftrace script (updated `%d` -> `%ld` where necessary, ty achow101).
Tree-SHA512: a4bf7a910cdf464622f2f3b5d44c15b891f24852df6e7f8c5b177fe3d8aaa4a1164593a24c3960eb22b16544fa7140e5c745345367b9e291b78395084c0ac8ff
This drops the `No proxy server specified. Use -proxy=<ip> or -proxy=<ip:port>`
error when a empty `-proxy=` command line argument, `bitcoin.conf` value, or
`settings.json` value is specified, and just makes bitcoin connect and listen
normally in these cases.
The error was originally added in https://github.com/bitcoin/bitcoin/pull/20003
to prevent a bare `-proxy` command line argument with no assignment from
clearing proxy settings. But it was implemented in an overbroad way breaking
empty `-proxy=` assignments as well.
The motivation for this change is to prevent a GUI bug that happens with
https://github.com/bitcoin/bitcoin/pull/15936, reported in
https://github.com/bitcoin/bitcoin/pull/15936#pullrequestreview-937685759 by
vasild, that happens after a proxy setting is enabled and disabled in the GUI.
But this change also makes sense on its own to remove a potentially confusing
error message.
dac44fc06f init: disallow reindex-chainstate with optional indexes (Martin Zumsande)
62e14285f9 doc: Add note that -reindex will rebuild optional indexes (Martin Zumsande)
Pull request description:
When started together with `-reindex-chainstate`, currently coinstatsindex gets corrupted and the blockfilterindex flatfiles duplicated. See the OP of #24630 for more a more detailed explanation on why this happens.
This is an alternative to #24630 which does not wipe and rebuild the indexes but returns an `InitError` when they are activated, thus requiring the user to deactivate them temporarily until the `-reindex-chainstate` run is finished.
This also disallows `-reindex-chainstate` in combination with `-txindex`, which is not leading to corruption, but currently still rebuilds the index unnecessarily and unexpectedly.
As a long-term goal, it would be desirable to have the indexes tolerate `reindex-chainstate` by ignoring their `BlockConnected` notifications (there is discussion in #24630 about this) or possibly move `reindex-chainstate` option into a `bitcoin-chainstate` executable, which could also solve the problem. But these would be larger projects - until then, it might be better to disallow the interaction than having corrupted indexes.
The first commit adjusts the `-reindex` doc to mention that this option does rebuild all active indexes.
ACKs for top commit:
ryanofsky:
Code review ACK dac44fc06f. Just fixed IsArgSet call and edited error messages since last review
Tree-SHA512: c1abf7d350648ae227c3fd6c95d9a54c3bac9de70915275dea1c87cca6d9a76a056c0e306d95ef8cfe4df1f8525b418e0e7a4f52ded3be464041c0dc297f8930
9f5ab670e7 tests: Use descriptor that requires both legacy and segwit (Andrew Chow)
8a04a386f7 tests: Calculate input weight more accurately (Andrew Chow)
Pull request description:
The external input tests with specifying input weight would sometimes result in a test failure because it would add 2 to the calculated byte size in order to account for some of the variation in signature and script sizes. However 1 in 128 signatures are actually 1 byte smaller than we expect, so the difference between the actual signature size and our calculated size becomes 3 bytes which is outside of the tolerance of `assert_fee_amount` and would thus cause the test failure.
To resolve this, the 2 byte buffer is reduced to 1 byte, so in the above scenario, the difference is 2 bytes which is within the tolerance of `assert_fee_amount`. Additionally, instead of putting a fixed size that we assume is the correct size for the length of the compact size length prefix of data, we actually get the length of the compact size uint.
Lastly, the size calculation for a scriptWitness was simply incorrect and used fields that did not exist. This is fixed, and the test slightly modified so that it also produces a scriptWitness.
Fixes#24151
ACKs for top commit:
jonatack:
re-ACK 9f5ab670e7
glozow:
code review ACK 9f5ab670e7
Tree-SHA512: b7c7ffe8fb0c07bc9e72fbff1f9ef57ee01a57c56bf54b8873345c8b9572c3ce9402b24dc211910b478114a9e6420faef5a4bf8866f38c299971354e54ec4745
ee02c8bd9a util/check: Add CHECK_NONFATAL identity function, NONFATAL_UNREACHABLE AND UNREACHABLE macros (Aurèle Oulès)
Pull request description:
This PR replaces the macro `CHECK_NONFATAL` with an identity function.
I simplified the usage of `CHECK_NONFATAL` where applicable in `src/rpc`.
This function is useful in sanity checks for RPC and command-line interfaces.
Context: https://github.com/bitcoin/bitcoin/pull/24804#discussion_r846182474.
Also adds `UNREACHABLE_NONFATAL` macro.
ACKs for top commit:
jonatack:
ACK ee02c8bd9a
MarcoFalke:
ACK ee02c8bd9a🍨
Tree-SHA512: 3cba09223cd7b22e62fe5d0b46c4a024c1d9957d4268ba6d3fb07fcc0a5854fc0886bb3266184e6a7df5df91373b3e84edd6adf6999c4e934aeef8c043b01aa2
MiniWallet's core method for creating txs (`create_self_transfer`)
right now always executes the `testmempoolaccept` RPC to check for
mempool validity or invalidity. In some test cases where we use
MiniWallet to create a huge number of transactions this can lead
to performance issues (e.g. feature_fee_estimation.py where the
execution time after MiniWallet usage almost doubled). Providing
the possibility to skip the mempool checks is a mitigation for
this.
master branch:
$ time ./test/functional/feature_fee_estimation.py
real 3m20.771s
user 2m52.360s
sys 0m39.340s
PR branch:
$ time ./test/functional/feature_fee_estimation.py
real 2m1.386s
user 1m42.510s
sys 0m22.980s
Also explicitly rehash in the cases where we modify a tx after signing
in feature_csv_activation.py. Parts of this test relied on the fact that
rehashing of transactions is done in the course of calculating a block's
merkle root (`calc_merkle_root`), which only works if no hash was
calculated before due to a caching mechanism.
In the following commit the txid in MiniWallet is calculated via
`rehash()`, i.e. this doesn't work anymore and we always have to
explicitely have the right hash before we calculate the merkle root.
bef61496ab test: compare `/mempool/contents` response with `getrawmempool` RPC (brunoerg)
5bc5cbaf31 doc: add reference to `getrawmempool` RPC in `/mempool/contents` REST doc (brunoerg)
Pull request description:
This PR is similar to #24797, it compares `/mempool/contents` REST response with `getrawmempool` RPC (verbose=True) since they use the same `MempoolToJSON` function.
Also, adds a reference to `getrawmempool` RPC help to get details about the fields from `/mempool/contents`.
ACKs for top commit:
0xB10C:
ACK bef6149
Tree-SHA512: b7e9e9c765ee837986ba167b9234a9b95c9ef0a9ebcc2a03d50f6be6d3aba1480bd77c78111d95df1e4023cde6dfc64bf1e7908d9e5b6f96ca46b76611a4a9b4
fa2153b05b test: Remove unused taproot node from wallet_taproot.py (MarcoFalke)
Pull request description:
Now that the wallet considers taproot always active after commit 064c729a96, there is no need to test for it.
ACKs for top commit:
dunxen:
Code review ACK fa2153b
brunoerg:
crACK fa2153b05b
Tree-SHA512: 24e4a66e43d1391acb63fd0c0c52677b0eef7f618b87a5b1a75224a9be58c9c3f8bba2de3b7510f25a686865b027f7f535e653d40d519d0e00ace38f0c7aba0c
917a89a814 test: use MiniWallet for p2p_segwit.py (Sebastian Falbesoner)
Pull request description:
This PR enables one more of the non-wallet functional tests (p2p_segwit.py) to be run even with the Bitcoin Code wallet by using the MiniWallet instead, as proposed in https://github.com/bitcoin/bitcoin/issues/20078.
This change only affects the subtest `test_superfluous_witness`. Note that instead of creating a raw transaction first and then signing it, we go the other direction here: MiniWallet creates a transaction spending a segwit v1 output (i.e. including a witness), then we turn it into a raw transaction by dropping the witness. Therefore, the debug log asserts are swapped.
Top commit has no ACKs.
Tree-SHA512: 163a93a527f60100487f0aff49a9d7baf392ceb4417c54521157b2678685f5728dd751a9747c6cf51666aae78252dd3bc44130e659f7a1262ec1c86e30225622
6f29409ad1 test: Add a test that creates a wallet with invalid parameters (w0xlt)
0359d9b6a3 Change wallet validation order (w0xlt)
Pull request description:
In the current code, the database is created before the last validation, which checks that passphrase is set and private keys are disabled.
Therefore, if this validation fails, it will result in an empty database and the user will not be able to recreate a wallet with the same name and with the correct parameters.
Behavior on the master branch:
```
$ ./src/bitcoin-cli -regtest -named createwallet wallet_name="invalid_wallet_01" disable_private_keys=true passphrase="passphrase"
error code: -4
error message:
Passphrase provided but private keys are disabled. A passphrase is only used to encrypt private keys, so cannot be used for wallets with private keys disabled.
$ ./src/bitcoin-cli -regtest -named createwallet wallet_name="invalid_wallet_01"
error code: -4
error message:
Wallet file verification failed. Failed to create database path '/home/w/.bitcoin/regtest/wallets/invalid_wallet'. Database already exists.
```
Behavior on the PR branch:
```
$ ./src/bitcoin-cli -regtest -named createwallet wallet_name="invalid_wallet_02" disable_private_keys=true passphrase="passphrase"
error code: -4
error message:
Passphrase provided but private keys are disabled. A passphrase is only used to encrypt private keys, so cannot be used for wallets with private keys disabled.
$ ./src/bitcoin-cli -regtest -named createwallet wallet_name="invalid_wallet_02"
{
"name": "invalid_wallet_01",
"warning": ""
}
```
ACKs for top commit:
achow101:
ACK 6f29409ad1
Tree-SHA512: d192955fc2285bf27ae5dd4c1b7cfd3d85441a7f3554b189b974aefb319c6b997543991dbb0ca2c8cb980f7058913a77cf0164c02e9b51ceb9c2cb601317c428
This change only affects the subtest `test_superfluous_witness`.
Note that instead of creating a raw transaction first and then
signing it, we go the other direction here: MiniWallet creates a
transaction spending a segwit v1 output (i.e. including a witness),
then we turn it into a raw transaction by dropping the witness.
Therefore, the debug log asserts are swapped.
88376c623c test: Test for disabling wallet flags (Andrew Chow)
17ab31aa46 rpc, wallet: setwalletflags warnings are optional (Andrew Chow)
Pull request description:
Trying to disable a wallet flag with `setwalletflag` results in `Internal bug detected: 'std::any_of(m_results.m_results.begin(), m_results.m_results.end(), [ret](const RPCResult& res) { return res.MatchesType(ret); })'`. This occurs because the `warnings` field was not marked as optional. This PR makes `warnings` optional to avoid this error.
Also added a test case because apparently we didn't already have one.
ACKs for top commit:
w0xlt:
ACK 88376c6
Tree-SHA512: 4f5d3bebf0d022a5ad0f75d70c6562a43c7da6e39e9c3118733327d015c435e2c8d5004fdb039d42407dde5b21231a0f8827623d718abf611a1f06c15af5c806
The path is stored in `self.options.bitcoinutil`, points to
`src/bitcoin-util` by default and can be overrided with the
`BITCOINUTIL` environment variable.
494455f8a5 test: use MiniWallet for feature_fee_estimation.py (Sebastian Falbesoner)
Pull request description:
This PR enables one more of the non-wallet functional tests (feature_fee_estimation.py) to be run even with the Bitcoin Core wallet disabled by using the MiniWallet instead, as proposed in https://github.com/bitcoin/bitcoin/issues/20078. It takes use of the recently introduced methods `{create,send}_self_transfer_multi` (#24637) which allows to specify multiple UTXOs to be spent rather than only one. Very likely the test can still be simplified (e.g. coin selection in `small_txpuzzle_randfee`), but this is a first step.
ACKs for top commit:
ayush933:
tACK 494455f8 . The test runs successfully with the wallet disabled.
vincenzopalazzo:
tACK 494455f8a5
Tree-SHA512: 89789fc34a4374c79c4b90acd926ac69153aad655dab50450ed796f03c770bd675ad872e906f516f90e8d4cb40b83b55f3c78a94b13bfb8fe8f5e27624937748
0f7dc893ea test: compare `/chaininfo` response with `getblockchaininfo` RPC (brunoerg)
Pull request description:
The `/chaininfo` REST endpoint gets its infos from `getblockchaininfo` RPC, so this PR adds an `assert_equal` (in `interface_rest`) to ensure both responses are the same. Obs: other endpoints do the same for their respective RPC.
ACKs for top commit:
0xB10C:
Concept and Code Review ACK 0f7dc893ea. Belts-and-spenders.
Tree-SHA512: 51cbcf988090272e406a47dc869710740b74e2222af29c05ddcbf53bd49765cdc59efb525e970867f091b3d2efec4fb13371a342d9e484e51144b760265bc5b8
4394733331 Add DEBUG_LOCKCONTENTION documentation to the developer notes (Jon Atack)
39a34b6877 Put lock logging behind DEBUG_LOCKCONTENTION preprocessor directive (Jon Atack)
Pull request description:
This is a more minimal, no-frills version of #24734 for backport. The other fixes and improvements in that pull can be done after.
*Copy of the PR 24734 description:*
PRs #22736, #22904 and #23223 changed lock contention logging from a `DEBUG_LOCKCONTENTION` compile-time preprocessor directive to a runtime `lock` log category and improved the logging output. This changed the locking from using `lock()` to `try_lock()`:
- `void Mutex::UniqueLock::lock()` acquires the mutex and blocks until it gains access to it
- `bool Mutex::UniqueLock::try_lock()` doesn't block but instead immediately returns whether it acquired the mutex; it may be used by `lock()` internally as part of the deadlock-avoidance algorithm
In theory the cost of `try_lock` might be essentially the [same](https://www.erisian.com.au/bitcoin-core-dev/log-2022-03-31.html#l-697) relative to `lock`. The test-and-set logic of these calls is purported to be ~ constant time, optimised and light/quick if used carefully (i.e. no mutex convoying), compared to system calls, memory/cache coherency and fences, wait queues, and (particularly) lock contentions. See the discussion around https://github.com/bitcoin/bitcoin/pull/22736#issuecomment-902851054 and after with respect to performance/cost aspects. However, there are reasonable concerns (see [here](https://github.com/bitcoin/bitcoin/pull/22736#discussion_r691277896) and [here](https://www.erisian.com.au/bitcoin-core-dev/log-2022-03-31.html#l-620)) that `Base::try_lock()` may be potentially [costly](https://www.erisian.com.au/bitcoin-core-dev/log-2022-03-31.html#l-700) or [risky](https://github.com/bitcoin/bitcoin/pull/22904#issuecomment-930484001) compared to `Base::lock()` in this very frequently called code.
One alternative to keep the run-time lock logging would be to gate the `try_lock` call behind the logging conditional, for example as proposed in ccd73de1dd and ACKed [here](https://github.com/bitcoin/bitcoin/pull/22736#issuecomment-901980815). However, this would add the [cost](https://github.com/bitcoin/bitcoin/pull/22736#issuecomment-910102353) of `if (LogAcceptCategory(BCLog::LOCK))` to the hotspot, instead of replacing `lock` with `try_lock`, for the most frequent happy path (non-contention).
It turns out we can keep the advantages of the runtime lock contention logging (the ability to turn it on/off at runtime) while out of prudence putting the `try_lock()` call and `lock` logging category behind a `DEBUG_LOCKCONTENTION` compile-time preprocessor directive, and also still retain the lock logging enhancements of the mentioned PRs, as suggested in https://github.com/bitcoin/bitcoin/pull/24734#issuecomment-1085785480 by W. J. van der Laan, in https://github.com/bitcoin/bitcoin/pull/22736#discussion_r691280693, and in the linked IRC discussion.
Proposed here and for backport to v23.
ACKs for top commit:
laanwj:
Code review ACK 4394733331
Tree-SHA512: 89b1271cae1dca0eb251914b1a60fc5b68320aab4a3939c57eec3a33a3c8f01688f05d95dfc31f91d71a6ed80cfe2d67b77ff14742611cc206175e47b2e5d3b1
Changes the color of skipped functional tests to the default text color of the terminal. This will make skipped tests easy to read on the majority of background colors rather than the original grey color (hard to read on dark backgrounds) and the proposed yellow change (hard to read on white backgrounds)
65c49ac750 test: throw `ValueError` for invalid base58 checksum (Sebastian Falbesoner)
219d2c7ee1 contrib: testgen: use base58 methods from test framework (Sebastian Falbesoner)
605fecfb66 scripted-diff: rename `chars` to `b58chars` in test_framework.address (Sebastian Falbesoner)
11c63e374d contrib: testgen: import OP_* constants from test framework (Sebastian Falbesoner)
7d755bb31c contrib: testgen: avoid need for manually setting PYTHONPATH (Sebastian Falbesoner)
Pull request description:
This PR removes the redundant base58 implementation [contrib/testgen/base58.py](https://github.com/bitcoin/bitcoin/blob/master/contrib/testgen/base58.py) for the test generation script `gen_key_io_test_vectors.py` and uses the one from the test framework instead. Additionally, three other cleanups/improvements are done:
- import script operator constants `OP_*` from test framework instead of manually defining them
- add Python path to test framework directly in the script (via `sys.path.append(...)`) instead of needing the caller to specify `PYTHONPATH=...` on the command line (the same approach is done for the signet miner and the message capture scripts)
- rename `chars` to `b58chars` in the test_framework.address module (is more explicit and makes the diff for the base58 replacement smaller)
ACKs for top commit:
laanwj:
Code review ACK 65c49ac750
Tree-SHA512: 92e1534cc320cd56262bf455de7231c6ec821bfcd0ed58aa5718271ecec1a89df7951bf31527a2306db6398e7f2664d2ff8508200c28163c0b164d3f5aaf8b0e
76c60d7b31 test: validation:block_connected tracepoint test (0xb10c)
260e28ece8 test: utxocache:* tracepoint tests (0xb10c)
34b27bac68 test: net:in/out_message tracepoint tests (0xb10c)
c934087b62 test: checks for tracepoint tests (0xb10c)
Pull request description:
This adds functional tests for the USDT tracepoints added in https://github.com/bitcoin/bitcoin/pull/22006 and https://github.com/bitcoin/bitcoin/pull/22902. This partially fixes#23296. The tests **are probably skipped** on most systems as these tests require:
- a Linux system with a kernel that supports BPF (and available kernel headers)
- that Bitcoin Core is compiled with tracepoints for USDT support (default when compiled with depends)
- [bcc](https://github.com/iovisor/bcc) installed
- the tests are run with a privileged user that is able to e.g. do BPF syscalls and load BPF maps
The tests are not yet run in our CI as the CirrusCI containers lack the required permissions (see https://github.com/bitcoin/bitcoin/issues/23296#issuecomment-1024920845). Running the tests in a VM in the CI could work, but I haven't experimented with this yet. The priority was to get the actual tests done first to ensure the tracepoints work as intended for the v23.0 release. Running the tracepoint tests in the CI is planned as the next step to finish #23296.
The tests can, however, be run against e.g. release candidates by hand. Additionally, they provide a starting point for tests for future tracepoints. PRs adding new tracepoint should include tests. This makes reviewing these PRs easier.
The tests require privileges to execute BPF sycalls (`CAP_SYS_ADMIN` before Linux kernel 5.8 and `CAP_BPF` and `CAP_PERFMON` on 5.8+) and permissions to `/sys/kernel/debug/tracing/`. It's currently recommended to run the tests in a virtual machine (or on a VPS) where it's sensible to use the `root` user to gain these privileges. Never run python scripts you haven't carefully reviewed with `root` permissions! It's unclear if a non-root user can even gain the required privileges. This needs more experimenting.
The goal here is to test the tracepoint interface to make sure the [documented interface](https://github.com/bitcoin/bitcoin/blob/master/doc/tracing.md#tracepoint-documentation) does not break by accident. The tracepoints expose implementation details. This means we also need to rely on implementation details of Bitcoin Core in these functional tests to trigger the tracepoints. An example is the test of the `utxocache:flush` tracepoint: On Bitcoin Core shutdown, the UTXO cache is flushed twice. The corresponding tracepoint test expects two flushes, too - if not, the test fails. Changing implementation details could cause these tests to fail and the tracepoint API to break. However, we purposefully treat the tracepoints only as [**semi-stable**](https://github.com/bitcoin/bitcoin/blob/master/doc/tracing.md#semi-stable-api). The tracepoints should not block refactors or changes to other internals.
ACKs for top commit:
jb55:
tACK 76c60d7b31
laanwj:
Tested ACK 76c60d7b31
Tree-SHA512: 9a63d945c68102e59d751bd8d2805ddd7b37185408fa831d28a9cb6641b701961389b55f216c475df7d4771154e735625067ee957fc74f454ad7a7921255364c
cccc4e879a Remove nHeightEnd and nHeight in generateBlocks helper (MarcoFalke)
fa38b1c8bd Remove buggy and confusing IncrementExtraNonce (MarcoFalke)
Pull request description:
IncrementExtraNonce has many issues:
* It is test-only code, but part of bitcoind
* It is using the block height of the tip, as opposed to the block's previous block as reference for the new height. See https://github.com/bitcoin/bitcoin/issues/24730#issuecomment-1085586193
* It has no use case in regtest testing. With a low difficulty the extra nonce won't be incremented. With a high difficulty the test-only functions are clumsy to handle anyway. For example, the generate* RPCs will return an empty array once they reached `maxtries`, as opposed to an error. Also the calls can't be aborted early unless the node shuts down completely. So I think it is fine to just remove the extra nonce functionality and leave it to the outside to implement, if needed. For example, a wrapper script can call the `generate*` RPCs once every second, to use the timestamp as extra nonce.
ACKs for top commit:
ajtowns:
ACK cccc4e879a
Tree-SHA512: d8a3989ad280ebd4b1b574159b3a396b8a42134347e6be3c88445162d86624d221c416456f45ae75aea62ed8c8a1a9bb3a2532924abca2ef7a879cb8e6b15654
54b39cfb34 Add release notes (stickies-v)
f959fc0397 Update /<count>/ endpoints to use a '?count=' query parameter instead (stickies-v)
a09497614e Add GetQueryParameter helper function (stickies-v)
fff771ee86 Handle query string when parsing data format (stickies-v)
c1aad1b3b9 scripted-diff: rename RetFormat to RESTResponseFormat (stickies-v)
9f1c54787c Refactoring: move declarations to rest.h (stickies-v)
Pull request description:
In RESTful APIs, [typically](https://rapidapi.com/blog/api-glossary/parameters/query/) path parameters (e.g. `/some/unique/resource/`) are used to represent resources, and query parameters (e.g. `?sort=asc`) are used to control how these resources are being loaded through e.g. sorting, pagination, filtering, ...
As first [discussed in #17631](https://github.com/bitcoin/bitcoin/pull/17631#discussion_r733031180), the [current REST api](https://github.com/bitcoin/bitcoin/blob/master/doc/REST-interface.md) contains two endpoints `/headers/` and `/blockfilterheaders/` that rather unexpectedly use path parameters to control how many (filter) headers are returned in the response. While this is no critical issue, it is unintuitive and we are still early enough to easily phase this behaviour out and ensure new endpoints (if any) do not have to stick to non-standard behaviour just for internal consistency.
In this PR, a new `HTTPRequest::GetQueryParameter` method is introduced to easily parse query parameters, as well as two new `/headers/` and `/blockfilterheaders/` endpoints that use a count query parameter are introduced. The old path parameter-based endpoints are kept without too much overhead, but the documentation now points to the new query parameter-based endpoints as the default interface to encourage standardness.
## Behaviour change
### New endpoints and default values
`/headers/` and `/blockfilterheaders/` now have 2 new endpoints that contain query parameters (`?count=<count>`) instead of path parameters (`/<count>/`), as described in REST-interface.md. Since query parameters can easily have default values, I have set this at 5 for both endpoints.
**headers**
`GET /rest/headers/<BLOCK-HASH>.<bin|hex|json>?count=<COUNT=5>`
should now be used instead of
`GET /rest/headers/<COUNT>/<BLOCK-HASH>.<bin|hex|json>`
**blockfilterheaders**
`GET /rest/blockfilterheaders/<FILTERTYPE>/<BLOCK-HASH>.<bin|hex|json>?count=<COUNT=5>`
should now be used instead of
`GET /rest/blockfilterheaders/<FILTERTYPE>/<COUNT>/<BLOCK-HASH>.<bin|hex|json>`
### Some previously invalid API calls are now valid
API calls that contained query strings in the URI could not be parsed prior to this PR. This PR changes behaviour in that previously invalid calls (e.g. `GET /rest/headers/5/somehash.json?someunusedparam=foo`) would now become valid, as the query parameters are properly parsed, and discarded if unused.
For example, prior to this PR, adding an irrelevant `someparam` parameter would be illegal:
```
GET /rest/headers/5/0000004c6aad0c89c1c060e8e116dcd849e0554935cd78ff9c6a398abeac6eda.json?someparam=true
->
Invalid hash: 0000004c6aad0c89c1c060e8e116dcd849e0554935cd78ff9c6a398abeac6eda.json?someparam=true
```
**This behaviour change affects all rest endpoints, not just the 2 new ones introduced here.**
*(Note: I'd be open to implementing additional logic to refuse requests containing unrecognized query parameters to minimize behaviour change, but for the endpoints that we currently have I don't really see the point for that added complexity. E.g. I don't see any scenarios where misspelling a parameter could lead to harmful outcomes)*
## Using the REST API
To run the API HTTP server, start a bitcoind instance with the `-rest` flag enabled. To use the
`blockfilterheaders` endpoint, you'll also need to set `-blockfilterindex=1`:
```
./bitcoind -signet -rest -blockfilterindex=1
```
As soon as bitcoind is fully up and running, you should be able to query the API, for example by
using curl on the command line: ```curl "127.0.0.1:38332/rest/chaininfo.json"```.
To more easily parse the JSON output, you can also use tools like 'jq' or `json_pp`, e.g.:
```
curl -s "localhost:38332/rest/blockfilterheaders/basic/0000004c6aad0c89c1c060e8e116dcd849e0554935cd78ff9c6a398abeac6eda.json?count=2" | json_pp .
```
## To do
- [x] update `doc/release-notes`
## Feedback
This is my first PR (hooray!). Please don't hold back on any feedback/comments/nits/... you may have, big or small, whether they are code, process, language, ... related. I welcome private messages too if there's anything you don't want to clutter the PR with. I'm here to learn and am grateful for everyone's input.
ACKs for top commit:
stickies-v:
I've had to push a tiny doc update to `REST-interface.md` (`git range-diff 219d728 9aac438 54b39cf`) since this was not merged for v23, but since there are no significant changes beyond theStack and jnewbery's ACKs I think this PR is now ready to be considered for merging? @MarcoFalke
jnewbery:
ACK 54b39cfb34
theStack:
re-ACK 54b39cfb34
Tree-SHA512: 3b393ffde34f25605ca12c0b1300799a19684b816a1d03aed38b0f5439df47bfe6a589ffbcd7b83fd2def6c9d00a1bae5e45b1d18df4ae998c617c709990f83f
In most RESTful APIs, path parameters are used to represent resources, and
query parameters are used to control how these resources are being filtered/sorted/...
The old /<count>/ functionality is kept alive to maintain backwards compatibility,
but new paths with query parameters are introduced and documented as the default
interface so future API methods don't break consistency by using query parameters.
fa9112aac0 Remove utxo db upgrade code (MarcoFalke)
Pull request description:
It is not possible to upgrade Bitcoin Core pre-segwit (pre-0.13.1) to a recent version without a full IBD from scratch after commit 19a56d1519 (released in version 22.0).
Any Bitcoin Core version with the new database format after commit 1088b02f0c (released in version 0.15), can upgrade to any version that is supported as of today.
This leaves the versions 0.13.1-0.14.x. Even though those versions are unsupported, some users with an existing datadir may want to upgrade to a recent version. However, it seems reasonable to simply ask them to `-reindex` to run a full IBD from scratch. This allows us to remove the utxo db upgrade code.
ACKs for top commit:
Sjors:
re-ACK fa9112aac0
laanwj:
Code review ACK fa9112aac0
Tree-SHA512: 4243bb35df9ac4892f9fad30fe486d338745952bcff4160bcb0937c772d57b13b800647da14695e21e3655e85ee0d95fa3dc7789ee309d59ad84f422297fecb8
d6bc2322ed test: -peerblockfilters without -blockfilterindex raises an error (brunoerg)
Pull request description:
This PR adds test coverage for the following init error:
2a3e8fb359/src/init.cpp (L850)
Setting -peerblockfilters without -blockfilterindex should raise an error when initializing.
ACKs for top commit:
ccdle12:
Tested ACK d6bc2322ed
Tree-SHA512: e740c2ccde6bb1bb8381bb676a6d01bd5746cf9ce0c8dadd62067a6b9b380027bfe8b8cdeae9846a0ab18385f3dc5dff607fe5274cb55107d47470db00015fb2
bb84b7145b add tests for no recipient and using send_max while inputs are specified (ishaanam)
49090ec402 Add sendall RPC née sweep (Murch)
902793c777 Extract FinishTransaction from send() (Murch)
6d2208a3f6 Extract interpretation of fee estimation arguments (Murch)
a31d75e5fb Elaborate error messages for outdated options (Murch)
35ed094e4b Extract prevention of outdated option names (Murch)
Pull request description:
Add sendall RPC née sweep
_Motivation_
Currently, the wallet uses a fSubtractFeeAmount (SFFO) flag on the
recipients objects for all forms of sending calls. According to the
commit discussion, this flag was chiefly introduced to permit sweeping
without manually calculating the fees of transactions. However, the flag
leads to unintuitive behavior and makes it more complicated to test
many wallet RPCs exhaustively. We proposed to introduce a dedicated
`sendall` RPC with the intention to cover this functionality.
Since the proposal, it was discovered in further discussion that our
proposed `sendall` rpc and SFFO have subtly different scopes of
operation.
• sendall:
Use _given UTXOs_ to pay a destination the remainder after fees.
• SFFO:
Use a _given budget_ to pay an address the remainder after fees.
While `sendall` will simplify cases of spending a given set of
UTXOs such as paying the value from one or more specific UTXOs, emptying
a wallet, or burning dust, we realized that there are some cases in
which SFFO is used to pay other parties from a limited budget,
which can often lead to the creation of change outputs. This cannot be
easily replicated using `sendall` as it would require manual
computation of the appropriate change amount.
As such, sendall cannot replace all uses of SFFO, but it still has a
different use case and will aid in simplifying some wallet calls and
numerous wallet tests.
_Sendall call details_
The proposed sendall call builds a transaction from a specific
subset of the wallet's UTXO pool (by default all of them) and assigns
the funds to one or more receivers. Receivers can either be specified
with a given amount or receive an equal share of the remaining
unassigned funds. At least one recipient must be provided without
assigned amount to collect the remainder. The `sendall` call will
never create change. The call has a `send_max` option that changes the
default behavior of spending all UTXOs ("no UTXO left behind"), to
maximizing the output amount of the transaction by skipping uneconomic
UTXOs. The `send_max` option is incompatible with providing a specific
set of inputs.
---
Edit: Replaced OP with latest commit message to reflect my updated motivation of the proposal.
ACKs for top commit:
achow101:
re-ACK bb84b7145b
Tree-SHA512: 20aaf75d268cb4b144f5d6437d33ec7b5f989256b3daeeb768ae1e7f39dc6b962af8223c5cb42ecc72dc38cecd921c53c077bc0ec300b994e902412213dd2cc3
_Motivation_
Currently, the wallet uses a fSubtractFeeAmount (SFFO) flag on the
recipients objects for all forms of sending calls. According to the
commit discussion, this flag was chiefly introduced to permit sweeping
without manually calculating the fees of transactions. However, the flag
leads to unintuitive behavior and makes it more complicated to test
many wallet RPCs exhaustively. We proposed to introduce a dedicated
`sendall` RPC with the intention to cover this functionality.
Since the proposal, it was discovered in further discussion that our
proposed `sendall` rpc and SFFO have subtly different scopes of
operation.
• sendall:
Use _specific UTXOs_ to pay a destination the remainder after fees.
• SFFO:
Use a _specific budget_ to pay an address the remainder after fees.
While `sendall` will simplify cases of spending from specific UTXOs,
emptying a wallet, or burning dust, we realized that there are some
cases in which SFFO is used to pay other parties from a limited budget,
which can often lead to the creation of change outputs. This cannot be
easily replicated using `sendall` as it would require manual computation
of the appropriate change amount.
As such, sendall cannot replace all uses of SFFO, but it still has a
different use case and will aid in simplifying some wallet calls and
numerous wallet tests.
_Sendall call details_
The proposed sendall call builds a transaction from a specific subset of
the wallet's UTXO pool (by default all of them) and assigns the funds to
one or more receivers. Receivers can either be specified with a specific
amount or receive an equal share of the remaining unassigned funds. At
least one recipient must be provided without assigned amount to collect
the remainder. The `sendall` call will never create change. The call has
a `send_max` option that changes the default behavior of spending all
UTXOs ("no UTXO left behind"), to maximizing the output amount of the
transaction by skipping uneconomic UTXOs. The `send_max` option is
incompatible with providing a specific set of inputs.
fa0758e145 test: Add diamond-shape prioritisetransaction test (MarcoFalke)
fa450c18db test: Rework create_self_transfer_multi (MarcoFalke)
Pull request description:
Looks like there is no test for diamonds, only for chains (in `mempool_packages.py`)
ACKs for top commit:
jamesob:
ACK fa0758e145
Tree-SHA512: d261184a81df77d24fc256f58ad5ed4a13b7cd4e33f74c8b79495c761ff417817602d8e5d4f63f4bb1000ac63f89bbfa54d8d8994a7b2bb2e8a484c467330984
45e67b2695 test: invalid -i2psam will raise an init error (brunoerg)
Pull request description:
This PR adds test coverage (at `feature_proxy.py`) for the following init error:
2f0f056e08/src/init.cpp (L1791)
It starts the node with an invalid -i2psam (`-i2psam=invalidhere`) and test if it raises an error when initializing.
ACKs for top commit:
dunxen:
Code review ACK 45e67b2
Tree-SHA512: b24e3f6e7a9316b9ebc0b6c8bcf1315faff60a9e258d7bb3dbeb9f6695a728bb3083aea2f81114072fe13822bfca34d4a0f44f229825f7c97a81619d810010c0
89bb25d22a test: check localaddresses in getnetworkinfo for nodes with proxy (brunoerg)
Pull request description:
This PR adds test coverage for the field `localaddresses` for `getnetworkinfo`. In this case, it verifies if this field is empty for all nodes since they are using proxy.
Reference:
515200298b/src/init.cpp (L449)
ACKs for top commit:
jonatack:
ACK 89bb25d22a
Tree-SHA512: 3c765c7060b6972c1ae5a1104734cd7669b650b5f6aa4f623f4299567732260da5083fef306a7c1e71c931f5d1396f24abad251d95c3d82b1f3ee0efee7fcd1f
9053f64fcb [doc] release notes for random change target (glozow)
46f2fed6c5 [wallet] remove MIN_CHANGE (glozow)
a44236addd [wallet] randomly generate change targets (glozow)
1e52e6bd0a refactor coin selection for parameterizable change target (glozow)
Pull request description:
Closes#24458 - the wallet always chooses 1 million sats as its change target, making it easier to fingerprint transactions created by the Core wallet. Instead of using a fixed value, choose one randomly each time (within a range). Using 50ksat (around $20) as the lower bound and `min(1 million sat, 2 * average payment value)` as the upper bound.
RFC: If the payment is <25ksat, this doesn't work, so we're using the range (payment amount, 50ksat) instead.
ACKs for top commit:
achow101:
ACK 9053f64fcb
Xekyo:
reACK 9053f64fcb
Tree-SHA512: 45ce5d064697065549473347648e29935733f3deffc71a6ab995449431f60302d1f9911a0994dfdb960b48c48b5d8859f168b396ff2a62db67d535a7db041d35
da2bc865d6 [wallet] don't create long chains by default (glozow)
Pull request description:
Default mempool policy doesn't let you have chains longer than 25 transactions. This is locally configurable of course, but it's not really safe to assume that a chain longer than 25 transactions will propagate. Thus, the wallet should probably avoid creating such transactions by default; set `DEFAULT_WALLET_REJECT_LONG_CHAINS` to true.
Closes#9752Closes#10004
ACKs for top commit:
MarcoFalke:
re-ACK da2bc865d6 only change is fixing typos in tests 🎏
Tree-SHA512: 65d8e4ec437fe928adf554aa7e819a52e0599b403d5310895f4e371e99bbc838219b3097c4d2f775bc870ac617ef6b4227b94291f2b376f824f14e8f2b152f31
0000ff0d6b test: move-only: Move all generate* tests to a single file (MarcoFalke)
Pull request description:
Seems a bit overkill to spread tests for the `generate*` methods over several files. Combining them into a single file has also a nice side-effect of requiring less node (re)starts, which are expensive in valgrind.
ACKs for top commit:
glozow:
utACK 0000ff0d6b
Tree-SHA512: 8269eb05649a871011bbfbd1838d0f7d1dac4a35b3b198fc43fe85131fda8a53803b75da78cbf422eabf086006dee4421e622fbe706f6781a3848b989024001b
1066d10f71 scripted-diff: rename TxRelay members (John Newbery)
575bbd0dea [net processing] Move tx relay data to Peer (John Newbery)
785f55f7ee [net processing] Move m_wtxid_relay to Peer (John Newbery)
36346703f8 [net] Add CNode.m_relays_txs and CNode.m_bloom_filter_loaded (John Newbery)
Pull request description:
This continues the work of moving application layer data into net_processing, by moving all tx data into the new Peer object added in #19607.
For motivation, see #19398.
ACKs for top commit:
dergoegge:
ACK 1066d10f71 - This is a good layer separation improvement with no behavior changes.
glozow:
utACK 1066d10f71
Tree-SHA512: 0c9d6b8a0a05e2d816b6d6588b7df133842ec960ae67667813422aa7bd8eb5308599c714f3822a98ddbdf364ffab9050b055079277ba4aff24092557ff99ebcc
cccc1e70b8 Enforce Taproot script flags whenever WITNESS is set (MarcoFalke)
fa42299411 Remove nullptr check in GetBlockScriptFlags (MarcoFalke)
faadc606c7 refactor: Pass const reference instead of pointer to GetBlockScriptFlags (MarcoFalke)
Pull request description:
Now that Taproot is active, it makes sense to enforce its rules on all blocks, even historic ones, regardless of the deployment status.
### Benefits:
(With "script flags" I mean "taproot script verification flags".)
* Script flags are known ahead for all blocks (even blocks not yet created) and do not change. This may benefit static analysis, code review, and development of new script features that build on Taproot.
* Any future bugs introduced in the deployment code won't have any effect on the script flags, as they are independent of deployment.
* Enforcing the taproot rules regardless of the deployment status makes testing easier because invalid blocks after activation are also invalid before activation. So there is no need to differentiate the two cases.
* It gives belt-and-suspenders protection against a practically expensive and theoretically impossible IBD reorg attack where the node is eclipsed. While `nMinimumChainWork` already protects against this, the cost for a few months worth of POW might be lowered until a major version release of Bitcoin Core reaches EOL. The needed work for the attack is the difference between `nMinimumChainWork` and the work at block 709632.
For reference, previously the same was done for P2SH and WITNESS in commit 0a8b7b4b33.
### Implementation:
I found one block which fails verification with the flags applied, so I added a `TaprootException`, similar to the `BIP16Exception`.
For reference, the debug log:
```
ERROR: ConnectBlock(): CheckInputScripts on b10c007c60e14f9d087e0291d4d0c7869697c6681d979c6639dbd960792b4d41 failed with non-mandatory-script-verify-flag (Witness program was passed an empty witness)
BlockChecked: block hash=0000000000000000000f14c35b2d841e986ab5441de8c585d5ffe55ea1e395ad state=non-mandatory-script-verify-flag (Witness program was passed an empty witness)
InvalidChainFound: invalid block=0000000000000000000f14c35b2d841e986ab5441de8c585d5ffe55ea1e395ad height=692261 log2_work=92.988459 date=2021-07-23T08:24:20Z
InvalidChainFound: current best=0000000000000000000067b17a4c0ffd77c29941b15ad356ca8f980af137a25d height=692260 log2_work=92.988450 date=2021-07-23T07:47:31Z
ERROR: ConnectTip: ConnectBlock 0000000000000000000f14c35b2d841e986ab5441de8c585d5ffe55ea1e395ad failed, non-mandatory-script-verify-flag (Witness program was passed an empty witness)
```
Hint for testing, make sure to set `-noassumevalid`.
### Considerations
Obviously this change can lead to consensus splits on the network in light of massive reorgs. Currently the last block before Taproot activation, that is the last block without the Taproot script flags set, is only buried by a few days of POW. However, when and if this patch is included in the next major release, it will be buried by a few months of POW. BIP90 considerations apply when looking at reorgs this large.
ACKs for top commit:
Sjors:
tACK cccc1e70b8
achow101:
ACK cccc1e70b8
laanwj:
Code review ACK cccc1e70b8
ajtowns:
ACK cccc1e70b8 ; code review; wrote a "getblockscriptflags" rpc to quickly check that blocks just had bit 17 (taproot) added; review of earlier revisions had established non-exception blocks do validate with taproot rules enabled.
jamesob:
ACK cccc1e70b8 ([`jamesob/ackr/23536.1.MarcoFalke.enforce_taproot_script_f`](https://github.com/jamesob/bitcoin/tree/ackr/23536.1.MarcoFalke.enforce_taproot_script_f))
Tree-SHA512: 00044de68939caef6420ffd588c1291c041a8b397c80a3df1e3e3487fbeae1821d23975c51c95e44e774558db76f943b00b4e27cbd0213f64a9253116dc6edde
58a14795b8 test: passing -onlynet=onion with -onion=0/-noonion raises expected init error (Jon Atack)
7000f66d36 test: passing -onlynet=onion without -proxy/-onion raises expected init error (Jon Atack)
8332e6e4cf test: passing invalid -onion raises expected init error (Jon Atack)
d5edb08708 test: passing invalid -proxy raises expected init error (Jon Atack)
bd57dcbaf2 test: hoist proxy out of 2 network loops in feature_proxy.py (Jon Atack)
afdf2de282 test: add CJDNS to LimitedAndReachable_Network unit tests (Jon Atack)
2b7a8180a9 net, init: assert each network reachability is true by default (Jon Atack)
Pull request description:
Adds missing network reachability test coverage and an assertion during init, noticed while reviewing #22834:
- assert during init that each network reachability is true by default
- add CJDNS to the `LimitedAndReachable_Network` unit tests
- hoist proxy out of two network loops in feature_proxy.py
- test that passing invalid `-proxy` raises expected init error
- test that passing invalid `-onion` raises expected init error
- test that passing `-onlynet=onion` without `-proxy` and `-onion` raises expected init error
- test that passing `-onlynet=onion` with `-onion=0` and with `-noonion` raises expected init error
ACKs for top commit:
vasild:
ACK 58a14795b8
brunoerg:
ACK 58a14795b8
dongcarl:
Code Review ACK 58a14795b8
Tree-SHA512: bdee6dd0c12bb63591ce7c9321fe77b509ab1265123054e774adc38a187746dddafe1627cbe89e990bcc78b45e194bfef8dc782710d5b217e2e2106ab0158827
bc90b8d869 [move only] remove `is_wallet_compiled` checks (josibake)
0bfbf7fb24 test: use MiniWallet in `interfaces_zmq` (josibake)
Pull request description:
While working on #24584 , `interface_zmq` started failing due to coin selection not running deterministically. The test doesn't actually need the wallet, so this PR migrates it to use MiniWallet
_Note for reviewers:_ the second commit moves large chunks of code out of an if block, so it may be helpful to review with something that ignores whitespace, e.g `git diff -w master`
ACKs for top commit:
vincenzopalazzo:
ACK bc90b8d869
Tree-SHA512: c618e23d00635d72dafdef28e68cbc88b9cc2030d4898fc5b7eac926fd621684c1958c075ed167192716b18308da5a0c1f1393396e31b99d0d3bde78b78fefc5
The external input test with specifying input weight would make a
pessimistic estimate of the input weight. However this would result in a
test failure as it is sometimes too pessimistic when an ECDSA signature
ends up being smaller than usual. To correct this, we can calculate the
input weight more accurately.
b2813980b8 init: disallow reindex-chainstate when pruning (Martin Zumsande)
Pull request description:
The combination of `-reindex-chainstate` and `-prune` currently makes the node stuck in an endless loop:
- `LoadChainstate()` will wipe the existing chainstate (so we have no genesis block anymore). It won't clean up unusable block files by calling `CleanupBlockRevFiles()` as for full `-reindex`.
- `ThreadImport()` has [logic](91d12344b1/src/node/blockstorage.cpp (L855)) of reloading Genesis after reindexing. This is what makes full `-reindex` work with `-prune` but it's not executed for `-reindex-chainstate`.
- Since we still don't have a genesis block, init will wait for it forever in an endless loop ([code](91d12344b1/src/init.cpp (L1630-L1640))).
Fix this by disallowing `-reindex-chainstate` together with `-prune`. This is discouraged in the help for `-reindex-chainstate` anyway ("When in pruning mode or if blocks on disk might be corrupted, use full -reindex instead.") but wasn't enforced.
Fixes#24242
ACKs for top commit:
MarcoFalke:
cr ACK b2813980b8
Tree-SHA512: 7220842daaf9a4f972d82b13b81fdeac2833bf5e665c5b0f8eaf6a4bcd0725c8e97d19ec956ca4b730065a983475bb3a2732713d338f4caf8666ccbf63d4d988
