201b9a02fd test: fix intermittent failure in feature_index_prune.py (Martin Zumsande)
Pull request description:
I can't reproduce the error from #26630 locally, but from analying the logs I think the problem is the following:
After calling `sync_blocks`, we didn't check that the indexes have caught up to the tip before performing the manual pruning. This could possibly lead to prune blockers with a lower height than the expected 2489, which do appear in the logs of the failed CI runs, e.g.
- `2022-10-27T21:14:17.703920Z [C:\Users\ContainerAdministrator\AppData\Local\Temp\cirrus-ci-build\src\validation.cpp:2395] [FlushStateToDisk] [prune] coinstatsindex limited pruning to height 2488` ([Cirrus](https://cirrus-ci.com/task/5443742333665280?logs=functional_tests#L2506))
So, this should be fixed by a call to `sync_index`.
Fixes#26330
ACKs for top commit:
brunoerg:
crACK 201b9a02fd
Tree-SHA512: fb7023c9eb2ba6d0e69e059a401453cbdf63abc6804543dffcf36ba9f93c9cd13209e57aa5536d94b2e420c9d4cd0b1a7eff1adadd19aa7b3c33f592502e1bc0
fa24239a1c net: Avoid SetTxRelay for feeler connections (MacroFake)
Pull request description:
Seems odd to reserve memory for the struct (the heaviest member being `m_tx_inventory_known_filter`) when it is never used.
This also avoids sending out `msg_sendtxrcncl` before disconnecting. This shouldn't matter, as other messages, such as `msg_wtxidrelay`, `msg_sendaddrv2`, `msg_verack` or `msg_getaddr` are still sent. Though, it allows to test the changes here as a side-effect.
ACKs for top commit:
naumenkogs:
ACK fa24239a1c
vasild:
ACK fa24239a1c
jonatack:
ACK fa24239a1c
mzumsande:
ACK fa24239a1c
Tree-SHA512: d7604c7eb4df8f2de811e600bdd312440ee03e508d3a0f09ae79f7f2d3eeec663bfd47a2d079fa50b756d61e35dfa998de068a7b9afaf35378fa0e62a538263d
f1ee974e8e test: remove unused `CHANGE_{XPRV,XPUB}` constants (Sebastian Falbesoner)
Pull request description:
These constants exist since the introduction of the functional test wallet_taproot.py (2667366aaa), but they have never been used.
ACKs for top commit:
instagibbs:
ACK f1ee974e8e
brunoerg:
ACK f1ee974e8e
Tree-SHA512: 2c6f8cd537c35af59c4cbe6abb07386b7fb9e4662c7bed27ad8935e004f9bf452705fb1caef4f5ea1f4ad509745afba966005f95a18cf2b59e00b7739a5df7f8
e049fd76f0 Bugfix: Check for readlink buffer overflow and handle gracefully (Luke Dashjr)
Pull request description:
If readlink returns the size of the buffer, an overflow may have (safely) occurred.
Pass a buffer size of MAX_PATH+1 (the size of the actual buffer) to detect this scenario.
ACKs for top commit:
hebasto:
ACK e049fd76f0.
Tree-SHA512: 188bace79cbe556efe7782e46b870c02729b07b104a9316b0f7d50013504972e85baf507403d2d6060bb2bf3e13f40d735bddd18255d97a60810208c3de87691
98868633d1 Bugfix: configure: bitcoin-{cli,tx,util} don't need UPnP, NAT-PMP, or ZMQ (Luke Dashjr)
Pull request description:
As with #23345, these other tools likewise don't use various deps.
ACKs for top commit:
achow101:
ACK 98868633d1
Tree-SHA512: 4be056b8e0c9f69834229aa257187457de1bc34214d320b770834e21ecc1f0ca7aa7b9689fba525928947bfabbb461528795f709014fb9618b82f088fe64f271
0f38524c31 doc: correct deriveaddresses RPC name (Bitcoin Hodler)
Pull request description:
There never was a `deriveaddress` RPC, from what I can tell. It was always called `deriveaddresses` (plural).
ACKs for top commit:
theStack:
ACK 0f38524c31
Zero-1729:
ACK 0f38524c31
Tree-SHA512: 3f405f5479a0d39cf150fd80b4d854ffe4eef718a358202c619e34a08d98c1252b82fc70d106cdf2215dc5a50c6f6cd5e26fe7ed87156f6b08f8e97d963affb7
8a9f1e4d18 test: fix intermittent failure in rpc_getblockfrompeer.py (Martin Zumsande)
Pull request description:
Fixes an intermittent failure in `rpc_getblockfrompeer.py` observed in https://cirrus-ci.com/task/6610115527704576 by adding a sync to make sure the node has processed the header we sent it before we query it for the corresponding block.
Fixes#26412
ACKs for top commit:
jonatack:
ACK 8a9f1e4d18
Tree-SHA512: f6188ab3cfd863034e44e9806d0d99a8781462bec94141501aefc71589153481ffb144e126326ab81807c2b2c93de7f4aac5d09dbcf26c4512a176e06fc6e5f8
ef97b89902 Exclude rand from debug log (Jeff Ruane)
Pull request description:
Currently, `debug.log` is spammed with messages like this from `random.cpp` when functional tests are run.
```
2022-10-25T19:24:34.787663Z [scheduler] [random.cpp:519] [SeedPeriodic] [rand] Feeding 36565 bytes of dynamic environment data into RNG
```
These logs are not useful for debugging and decrease the signal-to-noise ratio of the logs, so they should be suppressed by excluding the `rand` category, as the `libevent` and `leveldb` categories currently are.
ACKs for top commit:
LarryRuane:
ACK ef97b89902
kouloumos:
ACK ef97b89902, confirmed that this log level is only used in `random.cpp` and indeed it seems that it doesn't add any value to the debug.log during functional tests.
satsie:
ACK ef97b89902
theStack:
ACK ef97b89902
Tree-SHA512: 5cea384a3197f0ec77efa9efc77822914450ecf5546606568bbd432c3536040c772c57aef58d3bb083a2e5e756f690766fa1fb382ab1973748db238108a58746
To avoid a wallet potentially being able to sign a transaction using
keys from descriptors imported in previous tests, make new wallets for
each test case rather than sharing them.
In addition to the pubkeys in hd_keypaths and tap_bip32_keypaths, also
see if the descriptor can produce a SigningProvider for the output
pubkey.
Also slightly refactors this area to reduce code duplication.
Taproot pubkey info was not being added for multi_a signing. The filling
of this info is moved into the common function CreateTaprootScriptSig so
that any signing of taproot scripts will include the pubkey info.
fa29ef00ad refactor: Silence GCC Wmissing-field-initializers in ChainstateManagerOpts (MacroFake)
Pull request description:
The `std::optional` fields in the struct that fall back to chain param defaults if not provided should be initialized to `std::nullopt`. This already happens with the current code.
However, for consistency with `check_block_index` and to silence a GCC warning, add the "missing" `{}`.
ACKs for top commit:
achow101:
ACK fa29ef00ad
hebasto:
ACK fa29ef00ad, tested on Ubuntu 22.04 + GCC 11.3.
jonatack:
ACK fa29ef00ad
Tree-SHA512: bdec9c56df5d601a5616e107fed48737b13b0a7242b6526092fb682b5016544a4bc08666b60304c668d44c6f7ac69d3788093d921382c1d6c577c1f9fe31fc50
After syncing the blocks, we didn't check that the
indexes have caught up to the tip before manually pruning.
This could lead to prune blockers lower thatn the expected height.
Currently, debug.log is spammed with messages from random.cpp
when functional tests are run. These logs are not useful for
debugging, and decrease the signal to noise ratio of the logs.
3fcb545ab2 bench: benchmark transaction creation process (furszy)
a8a75346d7 wallet: SelectCoins, return early if target is covered by preset-inputs (furszy)
f41712a734 wallet: simplify preset inputs selection target check (furszy)
5baedc3351 wallet: remove fetch pre-selected-inputs responsibility from SelectCoins (furszy)
295852f619 wallet: encapsulate pre-selected-inputs lookup into its own function (furszy)
37e7887cb4 wallet: skip manually selected coins from 'AvailableCoins' result (furszy)
94c0766b0c wallet: skip available coins fetch if "other inputs" are disallowed (furszy)
Pull request description:
#### # Context (Current Flow on Master)
In the transaction creation process, in order to select which coins the new transaction will spend,
we first obtain all the available coins known by the wallet, which means walking-through the
wallet txes map, gathering the ones that fulfill certain spendability requirements in a vector.
This coins vector is then provided to the Coin Selection process, which first checks if the user
has manually selected any input (which could be internal, aka known by the wallet, or external),
and if it does, it fetches them by searching each of them inside the wallet and/or inside the
Coin Control external tx data.
Then, after finding the pre-selected-inputs and gathering them in a vector, the Coin Selection
process walks-through the entire available coins vector once more just to erase coins that are
in both vectors. So the Coin Selection process doesn’t pick them twice (duplicate inputs inside
the same transaction).
#### # Process Workflow Changes
Now, a new method, `FetchCoins` will be responsible for:
1) Lookup the user pre-selected-inputs (which can be internal or external).
2) And, fetch the available coins in the wallet (excluding the already fetched ones).
Which will occur prior to the Coin Selection process. Which allows us to never include the
pre-selected-inputs inside the available coins vector in the first place, as well as doing other
nice improvements (written below).
So, Coin Selection can perform its main responsibility without mixing it with having to fetch
internal/external coins nor any slow and unneeded duplicate coins verification.
#### # Summarizing the Improvements:
1) If any pre-selected-input lookup fail, the process will return the error right away.
(before, the wallet was fetching all the wallet available coins, walking through the
entire txes map, and then failing for an invalid pre-selected-input inside SelectCoins)
2) The pre-selected-inputs lookup failure causes are properly described on the return error.
(before, we were returning an "Insufficient Funds" error for everything, even if the failure
was due a not solvable external input)
3) **Faster Coin Selection**: no longer need to "remove the pre-set inputs from the available coins
vector so that Coin Selection doesn't pick them" (which meant to loop-over the entire
available coins vector at Coin Selection time, erasing duplicate coins that were pre-selected).
Now, the available coins vector, which is built after the pre-selected-inputs fetching,
doesn’t include the already selected inputs in the first place.
4) **Faster transaction creation** for transactions that only use manually selected inputs.
We now will return early, as soon as we finish fetching the pre-selected-inputs and
not perform the resources expensive calculation of walking-through the entire wallet
txes map to obtain the available coins (coins that we will not use).
---------------------------
Added a new bench (f6d0bb2) measuring the transaction creation process, for a wallet with ~250k UTXO, only using the pre-selected-inputs inside coin control. Setting `m_allow_other_inputs=false` to disallow the wallet to include coins automatically.
#### Result on this PR (tip f6d0bb2d):
| ns/op | op/s | err% | total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
| 1,048,675.00 | 953.58 | 0.3% | 0.06 | `WalletCreateTransaction`
vs
#### Result on master (tip 4a4289e2):
| ns/op | op/s | err% | total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
| 96,373,458.20 | 10.38 | 0.2% | 5.30 | `WalletCreateTransaction`
The benchmark took to run in master: **96.37 milliseconds**, while in this PR: **1 millisecond** 🚀 .
ACKs for top commit:
S3RK:
Code Review ACK 3fcb545ab2
achow101:
ACK 3fcb545ab2
aureleoules:
reACK 3fcb545ab2
Tree-SHA512: 42f833e92f40c348007ca565a4c98039e6f1ff25d8322bc2b27115824744779baf0b0a38452e4e2cdcba45076473f1028079bbd0f670020481ec5d3db42e4731
eb679a7896 rpc: make `address` field optional (w0xlt)
Pull request description:
Close https://github.com/bitcoin/bitcoin/issues/26338.
This PR makes optional the `address` field in the response of `listtransactions` and `listsinceblock` RPC.
And adds two tests that fail on master, but not on this branch.
ACKs for top commit:
achow101:
ACK eb679a7896
aureleoules:
ACK eb679a7896
Tree-SHA512: b267439626e2ec3134ae790c849949a4c40ef0cebd20092e8187be3db0a61941b2da10bbbba92ca880b8369f46c1aaa806d057eaa5159325f65cbec7cb33c52f
da16893474 ci: Use `macos-ventura-xcode:14.1` image for "macOS native" task (Hennadii Stepanov)
702836530f ci: Make `getopt` path architecture agnostic (Hennadii Stepanov)
Pull request description:
The "macOS native" CI task always uses the recent OS image.
This PR updates it up to the recent macOS release.
Cirrus Labs [stopped](https://github.com/bitcoin/bitcoin/pull/25160#issuecomment-1162829773) updating macOS images for `x86_64`, therefore, an `arm64` image been used.
Also `make test-security-check` has been dropped as it ["isn't even expected to pass"](https://github.com/bitcoin/bitcoin/issues/26386#issuecomment-1290318628) on `arm64` in CI.
ACKs for top commit:
Sjors:
utACK da16893
Tree-SHA512: 36785d33b7f11b3cdbc53bcfbf97d88bf821fad248c825982dd9f8e3413809a4ef11190eaf950e60fdf479b62ff66920c35d9ea42d534723f015742eec7e19b6
b8b59ff9fe gui: update the screen after loading wallet (w0xlt)
Pull request description:
Currently, the user loads a wallet and the screen does not switch to the selected wallet after loading (File -> Open Wallet -> wallet name).
This PR changes that by making the `OpenWalletActivity::opened` signal connection a `Qt::QueuedConnection` type.
ACKs for top commit:
jarolrod:
ACK b8b59ff9fe
hebasto:
ACK b8b59ff9fe, tested on Ubuntu 22.04.
Tree-SHA512: 43cd755638b643f481014a7933a0af25df2d109e859cb5f878bc04e562950d550716fa38465140060e28526b2441688580cbcbe4ec6819566b4f95162ca5e527
0cc23fc603 Fix typo in comment SHA256->SHA512 (Elichai Turkel)
Pull request description:
The comment says it's the SHA-256 state, while it's actually the SHA-512 state
ACKs for top commit:
andrewtoth:
ACK 0cc23fc603
aureleoules:
ACK 0cc23fc603
Tree-SHA512: 4e390ceefb847d3bbe4f5caab390a4fdd14892fe443f58c32b08b3444fccd611cff22938c3dfa611dfd2497736f779fae4165497b4208e48aa8fc9d2236f943b
Goal 1:
Benchmark the transaction creation process for pre-selected-inputs only.
Setting `m_allow_other_inputs=false` to disallow the wallet to include coins automatically.
Goal 2:
Benchmark the transaction creation process for pre-selected-inputs and coin selection.
-----------------------
Benchmark Setup:
1) Generates a 5k blockchain, loading the wallet with 5k transactions with two outputs each.
2) Fetch 4 random UTXO from the wallet's available coins and pre-select them as inputs inside CoinControl.
Benchmark (Goal 1):
Call `CreateTransaction` providing the coin control, who has set `m_allow_other_inputs=false` and
the manually selected coins.
Benchmark (Goal 2):
Call `CreateTransaction` providing the coin control, who has set `m_allow_other_inputs=true` and
the manually selected coins.
we are already computing the preset inputs total amount inside `PreSelectedInputs::Insert`,
which internally decides whether to use the effective value or the raw output value based on
the 'subtract_fee_outputs' flag.
so if there is an error in any of the pre-set coins, we can fail right away
without computing the wallet available coins set (calling `AvailableCoins`)
which is a slow operation as it goes through the entire wallet's txes map.
----------------------
And to make the Coin Selection flow cleared, have decoupled SelectCoins in two functions:
1) AutomaticCoinSelection.
2) SelectCoins.
1) AutomaticCoinSelection:
Receives a set of coins and selects the best subset of them to
cover the target amount.
2) SelectCoins
In charge of select all the user manually selected coins first ("pre-set inputs"), and
if coin_control 'm_allow_other_inputs=true', call 'AutomaticCoinSelection' to select a
subset of coins owned by the wallet to cover for the target - preset_inputs.total_amount
remaining value.
First step towards decoupling the pre-selected-inputs fetching functionality
from `SelectCoins`. Which, will let us not waste resources calculating the
available coins if one of the pre-set inputs has an error.
(right now, if one of the pre-set inputs is invalid, we first walk through
the entire wallet txes map just to end up failing right after it finish)
No need to walk through the entire wallet's txes map just to get
coins that we could have gotten by just doing a simple map.find(out.hash).
(Which is what we are doing inside `SelectCoins` anyway)
no need to waste resources calculating the wallet available coins if
they are not going to be used.
The 'm_allow_other_inputs=true` default value change is to correct
an ugly misleading behavior:
The tx creation process was having a workaround patch to automatically
fall back to select coins from the wallet if `m_allow_other_inputs=false`
(previous default value) and no manual inputs were selected.
This could be seen in master in flows like `sendtoaddress`, `sendmany`
and even the GUI, where the `m_allow_other_inputs` value isn't customized
and the wallet still selects and adds coins to the tx internally.
f5ff3d773c rpc: add missing lock around chainman.ActiveTip() (Andrew Toth)
Pull request description:
https://github.com/bitcoin/bitcoin/pull/23927 seems to have missed a lock around `chainman.ActiveChain()`.
ACKs for top commit:
aureleoules:
ACK f5ff3d773c
Tree-SHA512: 3f116ca44c1b2bc0c7042698249ea3417dfb7c0bb81158a7ceecd087f1e02baa89948f9bb7924b1757798a1691a7de6e886aa72a0a9e227c13a3f512cc59d6c9
fa54d3011e test: check for false-positives in rpc_scanblocks.py (Sebastian Falbesoner)
3bca6cd61a test: add compact block filter (BIP158) helper routines (Sebastian Falbesoner)
25ee74dd11 test: add SipHash implementation for generic data in Python (Sebastian Falbesoner)
Pull request description:
This PR adds a fixed false-positive element check to the functional test rpc_scanblocks.py by using a pre-calculated scriptPubKey that collides with the regtest genesis block's coinbase output. Note that determining a BIP158 false-positive at runtime would also be possible, but take too long (we'd need to create and check ~800k output scripts on average, which took at least 2 minutes on average on my machine).
The introduced check is related to issue #26322 and more concretely inspired by PR #26325 which introduces an "accurate" mode that filters out these false-positives. The introduced cryptography routines (siphash for generic data) and helpers (BIP158 ranged hash calculation, relevant scriptPubKey per block determination) could potentially also be useful for more tests in the future that involve compact block filters.
ACKs for top commit:
achow101:
ACK fa54d3011e
Tree-SHA512: c6af50864146028228d197fb022ba2ff24d1ef48dc7d171bccfb21e62dd50ac80db5fae0c53f5d205edabd48b3493c7aa2040f628a223e68df086ec2243e5a93
5826bf546e test: Add test for getblockfrompeer on syncing pruned nodes (Fabian Jahr)
7fa851fba8 rpc: Pruned nodes can not fetch unsynced blocks (Fabian Jahr)
Pull request description:
This PR prevents `getblockfrompeer` from getting used on blocks that the node has not synced past yet if the node is in running in prune mode.
### Problem
While a node is still catching up to the tip that it is aware of via the headers, the user can currently use to fetch blocks close to or at the tip. These blocks are stored in the block/rev file that otherwise contains blocks the node is receiving as part of the syncing process.
This creates a problem for pruned nodes: The files containing a fetched block are not pruned during syncing because they contain a block close to the tip. This means the entire file (~130MB) will not be pruned until the tip has moved on far enough from the fetched block. In extreme cases with heavy pruning (like 550) and multiple blocks being fetched this could mean that the disc usage far exceeds what the user expects, potentially running out of space.
### Approach
There would be certainly other approaches that could fix the problem while still allowing the current behavior, but all of the ideas I came up with seemed like overkill for a niche problem on a new RPC where it's still unclear how and how much it will be used.
### Testing
So far I did not see a simple enough way to test this I am still looking into it and if it's complex will potentially add it in a follow-up. What would be needed is a way to have a node fetch headers but not sync the blocks yet, that seems like a pattern that could be generally useful.
To manually reproduce the problematic behavior:
1. Start a node with current `master` with `-prune=550` and an empty/new datadir, Testnet and Mainnet should both work.
2. While the node is syncing run `getblockfrompeer` on the current tip and a few other recent blocks.
3. Go to your datadir and observe the blocks folder: There should be a few full `blk*.dat` and `rev*.dat` files that are not being pruned. When you "pinned" a few of these files the blocks folder should be significantly above the target size of 550MB.
ACKs for top commit:
Sjors:
utACK 5826bf546e
achow101:
ACK 5826bf546e
aureleoules:
tACK 5826bf546e
Tree-SHA512: aa3f477ec755a9df2331c047cb10b3cd08292522bf6ad7a36a7ea36d7eba4894b84de8bd23003c9baea5ac0c53b77142c3c2819ae7528cece9d10a0d06c850d8
