Commit graph

23118 commits

Author SHA1 Message Date
furszy
61c2265629
wallet: group AvailableCoins filtering parameters in a single struct
Plus clean callers that use the params default values
2022-10-29 08:50:38 -03:00
furszy
f0f6a3577b
RPC: listunspent, add "include immature coinbase" flag
so we can return the immature coinbase UTXOs as well.
2022-10-29 08:45:12 -03:00
Andrew Chow
8781a1b6bb psbt: Include output pubkey in additional pubkeys to sign
In addition to the pubkeys in hd_keypaths and tap_bip32_keypaths, also
see if the descriptor can produce a SigningProvider for the output
pubkey.

Also slightly refactors this area to reduce code duplication.
2022-10-28 20:03:22 -04:00
Andrew Chow
323890d0d7 sign: Fill in taproot pubkey info for all script path sigs
Taproot pubkey info was not being added for multi_a signing. The filling
of this info is moved into the common function CreateTaprootScriptSig so
that any signing of taproot scripts will include the pubkey info.
2022-10-28 20:03:22 -04:00
Andrew Chow
8b050762b1
Merge bitcoin/bitcoin#26409: refactor: Silence GCC Wmissing-field-initializers in ChainstateManagerOpts
fa29ef00ad refactor: Silence GCC Wmissing-field-initializers in ChainstateManagerOpts (MacroFake)

Pull request description:

  The `std::optional` fields in the struct that fall back to chain param defaults if not provided should be initialized to `std::nullopt`. This already happens with the current code.

  However, for consistency with `check_block_index` and to silence a GCC warning, add the "missing" `{}`.

ACKs for top commit:
  achow101:
    ACK fa29ef00ad
  hebasto:
    ACK fa29ef00ad, tested on Ubuntu 22.04 + GCC 11.3.
  jonatack:
    ACK fa29ef00ad

Tree-SHA512: bdec9c56df5d601a5616e107fed48737b13b0a7242b6526092fb682b5016544a4bc08666b60304c668d44c6f7ac69d3788093d921382c1d6c577c1f9fe31fc50
2022-10-28 15:37:17 -04:00
MacroFake
1bad29fe02
Merge bitcoin/bitcoin#26377: test: Make system_tests/run_command test locale and platform agnostic
884304e6c6 test: Make `system_tests/run_command` locale agnostic (Hennadii Stepanov)

Pull request description:

  Fixes bitcoin/bitcoin#26368.

ACKs for top commit:
  Sjors:
    tACK 884304e6c6

Tree-SHA512: 76d4941e02b3b119dcf4dacbe60ef45a9dc8cf775bdb31b5291cd8147665285d41caaf1f5688abdfc9a47c393ddb535af7b11af839660d30ef30f1ca0d936133
2022-10-28 11:32:13 +02:00
MacroFake
fa29ef00ad
refactor: Silence GCC Wmissing-field-initializers in ChainstateManagerOpts 2022-10-28 11:08:20 +02:00
omahs
180eac0f73 Fix: typos
Fix: typos

Fix: typos

Fix: typos
2022-10-28 09:39:36 +02:00
Andrew Chow
f37bd15d47
Merge bitcoin/bitcoin#25685: wallet: Faster transaction creation by removing pre-set-inputs fetching responsibility from Coin Selection
3fcb545ab2 bench: benchmark transaction creation process (furszy)
a8a75346d7 wallet: SelectCoins, return early if target is covered by preset-inputs (furszy)
f41712a734 wallet: simplify preset inputs selection target check (furszy)
5baedc3351 wallet: remove fetch pre-selected-inputs responsibility from SelectCoins (furszy)
295852f619 wallet: encapsulate pre-selected-inputs lookup into its own function (furszy)
37e7887cb4 wallet: skip manually selected coins from 'AvailableCoins' result (furszy)
94c0766b0c wallet: skip available coins fetch if "other inputs" are disallowed (furszy)

Pull request description:

  #### # Context (Current Flow on Master)

  In the transaction creation process, in order to select which coins the new transaction will spend,
  we first obtain all the available coins known by the wallet, which means walking-through the
  wallet txes map, gathering the ones that fulfill certain spendability requirements in a vector.

  This coins vector is then provided to the Coin Selection process, which first checks if the user
  has manually selected any input (which could be internal, aka known by the wallet, or external),
  and if it does, it fetches them by searching each of them inside the wallet and/or inside the
  Coin Control external tx data.

  Then, after finding the pre-selected-inputs and gathering them in a vector, the Coin Selection
  process walks-through the entire available coins vector once more just to erase coins that are
  in both vectors. So the Coin Selection process doesn’t pick them twice (duplicate inputs inside
  the same transaction).

  #### # Process Workflow Changes

  Now, a new method, `FetchCoins` will be responsible for:
  1) Lookup the user pre-selected-inputs (which can be internal or external).
  2) And, fetch the available coins in the wallet (excluding the already fetched ones).

  Which will occur prior to the Coin Selection process. Which allows us to never include the
  pre-selected-inputs inside the available coins vector in the first place, as well as doing other
  nice improvements (written below).

  So, Coin Selection can perform its main responsibility without mixing it with having to fetch
  internal/external coins nor any slow and unneeded duplicate coins verification.

  #### # Summarizing the Improvements:

  1) If any pre-selected-input lookup fail, the process will return the error right away.
      (before, the wallet was fetching all the wallet available coins, walking through the
      entire txes map, and then failing for an invalid pre-selected-input inside SelectCoins)

  2) The pre-selected-inputs lookup failure causes are properly described on the return error.
      (before, we were returning an "Insufficient Funds" error for everything, even if the failure
      was due a not solvable external input)

  3) **Faster Coin Selection**: no longer need to "remove the pre-set inputs from the available coins
      vector so that Coin Selection doesn't pick them" (which meant to loop-over the entire
      available coins vector at Coin Selection time, erasing duplicate coins that were pre-selected).

      Now, the available coins vector, which is built after the pre-selected-inputs fetching,
      doesn’t include the already selected inputs in the first place.

  4) **Faster transaction creation** for transactions that only use manually selected inputs.

      We now will return early, as soon as we finish fetching the pre-selected-inputs and
      not perform the resources expensive calculation of walking-through the entire wallet
      txes map to obtain the available coins (coins that we will not use).

  ---------------------------

  Added a new bench (f6d0bb2) measuring the transaction creation process, for a wallet with ~250k UTXO, only using the pre-selected-inputs inside coin control. Setting `m_allow_other_inputs=false` to disallow the wallet to include coins automatically.

  #### Result on this PR (tip f6d0bb2d):

  |               ns/op |                op/s |    err% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------:|:----------
  |        1,048,675.00 |              953.58 |    0.3% |      0.06 | `WalletCreateTransaction`

  vs

  #### Result on master (tip 4a4289e2):

  |               ns/op |                op/s |    err% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------:|:----------
  |       96,373,458.20 |               10.38 |    0.2% |      5.30 | `WalletCreateTransaction`

  The benchmark took to run in master: **96.37 milliseconds**, while in this PR: **1 millisecond**  🚀 .

ACKs for top commit:
  S3RK:
    Code Review ACK 3fcb545ab2
  achow101:
    ACK 3fcb545ab2
  aureleoules:
    reACK 3fcb545ab2

Tree-SHA512: 42f833e92f40c348007ca565a4c98039e6f1ff25d8322bc2b27115824744779baf0b0a38452e4e2cdcba45076473f1028079bbd0f670020481ec5d3db42e4731
2022-10-27 17:48:58 -04:00
Andrew Chow
551c8e9526
Merge bitcoin/bitcoin#26349: rpc: make address field optional list{transactions, sinceblock} response
eb679a7896 rpc: make `address` field optional (w0xlt)

Pull request description:

  Close https://github.com/bitcoin/bitcoin/issues/26338.

  This PR makes optional the `address` field in the response of `listtransactions` and `listsinceblock` RPC.
  And adds two tests that fail on master, but not on this branch.

ACKs for top commit:
  achow101:
    ACK eb679a7896
  aureleoules:
    ACK eb679a7896

Tree-SHA512: b267439626e2ec3134ae790c849949a4c40ef0cebd20092e8187be3db0a61941b2da10bbbba92ca880b8369f46c1aaa806d057eaa5159325f65cbec7cb33c52f
2022-10-27 13:17:39 -04:00
MacroFake
fa24239a1c
net: Avoid SetTxRelay for feeler connections 2022-10-27 16:09:33 +02:00
Hennadii Stepanov
39710f5635
Merge bitcoin-core/gui#665: Switch to the selected wallet after loading
b8b59ff9fe gui: update the screen after loading wallet (w0xlt)

Pull request description:

  Currently, the user loads a wallet and the screen does not switch to the selected wallet after loading (File -> Open Wallet -> wallet name).

  This PR changes that by making the `OpenWalletActivity::opened` signal connection a `Qt::QueuedConnection` type.

ACKs for top commit:
  jarolrod:
    ACK b8b59ff9fe
  hebasto:
    ACK b8b59ff9fe, tested on Ubuntu 22.04.

Tree-SHA512: 43cd755638b643f481014a7933a0af25df2d109e859cb5f878bc04e562950d550716fa38465140060e28526b2441688580cbcbe4ec6819566b4f95162ca5e527
2022-10-27 13:56:42 +01:00
glozow
2242de16cc
Merge bitcoin/bitcoin#26394: Fix typo in comment SHA256->SHA512
0cc23fc603 Fix typo in comment SHA256->SHA512 (Elichai Turkel)

Pull request description:

  The comment says it's the SHA-256 state, while it's actually the SHA-512 state

ACKs for top commit:
  andrewtoth:
    ACK 0cc23fc603
  aureleoules:
    ACK 0cc23fc603

Tree-SHA512: 4e390ceefb847d3bbe4f5caab390a4fdd14892fe443f58c32b08b3444fccd611cff22938c3dfa611dfd2497736f779fae4165497b4208e48aa8fc9d2236f943b
2022-10-27 11:02:22 +01:00
furszy
3fcb545ab2
bench: benchmark transaction creation process
Goal 1:
Benchmark the transaction creation process for pre-selected-inputs only.
Setting `m_allow_other_inputs=false` to disallow the wallet to include coins automatically.

Goal 2:
Benchmark the transaction creation process for pre-selected-inputs and coin selection.

-----------------------

Benchmark Setup:
1) Generates a 5k blockchain, loading the wallet with 5k transactions with two outputs each.
2) Fetch 4 random UTXO from the wallet's available coins and pre-select them as inputs inside CoinControl.

Benchmark (Goal 1):
Call `CreateTransaction` providing the coin control, who has set `m_allow_other_inputs=false` and
the manually selected coins.

Benchmark (Goal 2):
Call `CreateTransaction` providing the coin control, who has set `m_allow_other_inputs=true` and
the manually selected coins.
2022-10-26 15:54:31 -03:00
furszy
a8a75346d7
wallet: SelectCoins, return early if target is covered by preset-inputs 2022-10-26 15:54:31 -03:00
furszy
f41712a734
wallet: simplify preset inputs selection target check
we are already computing the preset inputs total amount inside `PreSelectedInputs::Insert`,
which internally decides whether to use the effective value or the raw output value based on
the 'subtract_fee_outputs' flag.
2022-10-26 15:54:31 -03:00
furszy
5baedc3351
wallet: remove fetch pre-selected-inputs responsibility from SelectCoins
so if there is an error in any of the pre-set coins, we can fail right away
without computing the wallet available coins set (calling `AvailableCoins`)
which is a slow operation as it goes through the entire wallet's txes map.

----------------------

And to make the Coin Selection flow cleared, have decoupled SelectCoins in two functions:

1) AutomaticCoinSelection.
2) SelectCoins.

1) AutomaticCoinSelection:
   Receives a set of coins and selects the best subset of them to
   cover the target amount.

2) SelectCoins
   In charge of select all the user manually selected coins first ("pre-set inputs"), and
   if coin_control 'm_allow_other_inputs=true', call 'AutomaticCoinSelection' to select a
   subset of coins owned by the wallet to cover for the target - preset_inputs.total_amount
   remaining value.
2022-10-26 15:54:31 -03:00
furszy
295852f619
wallet: encapsulate pre-selected-inputs lookup into its own function
First step towards decoupling the pre-selected-inputs fetching functionality
from `SelectCoins`. Which, will let us not waste resources calculating the
available coins if one of the pre-set inputs has an error.

(right now, if one of the pre-set inputs is invalid, we first walk through
the entire wallet txes map just to end up failing right after it finish)
2022-10-26 15:52:35 -03:00
furszy
37e7887cb4
wallet: skip manually selected coins from 'AvailableCoins' result
No need to walk through the entire wallet's txes map just to get
coins that we could have gotten by just doing a simple map.find(out.hash).
(Which is what we are doing inside `SelectCoins` anyway)
2022-10-26 15:52:35 -03:00
furszy
94c0766b0c
wallet: skip available coins fetch if "other inputs" are disallowed
no need to waste resources calculating the wallet available coins if
they are not going to be used.

The 'm_allow_other_inputs=true` default value change is to correct
an ugly misleading behavior:

The tx creation process was having a workaround patch to automatically
fall back to select coins from the wallet if `m_allow_other_inputs=false`
(previous default value) and no manual inputs were selected.

This could be seen in master in flows like `sendtoaddress`, `sendmany`
and even the GUI, where the `m_allow_other_inputs` value isn't customized
and the wallet still selects and adds coins to the tx internally.
2022-10-26 15:47:51 -03:00
Andrew Toth
f5ff3d773c rpc: add missing lock around chainman.ActiveTip() 2022-10-26 11:46:39 -04:00
Andrew Chow
88502ecf08
Merge bitcoin/bitcoin#23927: rpc: Pruning nodes can not fetch blocks before syncing past their height
5826bf546e test: Add test for getblockfrompeer on syncing pruned nodes (Fabian Jahr)
7fa851fba8 rpc: Pruned nodes can not fetch unsynced blocks (Fabian Jahr)

Pull request description:

  This PR prevents `getblockfrompeer` from getting used on blocks that the node has not synced past yet if the node is in running in prune mode.

  ### Problem

  While a node is still catching up to the tip that it is aware of via the headers, the user can currently use  to fetch blocks close to or at the tip. These blocks are stored in the block/rev file that otherwise contains blocks the node is receiving as part of the syncing process.

  This creates a problem for pruned nodes: The files containing a fetched block are not pruned during syncing because they contain a block close to the tip. This means the entire file (~130MB) will not be pruned until the tip has moved on far enough from the fetched block. In extreme cases with heavy pruning (like 550) and multiple blocks being fetched this could mean that the disc usage far exceeds what the user expects, potentially running out of space.

  ### Approach

  There would be certainly other approaches that could fix the problem while still allowing the current behavior, but all of the ideas I came up with seemed like overkill for a niche problem on a new RPC where it's still unclear how and how much it will be used.

  ### Testing

  So far I did not see a simple enough way to test this I am still looking into it and if it's complex will potentially add it in a follow-up. What would be needed is a way to have a node fetch headers but not sync the blocks yet, that seems like a pattern that could be generally useful.

  To manually reproduce the problematic behavior:
  1. Start a node with current `master` with `-prune=550` and an empty/new datadir, Testnet and Mainnet should both work.
  2. While the node is syncing run `getblockfrompeer` on the current tip and a few other recent blocks.
  3. Go to your datadir and observe the blocks folder: There should be a few full `blk*.dat` and `rev*.dat` files that are not being pruned. When you "pinned" a few of these files the blocks folder should be significantly above the target size of 550MB.

ACKs for top commit:
  Sjors:
    utACK 5826bf546e
  achow101:
    ACK 5826bf546e
  aureleoules:
    tACK 5826bf546e

Tree-SHA512: aa3f477ec755a9df2331c047cb10b3cd08292522bf6ad7a36a7ea36d7eba4894b84de8bd23003c9baea5ac0c53b77142c3c2819ae7528cece9d10a0d06c850d8
2022-10-26 11:27:31 -04:00
Andrew Chow
48af307481
Merge bitcoin/bitcoin#25957: wallet: fast rescan with BIP157 block filters for descriptor wallets
0582932260 test: add test for fast rescan using block filters (top-up detection) (Sebastian Falbesoner)
ca48a4694f rpc: doc: mention rescan speedup using `blockfilterindex=1` in affected wallet RPCs (Sebastian Falbesoner)
3449880b49 wallet: fast rescan: show log message for every non-skipped block (Sebastian Falbesoner)
935c6c4b23 wallet: take use of `FastWalletRescanFilter` (Sebastian Falbesoner)
70b3513904 wallet: add `FastWalletRescanFilter` class for speeding up rescans (Sebastian Falbesoner)
c051026586 wallet: add method for retrieving the end range for a ScriptPubKeyMan (Sebastian Falbesoner)
845279132b wallet: support fetching scriptPubKeys with minimum descriptor range index (Sebastian Falbesoner)
088e38d3bb add chain interface methods for using BIP 157 block filters (Sebastian Falbesoner)

Pull request description:

  ## Description

  This PR is another take of using BIP 157 block filters (enabled by `-blockfilterindex=1`) for faster wallet rescans and is a modern revival of #15845. For reviewers new to this topic I can highly recommend to read the corresponding PR review club (https://bitcoincore.reviews/15845).

  The basic idea is to skip blocks for deeper inspection (i.e. looking at every single tx for matches) if our block filter doesn't match any of the block's spent or created UTXOs are relevant for our wallet. Note that there can be false-positives (see https://bitcoincore.reviews/15845#l-199 for a PR review club discussion about false-positive rates), but no false-negatives, i.e. it is safe to skip blocks if the filter doesn't match; if the filter *does* match even though there are no wallet-relevant txs in the block, no harm is done, only a little more time is spent extra.

  In contrast to #15845, this solution only supports descriptor wallets, which are way more widespread now than back in the time >3 years ago. With that approach, we don't have to ever derive the relevant scriptPubKeys ourselves from keys before populating the filter, and can instead shift the full responsibility to that to the `DescriptorScriptPubKeyMan` which already takes care of that automatically. Compared to legacy wallets, the `IsMine` logic for descriptor wallets is as trivial as checking if a scriptPubKey is included in the ScriptPubKeyMan's set of scriptPubKeys (`m_map_script_pub_keys`): e191fac4f3/src/wallet/scriptpubkeyman.cpp (L1703-L1710)

  One of the unaddressed issues of #15845 was that [the filter was only created once outside the loop](https://github.com/bitcoin/bitcoin/pull/15845#discussion_r343265997) and as such didn't take into account possible top-ups that have happened. This is solved here by keeping a state of ranged `DescriptorScriptPubKeyMan`'s descriptor end ranges and check at each iteration whether that range has increased since last time. If yes, we update the filter with all scriptPubKeys that have been added since the last filter update with a range index equal or higher than the last end range. Note that finding new scriptPubKeys could be made more efficient than linearly iterating through the whole `m_script_pub_keys` map (e.g. by introducing a bidirectional map), but this would mean introducing additional complexity and state and it's probably not worth it at this time, considering that the performance gain is already significant.

  Output scripts from non-ranged `DescriptorScriptPubKeyMan`s (i.e. ones with a fixed set of output scripts that is never extended) are added only once when the filter is created first.

  ## Benchmark results

  Obviously, the speed-up indirectly correlates with the wallet tx frequency in the scanned range: the more blocks contain wallet-related transactions, the less blocks can be skipped due to block filter detection.

  In a [simple benchmark](https://github.com/theStack/bitcoin/blob/fast_rescan_functional_test_benchmark/test/functional/pr25957_benchmark.py), a regtest chain with 1008 blocks (corresponding to 1 week) is mined with 20000 scriptPubKeys contained (25 txs * 800 outputs) each. The blocks each have a weight of ~2500000 WUs and hence are about 62.5% full. A global constant `WALLET_TX_BLOCK_FREQUENCY` defines how often wallet-related txs are included in a block. The created descriptor wallet (default setting of `keypool=1000`, we have 8*1000 = 8000 scriptPubKeys at the start) is backuped via the `backupwallet` RPC before the mining starts and imported via `restorewallet` RPC after. The measured time for taking this import process (which involves a rescan) once with block filters (`-blockfilterindex=1`) and once without block filters (`-blockfilterindex=0`) yield the relevant result numbers for the benchmark.

  The following table lists the results, sorted from worst-case (all blocks contain wallte-relevant txs, 0% can be skipped) to best-case (no blocks contain walltet-relevant txs, 100% can be skipped) where the frequencies have been picked arbitrarily:

  wallet-related tx frequency; 1 tx per...    | ratio of irrelevant blocks  | w/o filters | with filters | speed gain
  --------------------------------------------|-----------------------------|-------------|--------------|-------------
  ~ 10 minutes (every block)                  |              0%             |   56.806s   |   63.554s    |  ~0.9x
  ~ 20 minutes (every 2nd block)              |           50% (1/2)         |   58.896s   |   36.076s    |  ~1.6x
  ~ 30 minutes (every 3rd block)              |          66.67% (2/3)       |   56.781s   |   25.430s    |  ~2.2x
  ~ 1 hour (every 6th block)                  |          83.33% (5/6)       |   58.193s   |   15.786s    |  ~3.7x
  ~ 6 hours (every 36th block)                |          97.22% (35/36)     |   57.500s   |    6.935s    |  ~8.3x
  ~ 1 day (every 144th block)                 |         99.31% (143/144)    |   68.881s   |    6.107s    | ~11.3x
    (no txs)                                  |              100%           |   58.529s   |    5.630s    | ~10.4x

  Since even the (rather unrealistic) worst-case scenario of having wallet-related txs in _every_ block of the rescan range obviously doesn't take significantly longer, I'd argue it's reasonable to always take advantage of block filters if they are available and there's no need to provide an option for the user.

  Feedback about the general approach (but also about details like naming, where I struggled a lot) would be greatly appreciated. Thanks fly out to furszy for discussing this subject and patiently answering basic question about descriptor wallets!

ACKs for top commit:
  achow101:
    ACK 0582932260
  Sjors:
    re-utACK 0582932260
  aureleoules:
    ACK 0582932260 - minor changes, documentation and updated test since last review
  w0xlt:
    re-ACK 0582932260

Tree-SHA512: 3289ba6e4572726e915d19f3e8b251d12a4cec8c96d041589956c484b5575e3708b14f6e1e121b05fe98aff1c8724de4564a5a9123f876967d33343cbef242e1
2022-10-26 11:19:19 -04:00
Elichai Turkel
0cc23fc603
Fix typo in comment SHA256->SHA512 2022-10-26 15:55:29 +03:00
dergoegge
784b023191 [net processing] Simplify use of IsContinuationOfLowWorkHeadersSync in TryLowWorkHeaderSync
`m_headers_sync` is already reset in IsContinuationOfLowWorkHeadersSync
if there is a failure, so there is no need to also reset in
TryLowWorkHeaderSync.
2022-10-26 11:12:03 +01:00
MacroFake
a1fff275e7
Merge bitcoin/bitcoin#25704: refactor: Remove almost all validation option globals
aaaa7bd0ba iwyu: Add missing includes (MacroFake)
fa9ebec096 Remove g_parallel_script_checks (MacroFake)
fa7c834b9f Move ::fCheckBlockIndex into ChainstateManager (MacroFake)
fa43188d86 Move ::fCheckpointsEnabled into ChainstateManager (MacroFake)
cccca83099 Move ::nMinimumChainWork into ChainstateManager (MacroFake)
fa29d0b57c Move ::hashAssumeValid into ChainstateManager (MacroFake)
faf44876db Move ::nMaxTipAge into ChainstateManager (MacroFake)

Pull request description:

  It seems preferable to assign globals to a class (in this case `ChainstateManager`), than to leave them dangling. This should clarify scope for code-readers, as well as clarifying unit test behaviour.

ACKs for top commit:
  dergoegge:
    Code review ACK aaaa7bd0ba
  ryanofsky:
    Code review ACK aaaa7bd0ba. No changes since last review, other than rebase
  aureleoules:
    reACK aaaa7bd0ba

Tree-SHA512: 83ec3ba0fb4f1dad95810d4bd4e578454e0718dc1bdd3a794cc4e48aa819b6f5dad4ac4edab3719bdfd5f89cbe23c2740a50fd56c1ff81c99e521c5f6d4e898d
2022-10-26 11:41:57 +02:00
MacroFake
cf288377c0
Merge bitcoin/bitcoin#26275: Fix crash on deriveaddresses when index is 2147483647 (2^31-1)
9153ff3e27 rpc: add non-regression test about deriveaddresses crash when index is 2147483647 (muxator)
addf9d6502 rpc: fix crash in deriveaddresses when derivation index is 2147483647 (muxator)

Pull request description:

  This PR is a proposal for fixing #26274 (better described there).

  The problem is due to a signed int wrapping when the `index` parameter of the `deriveaddresses` RPC call has the value `2^31-1`.

  ```C++
  for (int i = range_begin; i <= range_end; ++i) {
  ```

  * the first commit adds a "temporary" test case (`test/functional/rpc_deriveaddresses_crash.py`) that shows the crash, and can be used to generate a core dump;
  * the second commit fixes the problem giving an explicit size to the `i` variable in a for loop, from `int` to `int64_t`. The same commit also removes the ephemeral test case and adds a passing test to `test/functional/rpc_deriveaddresses.py`, in order to prevent future regressions.

  This is my first submission to this project and I do not know its conventions. Please advise if something needs to be changed.

ACKs for top commit:
  achow101:
    ACK 9153ff3e27

Tree-SHA512: 0477b57b15dc2c682cf539d6002f100d44a8c7e668041aa3340c39dcdbd40e083c75dec6896b6c076b044a01c2e5254272ae6696d8a1467539391926f270940a
2022-10-26 10:12:27 +02:00
w0xlt
eb679a7896 rpc: make address field optional 2022-10-26 01:18:28 -03:00
fanquake
28cf756971
Merge bitcoin/bitcoin#23578: Add external signer taproot support
796b020c37 wallet: add taproot support to external signer (Sjors Provoost)

Pull request description:

  Builds on #22558 (merged on 2022-06-28).

  [HWI 2.1.0](https://github.com/bitcoin-core/HWI/releases/tag/2.1.0) or newer is required to import and use taproot descriptors. Older versions will work, but won't import a taproot descriptor.

  Tested with HWI 2.1.1:
  * Trezor T (firmware v2.5.1) on Signet: signs, change detection works
  * Ledger Nano S (firmware 2.1.0, Bitcoin app 2.0.6): signs, change detection works

  Only the most basic `tr(key)` descriptor is supported, script path spending is completely untested (if it works at all).

ACKs for top commit:
  jb55:
    utACK 796b020c37
  achow101:
    ACK 796b020c37

Tree-SHA512: 6dcb7eeb45421a3bbf2bdabeacd29979867db69077d7bf192bb77faa4bfefe446487b8df07bc40f9457009a88e598bdc09f769e6106fed2833ace7ef205a157a
2022-10-26 11:10:23 +08:00
Sebastian Falbesoner
ca48a4694f rpc: doc: mention rescan speedup using blockfilterindex=1 in affected wallet RPCs 2022-10-25 15:57:39 +02:00
Sebastian Falbesoner
3449880b49 wallet: fast rescan: show log message for every non-skipped block
For that purpose, a new logging category BCLog::SCAN is introduced.
2022-10-25 15:57:38 +02:00
Sebastian Falbesoner
935c6c4b23 wallet: take use of FastWalletRescanFilter
Can be reviewed with `--ignore-all-space`.
2022-10-25 15:57:38 +02:00
Sebastian Falbesoner
70b3513904 wallet: add FastWalletRescanFilter class for speeding up rescans
This only supports wallet descriptors right now.
2022-10-25 15:57:38 +02:00
Sebastian Falbesoner
c051026586 wallet: add method for retrieving the end range for a ScriptPubKeyMan 2022-10-25 15:57:38 +02:00
Sebastian Falbesoner
845279132b wallet: support fetching scriptPubKeys with minimum descriptor range index
This extra method will be needed for updating the filter set for
faster wallet rescans; after an internal top-up has happened, we only
want to add the newly created scriptPubKeys.
2022-10-25 15:57:38 +02:00
Sebastian Falbesoner
088e38d3bb add chain interface methods for using BIP 157 block filters
This is useful for speeding up wallet rescans and is based on an
earlier version from PR #15845 ("wallet: Fast rescan with BIP157 block
filters"), which was never merged.

Co-authored-by: MacroFake <falke.marco@gmail.com>
2022-10-25 15:57:28 +02:00
dergoegge
e891aabf5a [net processing] Fixup TryLowWorkHeadersSync comment 2022-10-24 22:05:59 +01:00
Larry Ruane
db929893ef Faster -reindex by initially deserializing only headers
When a block is initially read from a blk*.dat file during reindexing,
it can be added to the block index only if all of its ancestor blocks
have been added, which is rare. If the block's ancestors have not been
added, the block must be re-read from disk later when it can be added.

This commit: During the initial block read, deserialize only its header,
rather than the entire block, since this is sufficient to determine
if its parent (and thus all its ancestors) has been added. This is a
performance improvement.
2022-10-24 13:02:37 -06:00
Larry Ruane
c72de9990a util: add CBufferedFile::SkipTo() to move ahead in the stream
SkipTo() reads data from the file into the CBufferedFile object
(memory), but, unlike this object's read() method, SkipTo() doesn't
transfer data into a caller's memory buffer. This is useful because
after skipping forward in the stream in this way, the user can, if
needed, rewind the stream (SetPos()) and access the object's memory
buffer including ranges that were skipped over (without needing to
read from the disk file).
2022-10-24 13:02:37 -06:00
Larry Ruane
48a68908ba Add LoadExternalBlockFile() benchmark 2022-10-24 13:02:35 -06:00
glozow
3d0fca1288
Merge bitcoin/bitcoin#26355: p2p: Handle IsContinuationOfLowWorkHeadersSync return value correctly when new headers sync is started
7ad15d1100 [net processing] Handle IsContinuationOfLowWorkHeadersSync return value correctly when new headers sync is started (dergoegge)

Pull request description:

  This PR fixes a bug in the headers sync logic that enables submitting headers to a nodes block index that don't lead to a chain that surpasses our DoS limit.

  The issue is that we ignore the return value on [the first `IsContinuationOfLowWorkHeadersSync` call after a new headers sync is started](fabc031048/src/net_processing.cpp (L2553-L2568)), which leads to us passing headers to [`ProcessNewBlockHeaders`](fabc031048/src/net_processing.cpp (L2856)) when that initial `IsContinuationOfLowWorkHeadersSync` call returns `false`. One easy way (maybe the only?) to trigger this is by sending 2000 headers where the last header has a different `nBits` value than the prior headers (which fails the pre-sync logic [here](fabc031048/src/headerssync.cpp (L189))). Those 2000 headers will be passed to `ProcessNewBlockHeaders`.

  I haven't included a test here so far because we can't test this without changing the default value for `CRegTestParams::consensus.fPowAllowMinDifficultyBlocks` or doing some more involved refactoring.

ACKs for top commit:
  sipa:
    ACK 7ad15d1100
  glozow:
    ACK 7ad15d1100

Tree-SHA512: 9aabb8bf3700401e79863d0accda0befd2a83c4d469a53f97d827e51139e2f826aee08cdfbc8866b311b153f61fdac9b7aa515fcfa2a21c5e2812c2bf3c03664
2022-10-24 15:38:37 +01:00
Hennadii Stepanov
884304e6c6
test: Make system_tests/run_command locale agnostic 2022-10-24 13:36:04 +01:00
MacroFake
fa2d01470a
test: Use type-safe NodeSeconds for TestMemPoolEntryHelper 2022-10-24 11:33:33 +02:00
MacroFake
3db23fd821
Merge bitcoin-core/gui#676: Update peers window "Transaction Relay" label and tooltip
a079103c94 gui: update peers window "Transaction Relay" label and tooltip (Jon Atack)

Pull request description:

  to current v24.0 p2p behavior.  Similar updates have been made to RPC getpeerinfo and CLI -netinfo.

Top commit has no ACKs.

Tree-SHA512: 400a794f655f799eefcb77c479fef6bcd3f81aede2af54a4a9bcb7c0c783e2e3f18bc5fd2484a79e8c30af279747a05fc0ebb69dbc47375d4c55b16ceba97b99
2022-10-24 10:42:47 +02:00
MacroFake
8fb3fd2ba4
Merge bitcoin-core/gui#673: Use fallback value for Version and User Agent during peer connection
c2a21c0670 gui: use fallback value for Version and User Agent during peer connection (Jon Atack)

Pull request description:

  During connection setup for a peer, getpeerinfo returns `"version": 0, "subver": ""` and the GUI Peers window displays 0 and an empty field, respectively.

  Give these fields the same behavior as the other fields in the GUI Peers window: display the fallback value in `src/qt/forms/debugwindow.ui` (i.e. `N/A`) until a valid result is available after the peer connection completes.

  An alternative would be to display nothing for both, as is the case currently for User Agent.

ACKs for top commit:
  jarolrod:
    ACK c2a21c0670
  furszy:
    code ACK c2a21c06

Tree-SHA512: 4f0060fa9abde120a2bb48c9dcc87894d9bb70c33e6ab43b22400a4bcd0ceff0fa098adf7f385b0a7a4cf5d7053463b36fe1232e19a8d5025eecd8db9833f73b
2022-10-24 10:40:05 +02:00
fanquake
50cc8ef5a7
Merge bitcoin/bitcoin#26302: refactor: Use type-safe time point for CWallet::m_next_resend
fa51cc9651 refactor: Use type-safe time point for CWallet::m_next_resend (MacroFake)

Pull request description:

  `GetTime` is not type-safe, thus deprecated, see 75cbbfa279/src/util/time.h (L62-L70)

ACKs for top commit:
  shaavan:
    Code Review ACK fa51cc9651
  aureleoules:
    ACK fa51cc9651

Tree-SHA512: 030de10070518580763ea75079442e2f934c54d3083be3ebe35e7f1bc6db2096745bb46d95aa1e6efe29ced30a048acfe5cd999178e6787b7647dfbec5ecb444
2022-10-24 10:11:13 +08:00
Hennadii Stepanov
45a0f4e014
Update minisketch subtree to latest upstream 2022-10-23 15:03:04 +01:00
Hennadii Stepanov
e9f1d8c272 Squashed 'src/minisketch/' changes from 47f0a2d26f..a571ba20f9
a571ba20f9 Merge sipa/minisketch#68: Add missed `#include <string>`
b9a7f7e2bc Merge sipa/minisketch#69: refactor: Drop unused `total` local variables
8a5af94edc Merge sipa/minisketch#70: build: Remove `-Qunused-arguments` workaround for clang + ccache
c36f1f03a3 Merge sipa/minisketch#72: Fix MSVC implementation of `CountBits()` function
0078bedda6 Ignore `HAVE_CLZ` macro when building with MSVC
1c772918c4 Fix MSVC implementation of `CountBits()` function
98f87c55f4 build: Remove `-Qunused-arguments` workaround for clang + ccache
11a1e25c81 refactor: Drop unused `total` local variables
ed6c8fcfd9 Add missed `#include <string>`

git-subtree-dir: src/minisketch
git-subtree-split: a571ba20f9dd1accab6a2309d066369878042ca6
2022-10-23 15:00:20 +01:00
dergoegge
7ad15d1100 [net processing] Handle IsContinuationOfLowWorkHeadersSync return value correctly when new headers sync is started 2022-10-21 11:05:34 +01:00
MacroFake
8c5c98db47
Merge bitcoin/bitcoin#26248: net: Set relay in version msg to peers with relay permission in -blocksonly mode
dddd1acf58 net: Set relay in version msg to peers with relay permission (MacroFake)

Pull request description:

  Seems odd to set the `relay` permission in -blocksonly mode and also ask the peer not to relay transactions.

ACKs for top commit:
  dergoegge:
    ACK dddd1acf58
  naumenkogs:
    ACK dddd1acf58
  mzumsande:
    ACK dddd1acf58

Tree-SHA512: 7bb0e964993ea4982747ae2801fe963ff88586e2ded03015b60ab83172b5b61f2d50e9cde9d7711b7ab207f8639467ecafc4d011ea151ec6c82c722f510f4df7
2022-10-21 11:18:48 +02:00