ee03c782ba wallet: Make GetOldestKeyPoolTime return nullopt for blank wallets (Hennadii Stepanov)
3e4f069d23 wallet, refactor: Make GetOldestKeyPoolTime return type std::optional (Hennadii Stepanov)
Pull request description:
The "keypoololdest" field in the `getwalletinfo` RPC response should be used for legacy wallets only.
Th current implementation (04437ee721) assumes that `CWallet::GetOldestKeyPoolTime()` always return `0` for descriptor wallets. This assumption is wrong for _blank_ descriptor wallets, when `m_spk_managers` is empty. As a result:
```
$ src/bitcoin-cli -signet -rpcwallet=211024-d-DPK getwalletinfo
{
"walletname": "211024-d-DPK",
"walletversion": 169900,
"format": "sqlite",
"balance": 0.00000000,
"unconfirmed_balance": 0.00000000,
"immature_balance": 0.00000000,
"txcount": 0,
"keypoololdest": 9223372036854775807,
"keypoolsize": 0,
"keypoolsize_hd_internal": 0,
"paytxfee": 0.00000000,
"private_keys_enabled": false,
"avoid_reuse": false,
"scanning": false,
"descriptors": true
}
```
This PR fixes this issue with direct checking of the `WALLET_FLAG_DESCRIPTORS` flag.
ACKs for top commit:
lsilva01:
re-ACK ee03c78
stratospher:
ACK ee03c78.
meshcollider:
Code review ACK ee03c782ba
Tree-SHA512: 9852f9f8ed5c08c07507274d7714f039bbfda66da6df65cf98f67bf11a600167d0f7f872680c95775399477f4df9ba9fce80ec0cbe0adb7f2bb33c3bd65b15df
Current CWalletTx state representation makes it possible to set
inconsistent states that won't be handled correctly by wallet sync code
or serialized & deserialized back into the same form.
For example, it is possible to call setConflicted without setting a
conflicting block hash, or setConfirmed with no transaction index. And
it's possible update individual m_confirm and fInMempool data fields
without setting an overall consistent state that can be serialized and
handled correctly.
Fix this without changing behavior by using std::variant, instead of an
enum and collection of fields, to represent sync state, so state
tracking code is safer and more legible.
This is a first step to fixing state tracking bugs
https://github.com/bitcoin-core/bitcoin-devwiki/wiki/Wallet-Transaction-Conflict-Tracking,
by adding an extra margin of safety that can prevent new bugs from being
introduced as existing bugs are fixed.
fa93ef5a8a refactor: Take Span in SetSeed (MarcoFalke)
Pull request description:
This makes calling code less verbose and less fragile. Also, by adding
the CKey::data() member function, it is now possible to call HexStr()
with a CKey object.
ACKs for top commit:
sipa:
utACK fa93ef5a8a
laanwj:
Code review ACK fa93ef5a8a
theStack:
Code-review ACK fa93ef5a8a
Tree-SHA512: 73fb999320719ad4b9ab5544018a7a083d140545c2807ee3582ecf7f441040a30b5157e85790b6b840af82f002a7faf30bd8162ebba5caaf2067391c43dc7e25
11115169a1 ci: Build fuzz with libsqlite3-dev (MarcoFalke)
fa7c6efca6 fuzz: Add wallet fuzz test (MarcoFalke)
fa59d2ce5b refactor: Use local args instead of global gArgs in CWallet::Create (MarcoFalke)
fadb44606f build: Inline FUZZ_SUITE_LDFLAGS_COMMON (MarcoFalke)
Pull request description:
Initial sketch to fuzz descriptor wallets. Can be improved in the future.
ACKs for top commit:
mjdietzx:
Code review ACK 1111516
Tree-SHA512: b1d2f24504d1ed5f3c6a031210f04c27c13d4e15576c4acbf50ded37ac45f7b7a5c7553e91d81d4a06e9ea73b3d745a552218d3ef3b2932fa5325a8331b0d3fd
This makes calling code less verbose and less fragile. Also, by adding
the CKey::data() member function, it is now possible to call HexStr()
with a CKey object.
54011e7aa2 refactor: use CWallet const shared pointers when possible (Karl-Johan Alm)
96461989a2 refactor: const shared_ptrs (Karl-Johan Alm)
Pull request description:
```C++
const std::shared_ptr<CWallet> wallet = x;
```
means we can not do `wallet = y`, but we can totally do `wallet->DestructiveOperation()`, contrary to what that line looks like.
This PR
* introduces a new convention: always use const shared pointers to `CWallet`s (even when we mutate the pointed-to thing)
* uses `const shared_ptr<const CWallet>` everywhere where wallets are not modified
In the future, this should preferably apply to all shared pointers, not limited to just `CWallet`s.
Both of these serve the same purpose: to dispell the misconception that `const shared_ptr<X>` immutates `X`. It doesn't, and it's dangerous to leave this misconception as is, for obvious reasons.
ACKs for top commit:
theStack:
re-ACK 54011e7aa2
Tree-SHA512: 3bf4062fc821751be30770c6b4ead10a016847970f155a0a5156f304347d221b9830840030c2fbfba8cd1e282f4eda45f5b4107fe6df8138afdcb6c2e95a2836
Introduce convention to use const shared pointers everywhere, unless the shared pointer is modified at some point, which it very rarely is.
We want this convention, as it helps alleviate the misconception that a const shared pointer somehow results in a pointer to an immutable object, which is false.
Follow-up to:
* commit 700c42b85d, which replaced pIndex
with block_hash in AddToWalletIfInvolvingMe.
* commit 9700fcb47f, which replaced
posInBlock with confirm.nIndex.
In the method `CWallet::LoadActiveScriptPubKeyMan`, the map
`external_spk_managers` (or `internal_spk_managers`, if parameter
`internal` is false) is accessed via std::map::operator[], which means
that a default-ctored entry is created with a null-pointer as value, if
the key doesn't exist. As soon as this value is dereferenced, a
segmentation fault occurs, e.g. in `CWallet::KeypoolCountExternalKeys`.
The bevaviour can be reproduced by the following steps (starting with empty regtest datadir):
$ ./src/bitcoind -regtest -daemon
$ ./src/bitcoin-cli -regtest -named createwallet_name=wallet descriptors=true blank=true
$ cat regtest-descriptors.txt
[
{
"desc": "tr([e4445899/49'/1'/0']tprv8ZgxMBicQKsPd8jCeBWsYLEoWxbVgzJDatJ7XkwQ6G3uF4FsHuaziHQ5JZAW4K515nj6kVVwPaNWZSMEcR7aFCwL4tQqTcaoprMKTTtm6Zg/1/*)#mr3llm7f",
"timestamp": 1634652324,
"active": true,
"internal": true,
"range": [
0,
999
],
"next": 0
}
]
$ ./src/bitcoin-cli -regtest importdescriptors "$(cat regtest-descriptors.txt)"
[
{
"success": true
}
]
$ ./src/bitcoin-cli -regtest getwalletinfo
error: timeout on transient error: Could not connect to the server 127.0.0.1:18443 (error code 1 - "EOF reached")
Bug reported by Josef Vondrlik (josef-v).
a0efe529e4 Fix outdated comments referring to ::ChainActive() (Samuel Dobson)
Pull request description:
After #21866 there are a few outdated comments referring to `::ChainActive()`, which should instead refer to `ChainstateManager::ActiveChain()`.
ACKs for top commit:
jamesob:
ACK a0efe529e4
Tree-SHA512: 80da19c105ed29ac247e6df4c8e916c3bf3f37230b63f07302114eef9c115add673e9649f0bbe237295be0c6da7b1030b5b93e14daf6768f17ce5de7cf2c9ff2
There is no change in behavior. This just helps prepare for the
transition from boost::filesystem to std::filesystem by avoiding calls
to methods which will be unsafe after the transaction to std::filesystem
to due lack of a boost::filesystem::path::imbue equivalent and inability
to set a predictable locale.
Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
Co-authored-by: Kiminuo <kiminuo@protonmail.com>
Co-authored-by: MarcoFalke <falke.marco@gmail.com>
9d0379cea6 consensus: use <cstdint> over <stdint.h> in amount.h (fanquake)
863e52fe63 consensus: make COIN & MAX_MONEY constexpr (fanquake)
d09071da5b [MOVEONLY] consensus: move amount.h into consensus (fanquake)
Pull request description:
A first step (of a few) towards some source code reorganization, as well as making libbitcoinconsensus slightly more self contained.
Related to #15732.
ACKs for top commit:
MarcoFalke:
concept ACK 9d0379cea6 🏝
Tree-SHA512: 97fc79262dcb8c00996852a288fee69ddf8398ae2c95700bba5b326f1f38ffcfaf8fa66e29d0cb446d9b3f4e608a96525fae0c2ad9cd531ad98ad2a4a687cd6a
928af61cdb allow send rpc take external inputs and solving data (Andrew Chow)
e39b5a5e7a Tests for funding with external inputs (Andrew Chow)
38f5642ccc allow fundtx rpcs to work with external inputs (Andrew Chow)
d5cfb864ae Allow Coin Selection be able to take external inputs (Andrew Chow)
a00eb388e8 Allow CInputCoin to also be constructed with COutPoint and CTxOut (Andrew Chow)
Pull request description:
Currently `fundrawtransaction` and `walletcreatefundedpsbt` both do not allow external inputs as the wallet does not have the information necessary to estimate their fees.
This PR adds an additional argument to both those RPCs which allows the user to specify solving data. This way, the wallet can use that solving data to estimate the size of those inputs. The solving data can be public keys, scripts, or descriptors.
ACKs for top commit:
prayank23:
reACK 928af61cdb
meshcollider:
Re-utACK 928af61cdb
instagibbs:
crACK 928af61cdb
yanmaani:
utACK 928af61.
Tree-SHA512: bc7a6ef8961a7f4971ea5985d75e2d6dc50c2a90b44c664a1c4b0f1be5c1c97823516358fdaab35771a4701dbefc0862127b1d0d4bfd02b4f20d2befa4434700
240ea294d5 doc: update doxygen documention of ComputeTimeSmart() and AddToWalletIfInvolvingMe() regarding rescanning_old_block parameter (BitcoinTsunami)
d6eb39af21 test: add functional test to check transaction time determination during block rescanning (BitcoinTsunami)
07b44f16e7 wallet: fix ComputeTimeSmart algorithm to use blocktime during old block rescanning (BitcoinTsunami)
Pull request description:
The function ComputeTimeSmart in wallet.cpp assume that transaction are discovered in the right order.
Moreover the 'smarttime' determination algorithm is coded with realtime scenario in mind and not rescanning of old block.
The functional test demonstrate that if the user import a wallet, then rescan only recent history, and then rescan the entire history, the older transaction discovered would have an incorrect time determination.
In the context of rescanning old block, the only time value that as a meaning is the blocktime.
That's why I've fixed the problem with a simple separation between rescanning of old block and realtime time determination. The fix is written to have no impact on every realtime scenario and only impact the behaviour during a rescanning process.
This PR Fixes#20181.
To be fair, I don't think that this bug could be triggered with the wallet GUI, because it always proceed with a proper rescan.
But RPC API provide the possibility to trigger it. I've discovered it, because Specter desktop v0.10.0 was impacted. (https://github.com/cryptoadvance/specter-desktop/issues/680).
ACKs for top commit:
jonatack:
ACK 240ea294d5 per `git diff b92d552 240ea29`, re-verified rebase to latest master + debug build clean + new test passes on the branch and fails on master, only change since my review a few hours ago is incorporation of latest review suggestions
meshcollider:
re-utACK 240ea294d5
Tree-SHA512: 514b02e41d011ddfa325f5e8080b93800e1ea4ed5853fa420670a6ac700e8b463000dbea65f8ced8565cfb950c7f51b69154034dcb111e67aca3b964a0061494
Improve readability of code, simplify future scripted diff cleanup PRs, and be
more consistent with naming for GetBoolArg.
This will also be useful for replacing runtime settings type checking
with compile time checking.
-BEGIN VERIFY SCRIPT-
git grep -l GetArg | xargs sed -i 's/GetArg(\([^)]*\( [0-9]\+\|-1\|port\|BaseParams().RPCPort()\|Params().GetDefaultPort()\|_TIMEOUT\|Height\|_WORKQUEUE\|_THREADS\|_CONNECTIONS\|LIMIT\|SigOp\|Bytes\|_VERSION\|_AGE\|_CHECKS\|Checks() ? 1 : 0\|_BANTIME\|Cache\|BLOCKS\|LEVEL\|Weight\|Version\|BUFFER\|TARGET\|WEIGHT\|TXN\|TRANSACTIONS\|ADJUSTMENT\|i64\|Size\|nDefault\|_EXPIRY\|HEIGHT\|SIZE\|SNDHWM\|_TIME_MS\)\))/GetIntArg(\1)/g'
-END VERIFY SCRIPT-
Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
Followup to commit "MOVEONLY: CWallet transaction code out of
wallet.cpp/.h" that detaches and renames some CWalletTx methods, making
into them into standalone functions or CWallet methods instead.
There are no changes in behavior and no code changes that aren't purely
mechanical. It just gives spend and receive functions more consistent
names and removes the circular dependencies added by the earlier
MOVEONLY commit.
There are also no comment or documentation changes. Removed comments
from transaction.h are just migrated to spend.h, receive.h, and
wallet.h.
86beee0579 Use waste metric for deciding which selection to use (Andrew Chow)
b3df0caf7c tests: Test GetSelectionWaste (Andrew Chow)
4f5ad43b1e Add waste metric calculation function (Andrew Chow)
935b3ddf72 scripted-diff: tests: Use KnapsackSolver directly (Andrew Chow)
6a023a6f90 tests: Add KnapsackGroupOutputs helper function (Andrew Chow)
d5069fc1aa tests: Use SelectCoinsBnB directly instead of AttemptSelection (Andrew Chow)
54de7b4746 Allow the long term feerate to be configured, default of 10 sat/vb (Andrew Chow)
Pull request description:
Branch and Bound introduced a metric that we call waste. This metric is used as part of bounding the search tree, but it can be generalized to all coin selection solutions, including those with change. As such, this PR introduces the waste metric at a higher level so that we can run both of our coin selection algorithms (BnB and KnapsackSolver) and choose the one which has the least waste. In the event that both find a solution with the same change, we choose the one that spends more inputs.
Also this PR sets the long term feerate to 10 sat/vb rather than using the 1008 block estimate. This allows the long term feerate to be the feerate that we switch between consolidating and optimizing for fees. This also removes a bug where the long term feerate would incorrectly be set to the fallback fee. While this doesn't matter prior to this PR, it does have an effect following this. The long term feerate can be configured by the user through a new `-consolidatefeerate` option.
ACKs for top commit:
Xekyo:
reACK 86beee0 via git range-diff fe47558...86beee0
meshcollider:
re-utACK 86beee0579
Tree-SHA512: 54b154b346538eca68ae2a3b83a033b495c1605c14f842bfc43ded2256b110983ce674c647fe753cf0305b1b178403d8d60d6d4203c7a712bec784be52e90d42
The long term feerate is really the highest feerate that the user is
comfortable with making consolidatory transactions. This is should thus
be something that can be configured by the user via a new startup option
-consolidatefeerate. The default value is 10 sat/vbyte, chosen
arbitrarily (it seems like a reasonable number).
c3c213215b Use `context.args` in `src/wallet/load.cpp`. (Kiminuo)
25de4e77fe Use `context.args` in `CWallet::Create` instead of `gArgs`. (Kiminuo)
aa5e7c9471 Fix typo in bitcoin-cli.cpp (Kiminuo)
Pull request description:
The PR attempts to move us an inch towards the [goal](https://github.com/bitcoin/bitcoin/pull/21244#discussion_r615307465) by using `WalletContext` in `wallet.{h|cpp}` code instead of relying on the global state (i.e. `gArgs`).
Edit: The PR builds on #19101.
ACKs for top commit:
ryanofsky:
Code review ACK c3c213215b. Changes since last review: just rebasing and adding wallet load commit
Tree-SHA512: 2b436f5a219e32c2d529f55a89edca086d949396cebf9e089a21aa7b1c180e3c0fb17468f415dfc834f8e1614f8b3914c7e9a0bd33b95e7e0199c0dfe5ca9490
But in case of no keys or a blank hd wallet the iterator would be skipped
and not set to false but true, since the loop would be not entered.
That had resulted in a wrong return and subsequent false HD and watch-only
icon display in gui when reloading a wallet after closing.
Update src/wallet/wallet.cpp
Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
92993aa5cf Change SignTransaction's input_errors to use bilingual_str (Andrew Chow)
171366e89b Use bilingual_str for address fetching functions (Andrew Chow)
9571c69b51 Add bilingual_str::clear() (Andrew Chow)
Pull request description:
In a couple of places in the wallet, errors are `std::string`. In order for these errors to be translated, change them to use `bilingual_str`.
ACKs for top commit:
hebasto:
re-ACK 92993aa5cf, only rebased since my [previous](https://github.com/bitcoin/bitcoin/pull/22337#pullrequestreview-694542729) review, verified with
klementtan:
Code review ACK 92993aa5cf
meshcollider:
Code review ACK 92993aa5cf
Tree-SHA512: 5400e419dd87db8c49b67ed0964de2d44b58010a566ca246f2f0760ed9ef6a9b6f6df7a6adcb211b315b74c727bfe8c7d07eb5690b5922fa5828ceef4c83461f
fa6fd3dd6a wallet: Properly set fInMempool in mempool notifications (MarcoFalke)
Pull request description:
A wallet method (like bumping the fee) might have set `fInMempool` to false because the transaction was removed from the mempool (See commit fa4e088cba).
Avoid setting it back to true (incorrectly) in the validation interface background thread.
Fixes#22357
ACKs for top commit:
ryanofsky:
Code review ACK fa6fd3dd6a. Only change since last review is extending workaround to `transactionRemovedFromMempool`. Since we know this workaround is imperfect and the goal of this PR is mainly to fix CI errors, I would probably be inclined to limit the workaround to as few places as possible where we have seen actual failures, instead of adding the workaround to as many places as possible, where there is some chance it might trigger new failures. But since this workaround is so straightforward and almost looks like a real fix, probably it doesn't matter.
meshcollider:
utACK fa6fd3dd6a
Tree-SHA512: d690136a577f1f532aa1fee80d3f6600ff7fc61286fbf564a53d7938d5ae52d33f0dbb0fef8b8c041a4970fb424f0b9f1ee7ce791e0ff8354e0000ecc9e22b84
DEBUG_LOCKORDER expects cs_wallet, cs_main, and cs_KeyStore to be
acquired in that order. However dumpwallet would take these in the order
cs_wallet, cs_KeyStore, cs_main. So when configured with
`--enable-debug`, it is possible to hit the lock order assertion when
using dumpwallet.
To fix this, cs_wallet and cs_KeyStore are no longer locked at the same
time. Instead cs_wallet will be locked first. Then the functions which
lock cs_main will be run. Lastly cs_KeyStore will be locked afterwards.
This avoids the lock order issue.
Furthermore, since GetKeyBirthTimes (only used by dumpwallet) also uses
a function that locks cs_main, and itself also locks cs_KeyStore, the
same reordering is done here.
6084d2caed wallet: do not spam about non-existent spk managers (S3RK)
Pull request description:
Avoid spam in logs during `loadwallet`, `listdescriptors` and probably other commands as well.
**`loadwallet` Before:**
```
2021-06-24T06:31:45Z init message: Loading wallet…
2021-06-24T06:31:45Z [desc] Wallet File Version = 169900
2021-06-24T06:31:45Z [desc] Keys: 0 plaintext, 0 encrypted, 0 w/ metadata, 0 total. Unknown wallet records: 0
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 0 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 1 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 2 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 0 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 1 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 2 does not exist
2021-06-24T06:31:45Z [desc] Wallet completed loading in 197ms
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 0 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 1 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 2 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 0 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 1 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 2 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 0 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 1 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 2 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 0 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 1 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 2 does not exist
2021-06-24T06:31:45Z [desc] setKeyPool.size() = 0
2021-06-24T06:31:45Z [desc] mapWallet.size() = 0
2021-06-24T06:31:45Z [desc] m_address_book.size() = 0
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 0 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 1 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 2 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 0 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 1 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 2 does not exist
{
"name": "desc",
"warning": ""
}
```
**After:**
```
2021-06-24T06:26:58Z init message: Loading wallet…
2021-06-24T06:26:58Z [desc] Wallet File Version = 169900
2021-06-24T06:26:58Z [desc] Keys: 0 plaintext, 0 encrypted, 0 w/ metadata, 0 total. Unknown wallet records: 0
2021-06-24T06:26:58Z [desc] Wallet completed loading in 158ms
2021-06-24T06:26:58Z [desc] setKeyPool.size() = 0
2021-06-24T06:26:58Z [desc] mapWallet.size() = 0
2021-06-24T06:26:58Z [desc] m_address_book.size() = 0
{
"name": "desc",
"warning": ""
}
```
ACKs for top commit:
achow101:
ACK 6084d2caed
Tree-SHA512: c7d7345c3182a575db088fd731b7f6e428c42e4f3f2e10d5adb50bf74a2defe88768e65ebb91a08590be48cf766a5697e36fafa73f68ffe45e76a60600f072e2
In many places in ScriptPubKeyMan managing code, we assume that the
ScriptPubKeyMan being retrieved actually exists and is not a nullptr.
Thus removing a ScriptPubKeyMan requires erasing the object from the
map rather than setting it to a nullptr.
181181019c refactor: remove m_internal from DescriptorSPKman (S3RK)
Pull request description:
Rationale: improve consistency between `CWallet` and `DescriptorScriptPubKeyMan`; simplify `ScriptPubKeyMan` interface.
Descriptor in itself is neither internal or external. It's responsibility of a wallet to assign and manage descriptors for a specific purpose. Duplicating information about internalness of a descriptor could lead to inconsistencies and unexpected behaviour (for example misreporting keypool size).
ACKs for top commit:
instagibbs:
reACK 181181019c
achow101:
reACK 181181019c
Tree-SHA512: d5613b7f6795b290bfa0fd8cb0536de1714d0cf72cba402266bd06d550758ebad690b54fc0a336a1c7414b5814aa4a37c90a6ae89926474a97d30956d7e034ff
3efaf83c75 wallet: deactivate descriptor (S3RK)
6737d9655b test: wallet importdescriptors update existing (S3RK)
586f1d53d6 wallet: maintain SPK consistency on internal flag change (S3RK)
f1b7db1474 wallet: don't mute exceptions in importdescriptors (S3RK)
bf68ebc1cd wallet: allow to import same descriptor twice (S3RK)
Pull request description:
Rationale: allow updating existing descriptors with `importdescriptors` command.
Currently if you run same `importdescriptors` command twice with a descriptor containing private key you will get very confusing error — `Missing required fields`. What happens is that Wallet tries to write imported private key to the disk, but it exists already so we get `DB_KEYEXIST (-30995)` from BerkelyDB. Please note, that we set `DB_NOOVERWRITE` (I guess not to lose some keys accidentally). The exception is caught in `catch (...)` in rpcdump.cpp with a generic error.
With this PR if a descriptor is already present than we will update its activeness, internalness, label, range and next_index.
For the range only expansion is allowed (range start can only decrease, range end increase).
ACKs for top commit:
achow101:
re-ACK 3efaf83c75
meshcollider:
Code review ACK 3efaf83c75
jonatack:
Light ACK 3efaf83c75 per `git range-diff a000cb0 5d96704 3efaf83` and as a sanity check, re-debug-built on debian with gcc 10.2.1 and clang 11, ran wallet_importdescriptors.py
Tree-SHA512: 122c4b621d64ec8a3b625f3aed9f01a2b5cbaf2029ad0325b5ff38d67fff5cd35324335fabe2dd5169548b01b267c81be6ae0f5c834342f3d5f6eeed515c4843
e6cf0ed92d wallet, rpc: listdescriptors does not need unlocked (Andrew Chow)
3280704886 Pass in DescriptorCache to ToNormalizedString (Andrew Chow)
7a26ff10c2 Change DescriptorImpl::ToStringHelper to use an enum (Andrew Chow)
75530c93a8 Remove priv option for ToNormalizedString (Andrew Chow)
74fede3b8b wallet: Upgrade existing descriptor caches (Andrew Chow)
432ba9e543 wallet: Store last hardened xpub cache (Andrew Chow)
d87b544b83 descriptors: Cache last hardened xpub (Andrew Chow)
cacc391098 Move DescriptorCache writing to WalletBatch (Andrew Chow)
0b4c8ef75c Refactor Cache merging and writing (Andrew Chow)
976b53b085 Revert "Cache parent xpub inside of BIP32PubkeyProvider" (Andrew Chow)
Pull request description:
Currently fetching a normalized descriptor requires the wallet to be unlocked as it needs the private keys to derive the last hardened xpub. This is not very user friendly as normalized descriptors shouldn't require and don't involve the private keys except for derivation. We solve this problem by caching the last hardened xpub (which has to be derived at some point when generating the address pool).
However the last hardened xpub was not already being cached. We only cached the immediate parent xpub and derived child keys. For example, with a descriptor derivation path of `/84'/0'/0'/0/*`, the parent xpub that is cached is `m/84'/0'/0'/0`, and the child keys of `m/84'/0'/0'/0/i` (note that child keys would not be cached in this case). This parent xpub is not suitable for the normalized descriptor form as we want the key at `m/84'/0'/0'`. So this PR adds another field to `DescriptorCache` to cache the last hardened xpub so that we can use them for normalized descriptors.
Since `DescriptorCache` is changing, existing descriptor wallets need to be upgraded to use this new cache. The upgrade will occur in the background either at loading time (if the wallet is not encrypted) or at unlocking time in the same manner that `UpgradeKeyMetadata` operates. It will use a new wallet flag `WALLET_FLAG_LAST_HARDENED_XPUB_CACHED` to indicate whether the descriptor wallet has the last hardened xpub cache.
Lastly `listdescriptors` will not require the wallet to be locked and `getaddressinfo`'s `parent_desc` will always be output (assuming the upgrade has occurred).
ACKs for top commit:
fjahr:
tACK e6cf0ed92d
S3RK:
reACK e6cf0ed
jonatack:
Semi ACK e6cf0ed92d reviewed, debug-built and ran unit tests and some of the descriptor functional tests at each commit. I'm not very familiar with this code and it could be clearer to the uninitiated IMHO, so I'm not confident enough to give a full ACK. Various minor suggestions follow, most of them for readability, feel free to pick and choose.
meshcollider:
Code review + functional test run ACK e6cf0ed92d
Tree-SHA512: ac27aade8644525cd65bfcaf27ff32afb974085b1451faf4ff68c6671a690bd6a41d4f39a33cbf461ae0fbe85995c0a4c08dbd36171da1c1d2a1d00053ad298d
Descriptor in itself is neither internal or external.
It's responsibility of a wallet to assign and manage descriptors
for a specific purpose. Duplicating such information could lead to
inconsistencies and unexpected behaviour.
Adds an error output parameter to all GetReservedDestination functions
so that callers can get the actual reason that a change address could
not be fetched. This more closely matches GetNewDestination. This allows
for more granular error messages, such as one that indicates that
bech32m addresses cannot be generated yet.
If a transaction as a segwit output, use a bech32m change address if
they are available. If not, fallback to bech32. If bech32 change
addresses are unavailable, fallback to the default address type.
Bech32m addresses need their own OutputType
We are not ready to create DescriptorScriptPubKeyMans which produce
bech32m addresses. So don't allow generating them.
458a345b05 Add support for SIGHASH_DEFAULT in RPCs, and make it default (Pieter Wuille)
c0f0c8eccb tests: check spending of P2TR (Pieter Wuille)
a2380127e9 Basic Taproot signing logic in script/sign.cpp (Pieter Wuille)
49487bc3b6 Make GetInputUTXO safer: verify non-witness UTXO match (Pieter Wuille)
fd3f6890f3 Construct and use PrecomputedTransactionData in PSBT signing (Pieter Wuille)
5cb6502ac5 Construct and use PrecomputedTransactionData in SignTransaction (Pieter Wuille)
5d2e22437b Don't nuke witness data when signing fails (Pieter Wuille)
ce9353164b Permit full precomputation in PrecomputedTransactionData (Pieter Wuille)
e841fb503d Add precomputed txdata support to MutableTransactionSignatureCreator (Pieter Wuille)
a91d532338 Add CKey::SignSchnorr function for BIP 340/341 signing (Pieter Wuille)
e77a2839b5 Use HandleMissingData also in CheckSchnorrSignature (Pieter Wuille)
dbb0ce9fbf Add TaprootSpendData data structure, equivalent to script map for P2[W]SH (Pieter Wuille)
Pull request description:
Builds on top of #22051, adding signing support after derivation support.
Nothing is changed in descriptor features. Signing works for key path and script path spending, through the normal sending functions, and PSBT-based RPCs. However, PSBT usability is rather low as no extensions have been defined to convey Taproot-specific information, so all script information must be known to the signing wallet.
ACKs for top commit:
achow101:
re-ACK 458a345b05
fjahr:
Code review ACK 458a345b05
Sjors:
ACK 458a345b05
Tree-SHA512: 30ed212cf7754763a4a81624ebc084c51727b8322711ac0b390369213c1a891d367ed8b123882ac08c99595320c11ec57ee42304ff22a69afdc3d1a0d55cc711
f5ba424cd4 wallet: Add IsAddressUsed / SetAddressUsed methods (Russell Yanofsky)
62252c95e5 interfaces: Stop exposing wallet destdata to gui (Russell Yanofsky)
985430d9b2 test: Add gui test for wallet receive requests (Russell Yanofsky)
Pull request description:
Stop giving GUI access to destdata rows in database. Replace with narrow API just for saving and reading receive request information.
This simplifies code and should prevent the GUI from interfering with other destdata like address-used status. It also adds some more GUI test coverage.
There are no changes in behavior.
ACKs for top commit:
jarolrod:
tACK f5ba424cd4
laanwj:
Code review ACK f5ba424cd4
Tree-SHA512: 5423df4786e537a59013cb5bfb9e1bc29a7ca4b8835360c00cc2165a59f925fdc355907a4ceb8bca0285bb4946ba235bffa7645537a951ad03fd3b4cee17b6b0
c7bd5842e4 MOVEONLY: CWallet transaction code out of wallet.cpp/.h (Russell Yanofsky)
Pull request description:
This commit just moves function without making any changes. It can be reviewed with `git log -p -n1 --color-moved=dimmed_zebra`
Motivation for this change is to make `wallet.cpp/h` less monolithic and start to make wallet transaction state tracking comprehensible so bugs in https://github.com/bitcoin-core/bitcoin-devwiki/wiki/Wallet-Transaction-Conflict-Tracking can be fixed safely without introducing new problems.
This moves wallet classes and methods that deal with transactions out of `wallet.cpp/.h` into better organized files:
- `transaction.cpp/.h` - CWalletTx and CMerkleTx class definitions
- `receive.cpp/.h` - functions checking received transactions and computing balances
- `spend.cpp/.h` - functions creating transactions and finding spendable coins
After #20773, when loading is separated from syncing it will also be possible to move more `wallet.cpp/.h` functions to:
- `sync.cpp/.h` - functions handling chain notifications and rescanning
This commit arranges `receive.cpp` and `spend.cpp` functions in dependency order so it's possible to skim `receive.cpp` and get an idea of how computing balances works, and skim `spend.cpp` and get an idea of how transactions are created, without having to jump all over `wallet.cpp` where functions are not in order and there is a lot of unrelated code.
Followup commit "refactor: Detach wallet transaction methods" in https://github.com/bitcoin/bitcoin/pull/21206 follows up this PR and tweaks function names and arguments to reflect new locations. The two commits are split into separate PRs because this commit is more work to maintain and less work to review, while the other commit is less work to maintain and more work to review, so hopefully this commit can be merged earlier.
ACKs for top commit:
Sjors:
re-utACK c7bd5842e4
fjahr:
utACK c7bd5842e4
promag:
Code review ACK c7bd5842e4, verified move only claim.
meshcollider:
Dimmed-zebra-check and functional test run ACK c7bd5842e4
Tree-SHA512: 4981de6911cb1196774db375494355cc9af59b52456129c002d264a77cd9ed6175f8ecbb6b2f492a59a4d5a0def21a39d96fa79c9f4d99be0992985f553be32f
This commit just moves functions without making any changes. It can be
reviewed with `git log -p -n1 --color-moved=dimmed_zebra`
Motivation for this change is to make wallet.cpp/h less monolithic and
start to make wallet transaction state tracking comprehensible so bugs
in
https://github.com/bitcoin-core/bitcoin-devwiki/wiki/Wallet-Transaction-Conflict-Tracking
can be fixed safely without introducing new problems.
This commit moves wallet classes and methods that deal with transactions
out of wallet.cpp/.h into better organized files:
- transaction.cpp/.h - CWalletTx and CMerkleTx class definitions
- receive.cpp/.h - functions checking received transactions and computing balances
- spend.cpp/.h - functions creating transactions and finding spendable coins
After #20773, when loading is separated from syncing it will also be
possible to move more wallet.cpp/.h functions to:
- sync.cpp/.h - functions handling chain notifications and rescanning
This commit arranges receive.cpp and spend.cpp functions in dependency
order so it's possible to skim receive.cpp and get an idea of how
computing balances works, and skim spend.cpp and get an idea of how
transactions are created, without having to jump all over wallet.cpp
where functions are not in order and there is a lot of unrelated code.
Followup commit "refactor: Detach wallet transaction methods" in
https://github.com/bitcoin/bitcoin/pull/21206 follows up this PR and
tweaks function names and arguments to reflect new locations. The two
commits are split into separate PRs because this commit is more work to
maintain and less work to review, while the other commit is less work to
maintain and more work to review, so hopefully this commit can be merged
earlier.
e6fe1c37d0 rpc: Improve avoidpartialspends and avoid_reuse documentation (Fabian Jahr)
8f073076b1 wallet: Increase OUTPUT_GROUP_MAX_ENTRIES to 100 (Fabian Jahr)
Pull request description:
Follow-up to #17824.
This increases OUTPUT_GROUP_MAX_ENTRIES to 100 which means that OutputGroups will now be up to 100 outputs large, up from previously 10. The main motivation for this change is that during the PR review club on #17824 [several participants signaled](https://bitcoincore.reviews/17824.html#l-339) that 100 might be a better value here.
I think fees should be manageable for users but more importantly, users should know what they can expect when using the wallet with this configuration, so I also tried to clarify the documentation on `-avoidpartialspends` and `avoid_reuse` a bit. If there are other additional ways how or docs where users can be made aware of the potential consequences of using these parameters, please let me know. Another small upside is that [there seem to be a high number of batching transactions with 100 and 200 inputs](https://miro.medium.com/max/3628/1*sZ5eaBSbsJsHx-J9iztq2g.png)([source](https://medium.com/@hasufly/an-analysis-of-batching-in-bitcoin-9bdf81a394e0)) giving these transactions a bit of a larger anonymity set, although that is probably a very weak argument.
ACKs for top commit:
jnewbery:
ACK e6fe1c37d0
Xekyo:
retACK e6fe1c37d0
rajarshimaitra:
tACK `e6fe1c3`
achow101:
ACK e6fe1c37d0
glozow:
code review ACK e6fe1c37d0
Tree-SHA512: 79685c58bafa64ed8303b0ecd616fce50fc9a2b758aa79833e4ad9f15760e09ab60c007bc16ab4cbc4222e644cfd154f1fa494b0f3a5d86faede7af33a6f2826
51a3ac242c Have OutputGroup determine the value to use (Andrew Chow)
6d6d278475 Change SelectCoins_test to actually test SelectCoins (Andrew Chow)
9d3bd74ab4 Remove CreateTransaction while loop and some related variables (Andrew Chow)
6f0d5189af Remove use_bnb and bnb_used (Andrew Chow)
de26eb0e1f Do both BnB and Knapsack coin selection in SelectCoinsMinConf (Andrew Chow)
01dc8ebda5 Have KnapsackSolver actually use effective values (Andrew Chow)
bf26e018de Roll static tx fees into nValueToSelect instead of having it be separate (Andrew Chow)
cc3f14b27c Move output reductions for fee to after coin selection (Andrew Chow)
d97d25d950 Make cost_of_change part of CoinSelectionParams (Andrew Chow)
af5867c896 Move some calculations to common code in SelectCoinsMinConf (Andrew Chow)
1bf4a62cb6 scripted-diff: rename some variables (Andrew Chow)
Pull request description:
Changes `KnapsackSolver` to use effective values instead of just the nominal txout value. Since fees are taken into account during the selection itself, we finally get rid of the `CreateTransaction` loop as well as a few other things that only were only necessary because of that loop.
This should not change coin selection behavior at all (except maybe remove weird edge cases that were caused by the loop). In order to keep behavior the same, `KnapsackSolver` will select outputs with a negative effective value (as it did before).
ACKs for top commit:
ryanofsky:
Code review ACK 51a3ac242c. Looks good to go!
instagibbs:
review ACK 51a3ac242c
meshcollider:
re-light-utACK 51a3ac242c
Tree-SHA512: 372c27e00edcd5dbf85177421ba88f20bfdaf1791b6e3dc022c44876ecc379403e2375ed69e71c512c49e6af87641001ff385c4b25ab93684b3a08a53bf3824e
Instead of hijacking the effective_feerate to use the correct value
during coin selection, have OutputGroup be aware of whether we are
subtracting the fee from the outputs and provide the correct value to
use for selection.
To do this, OutputGroup now takes CoinSelectionParams and has a new
function GetSelectionAmount().
Remove the CreateTransaction while loop. Removes variables that were
only needed because of that loop. Also renames a few variables and
moves their declarations to where they are used.
Some subtractFeeFromOutputs handling is moved to after coin selection
in order to reduce their amounts once the fee is known.
If subtracting the fee reduces the change to dust, we will also now
remove the change output
Although the CreateTransaction loop currently remains, it should be
largely unused. KnapsackSolver will now account for transaction fees
when doing its selection.
In the previous commit, SelectCoinsMinConf was refactored to have some
calculations become shared for KnapsackSolver and SelectCoinsBnB. In
this commit, KnapsackSolver will now use the not_input_fees and
effective_feerate so that it include the fee for non-input things
(excluding a change output) so that the algorithm will select enough to
cover those fees. This is necessary for selecting on effective values.
Additionally, the OutputGroups
created for KnapsackSolver will actually have their effective values
calculated and set, and KnapsackSolver will do its selection on those
effective values.
Lastly, SelectCoins is modified to use the same value for preselected
inputs for BnB and KnapsackSolver. While it will still use the real
value when subtracting the fee from outputs, this behavior will be
the same regardless of the algo used for selecting additional inputs.
The fees for transaction overhead and recipient outputs are now included
in nTargetValue instead of being a separate parameter. For the coin
selection algorithms, it doesn't matter that these are separate as in
either case, the algorithm needs to select enough to cover these fees.
Note that setting nValueToSelect is changed as it now includes
not_input_fees. Without the change to how nValueToSelect is increased
for KnapsackSolver, this would result in overpaying fees. The change to
increase by the difference between nFeeRet and not_input_fees allows
this to have the same behavior as previously.
Additionally, because we assume that KnapsackSolver will always find a
solution that requires change (we assume that BnB always finds a
non-change solution), we also include the fee for the change output in
KnapsackSolver's target. As part of this, we also use the changeless
nFeeRet when iterating for KnapsackSolver. This is because we include
the change fee when doing KnapsackSolver, so nFeeRet on further
iterations won't include the change fee.
Simplifies CreateTransactionInternal without changing behavior. Removes
the pick_new_inputs variable by moving the subtract fee from amount
implementation to later in the loop to where it is possible to calculate
the fee for the transaction. This allows the fee to be subtracted from
the outputs within a single iteration, instead of calculating the fee in
the first iteration, and subtracting the fee in the second.
This also removes another scenario where a second iteration of the loop
finds a smaller input set (and thus smaller fees than the first
iteration) with no change and so a third iteration of the loop is done in order to make
a change output that contains the excess fees.
To handle these cases, we always create a change output which contains
the difference between selected input values and the recipient amounts.
Once the transaction fee is calculated, the change output is reduced (in
the normal case) or the recipient amounts are reduced (in the subtract
fee from amount case). All of this is done in a single iteration of the
loop.
This commit does not change behavior, it just moves code from
CWallet::CreateWalletFromFile to CWallet:::AttachChain so it can be updated in
the next commit.
This commit is most easily reviewed with
"git diff -w --color-moved=dimmed_zebra" or by diffing CWallet:::AttachChain
against the previous code with an external diff tool.
To prepare for KnapsackSolver to use effective values, these
calculations are moved out of the BnB if block to allow for them to be
shared with KnapsackSolver in the future.
The fOnlySafe argument to AvailableCoins is now redundant, since #21359
added a similar field inside the CCoinControl struct.
Not all code paths set a CCoinControl instance, but when it's missing we
can default to using only safe inputs which is backwards-compatible.
11d6459b6e rpc: include_unsafe option for fundrawtransaction (t-bast)
Pull request description:
Allow RPC users to opt-in to unsafe inputs when funding a raw transaction.
Applications that need to manage a complex RBF flow (such as lightning nodes using anchor outputs) are very limited if they can only use safe inputs.
I also added this option to `send` and `walletcreatefundedpsbt` who internally delegate to `fundrawtransaction`.
Fixes#21299
ACKs for top commit:
laanwj:
Code review ACK 11d6459b6e
Tree-SHA512: 5e542a4febcfd6f41cf784678ff02ec9282eae2082c274983f72c5ea87b7ebbe1bd5fdc6a020d7a9d5996157754eb4966b8aeb6c1ceebf0b1519f735579b8bac
d66f283ac0 scripted-diff: Replace three dots with ellipsis in the UI strings (Hennadii Stepanov)
Pull request description:
This PR is split from #21463.
The change was suggested on [Transifex.com](https://www.transifex.com/bitcoin/bitcoin/), and it does not touch `LogPrint` and `LogPrintf` calls.
The only comment on #21463 [was](9030e4b5a6 (r597220100)):
> Mind that these messages also end up in the log. In principle the log is already UTF-8 (as are all strings and text in bitcoind). But, just noting, that it might make browsing the log a less pleasant experience on systems with misconfigured locale like some BSDs by default.
ACKs for top commit:
laanwj:
ACK d66f283ac0
Tree-SHA512: 5ab1cb3160f3f996f1ad7d7486662da3eb7f06a857f4a1874963ce10caed5b86b0ad6151b1b9ebeb2b8aa5f0c85efad3b768ea9cafe5db86f78f88912b756d1e
