Andrew Chow 59bd6b6d37 Merge bitcoin/bitcoin#24699: wallet: Improve AvailableCoins performance by reducing duplicated operations ... bc886fcb31 Change mapWallet to be a std::unordered_map (Andrew Chow) 272356024d Change getWalletTxs to return a set instead of a vector (Andrew Chow) 97532867cf Change mapTxSpends to be a std::unordered_multimap (Andrew Chow) 1f798fe85b wallet: Cache SigningProviders (Andrew Chow) 8a105ecd1a wallet: Use CalculateMaximumSignedInputSize to indicate solvability (Andrew Chow) Pull request description: While running my coin selection simulations, I noticed that towards the end of the simulation, the wallet would become slow to make new transactions. The wallet generally performs much more slowly when there are a large number of transactions and/or a large number of keys. The improvements here are focused on wallets with a large number of transactions as that is what the simulations produce. Most of the slowdown I observed was due to `DescriptorScriptPubKeyMan::GetSigningProvider` re-deriving keys every time it is called. To avoid this, it will now cache the `SigningProvider` produced so that repeatedly fetching the `SigningProvider` for the same script will not result in the same key being derived over and over. This has a side effect of making the function non-const, which makes a lot of other functions non-const as well. This helps with wallets with lots of address reuse (as my coin selection simulations are), but not if addresses are not reused as keys will end up needing to be derived the first time `GetSigningProvider` is called for a script. The `GetSigningProvider` problem was also exacerbated by unnecessarily fetching a `SigningProvider` for the same script multiple times. A `SigningProvider` is retrieved to be used inside of `IsSolvable`. A few lines later, we use `GetTxSpendSize` which fetches a `SigningProvider` and then calls `CalculateMaximumSignedInputSize`. We can avoid a second call to `GetSigningProvider` by using `CalculateMaximumSignedInputSize` directly with the `SigningProvider` already retrieved for `IsSolvable`. There is an additional slowdown where `ProduceSignature` with a dummy signer is called twice for each output. The first time is `IsSolvable` checks that `ProduceSignature` succeeds, thereby informing whether we have solving data. The second is `CalculateMaximumSignedInputSize` which returns -1 if `ProduceSignature` fails, and returns the input size otherwise. We can reduce this to one call of `ProduceSignature` by using `CalculateMaximumSignedInputSize`'s result to set `solvable`. Lastly, a lot of time is spent looking in `mapWallet` and `mapTxSpends` to determine whether an output is already spent. The performance of these lookups is slightly improved by changing those maps to use `std::unordered_map` and `std::unordered_multimap` respectively. ACKs for top commit: Xekyo: ACK bc886fcb31 furszy: diff re-reACK bc886fcb Tree-SHA512: fd710fe1224ef67d2bb83d6ac9e7428d9f76a67f14085915f9d80e1a492d2c51cb912edfcaad1db11c2edf8d2d97eb7ddd95bfb364587fb1f143490fd72c9ec1		2022-08-05 15:31:45 -04:00
..
bench	Merge bitcoin/bitcoin#25721: refactor: Replace BResult with util::Result	2022-08-05 15:33:45 +02:00
common	refactor: make GetRand a template, remove GetRandInt	2022-04-22 09:04:39 -05:00
compat	compat: document redefining ssize_t when using MSVC	2022-07-20 13:10:12 +01:00
config
consensus	Remove LOCKTIME_MEDIAN_TIME_PAST constant	2022-06-22 09:54:15 +02:00
crc32c	Update crc32c subtree	2021-09-29 14:10:29 +02:00
crypto	Use HashWriter where possible	2022-07-20 15:34:36 +02:00
index	Merge bitcoin/bitcoin#22485: doc: BaseIndex sync behavior with empty datadir	2022-07-21 19:54:18 +02:00
init	Merge bitcoin/bitcoin#24675: util: Use ArgsManager::GetPathArg more widely	2022-08-04 16:58:01 +01:00
interfaces	Merge bitcoin/bitcoin#24699: wallet: Improve AvailableCoins performance by reducing duplicated operations	2022-08-05 15:31:45 -04:00
ipc	refactor: Block unsafe fs::path std::string conversion calls	2021-10-05 11:10:47 -04:00
kernel	validationcaches: Use size_t for sizes	2022-08-03 12:03:28 -04:00
leveldb	Update leveldb-subtree subtree to latest upstream	2022-07-29 14:43:11 +01:00
logging	util, refactor: Add UNIQUE_NAME helper macro	2022-02-16 14:59:20 +02:00
minisketch	Update minisketch subtree to latest master	2022-06-29 16:35:02 +01:00
node	validationcaches: Use size_t for sizes	2022-08-03 12:03:28 -04:00
policy	Remove ::g_max_datacarrier_bytes global	2022-08-02 15:29:16 +02:00
primitives	refactor: Make CTransaction constructor explicit	2022-07-25 12:16:54 +02:00
qt	Merge bitcoin/bitcoin#25721: refactor: Replace BResult with util::Result	2022-08-05 15:33:45 +02:00
rpc	Merge bitcoin/bitcoin#22751: rpc/wallet: add simulaterawtransaction RPC	2022-08-05 15:19:03 -04:00
script	validationcaches: Use size_t for sizes	2022-08-03 12:03:28 -04:00
secp256k1	Update secp256k1 subtree to latest upstream master	2022-06-11 14:55:18 +01:00
support	build: globally define NOMINMAX	2022-06-20 12:22:05 +01:00
test	Merge bitcoin/bitcoin#25721: refactor: Replace BResult with util::Result	2022-08-05 15:33:45 +02:00
univalue	univalue: Remove unused and confusing set*() return value	2022-07-29 15:24:42 +02:00
util	Merge bitcoin/bitcoin#25721: refactor: Replace BResult with util::Result	2022-08-05 15:33:45 +02:00
wallet	Merge bitcoin/bitcoin#24699: wallet: Improve AvailableCoins performance by reducing duplicated operations	2022-08-05 15:31:45 -04:00
zmq	scripted-diff: remove duplicate categories from LogPrint output	2022-06-06 12:12:03 +02:00
.bear-tidy-config	tidy: enable modernize-use-nullptr	2022-04-26 10:43:33 +01:00
.clang-format	Use c++17 in clang-format	2021-11-12 11:46:34 +01:00
.clang-tidy	tidy: enable readability-redundant-string-init	2022-07-26 10:16:42 +01:00
addrdb.cpp	Merge bitcoin/bitcoin#24925: refactor: make GetRand a template, remove GetRandInt	2022-05-12 08:57:22 +02:00
addrdb.h	[net] Move asmap into NetGroupManager	2022-04-20 14:29:29 +01:00
addrman.cpp	addrman: Use system time instead of adjusted network time	2022-07-30 11:04:09 +02:00
addrman.h	addrman: Use system time instead of adjusted network time	2022-07-30 11:04:09 +02:00
addrman_impl.h	addrman: Use system time instead of adjusted network time	2022-07-30 11:04:09 +02:00
arith_uint256.cpp	Merge bitcoin/bitcoin#24077: util: Make base_uint::GetHex() and base_uint::SetHex() not depend on uint256	2022-04-14 07:15:22 +02:00
arith_uint256.h	Remove duplicate static_asserts	2022-04-14 19:26:22 +02:00
attributes.h
banman.cpp	scripted-diff: Avoid incompatibility with CMake AUTOUIC feature	2022-06-14 10:38:51 +02:00
banman.h	refactor: Add thread safety annotation to BanMan::SweepBanned()	2022-05-20 15:17:00 +02:00
base58.cpp	scripted-diff: Rename ValidAsCString to ContainsNoNUL	2022-04-27 14:16:35 +02:00
base58.h	refactor: Remove defunct attributes.h includes	2022-05-21 13:54:33 -05:00
bech32.cpp	Avoid implicit-integer-sign-change in bech32.cpp	2022-02-25 09:43:54 +01:00
bech32.h	Make Bech32 LocateErrors return error list rather than using out-arg	2021-12-06 14:17:41 +13:00
bitcoin-chainstate.cpp	validationcaches: Add and use ValidationCacheSizes	2022-08-03 12:03:27 -04:00
bitcoin-cli-res.rc	windres: use PACKAGE_VERSION rather than building more version numbers	2021-08-17 16:54:47 +08:00
bitcoin-cli.cpp	Merge bitcoin/bitcoin#24675: util: Use ArgsManager::GetPathArg more widely	2022-08-04 16:58:01 +01:00
bitcoin-tx-res.rc	windres: use PACKAGE_VERSION rather than building more version numbers	2021-08-17 16:54:47 +08:00
bitcoin-tx.cpp	refactor: move compat.h into compat/	2022-07-20 10:34:46 +01:00
bitcoin-util-res.rc	windres: use PACKAGE_VERSION rather than building more version numbers	2021-08-17 16:54:47 +08:00
bitcoin-util.cpp	refactor: move compat.h into compat/	2022-07-20 10:34:46 +01:00
bitcoin-wallet-res.rc	windres: use PACKAGE_VERSION rather than building more version numbers	2021-08-17 16:54:47 +08:00
bitcoin-wallet.cpp	refactor: move compat.h into compat/	2022-07-20 10:34:46 +01:00
bitcoind-res.rc	windres: use PACKAGE_VERSION rather than building more version numbers	2021-08-17 16:54:47 +08:00
bitcoind.cpp	refactor: move compat.h into compat/	2022-07-20 10:34:46 +01:00
blockencodings.cpp	Remove fUseWTXID parameter from CBlockHeaderAndShortTxIDs constructor	2022-05-17 10:37:10 +01:00
blockencodings.h	Remove fUseWTXID parameter from CBlockHeaderAndShortTxIDs constructor	2022-05-17 10:37:10 +01:00
blockfilter.cpp	refactor: remove unnecessary string initializations	2022-07-26 10:16:42 +01:00
blockfilter.h	Make sanity check in GCSFilter constructor optional	2022-05-02 16:04:00 +09:00
chain.cpp	Remove unused SetTip(nullptr) code	2022-08-03 09:21:53 +02:00
chain.h	Remove unused SetTip(nullptr) code	2022-08-03 09:21:53 +02:00
chainparams.cpp	Use HashWriter where possible	2022-07-20 15:34:36 +02:00
chainparams.h	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
chainparamsbase.cpp	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
chainparamsbase.h
chainparamsseeds.h	net: Update hardcoded seeds for 23.x	2022-02-22 15:15:27 +01:00
checkqueue.h	Increase threadsafety annotation coverage	2022-05-12 02:25:55 +10:00
clientversion.cpp	refactor: shift CopyrightHolders() and LicenseInfo() to clientversion.cpp	2022-02-22 15:36:19 +00:00
clientversion.h	refactor: shift CopyrightHolders() and LicenseInfo() to clientversion.cpp	2022-02-22 15:36:19 +00:00
coins.cpp	tracing: do not use coin after move in CCoinsViewCache::AddCoin	2022-07-21 12:55:20 +03:00
coins.h	refactor: remove unused methods {CDBIterator,CCoinsViewDBCursor}::GetValueSize()	2022-06-21 16:19:10 +02:00
compressor.cpp	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
compressor.h	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
core_io.h	refactor: Remove defunct attributes.h includes	2022-05-21 13:54:33 -05:00
core_memusage.h
core_read.cpp	core_read: Replace boost::split with SplitString	2022-05-04 07:34:47 +02:00
core_write.cpp	refactor: add stdd:: includes to core_write	2022-03-30 20:11:15 +01:00
cuckoocache.h	validationcaches: Use size_t for sizes	2022-08-03 12:03:28 -04:00
dbwrapper.cpp	Remove unused includes from dbwrapper.h	2022-07-19 14:32:53 +02:00
dbwrapper.h	Remove unused includes from dbwrapper.h	2022-07-19 14:32:53 +02:00
deploymentinfo.cpp	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
deploymentinfo.h	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
deploymentstatus.cpp	validation: move g_versionbitscache into ChainstateManager	2022-05-10 12:09:33 +10:00
deploymentstatus.h	validation: move g_versionbitscache into ChainstateManager	2022-05-10 12:09:33 +10:00
dummywallet.cpp	wallet: ensure wallet files are not reused across chains	2022-02-16 15:02:26 +02:00
external_signer.cpp	refactor: remove unnecessary string initializations	2022-07-26 10:16:42 +01:00
external_signer.h	refactor: make ExternalSigner NetworkArg() and m_chain private	2021-06-16 10:48:58 +02:00
flatfile.cpp	Disallow more unsafe string->path conversions allowed by path append operators	2022-04-21 12:01:00 -05:00
flatfile.h
fs.cpp	build: globally define NOMINMAX	2022-06-20 12:22:05 +01:00
fs.h	Move FopenFn to fsbridge namespace	2022-07-15 12:25:51 -04:00
hash.cpp	Use HashWriter where possible	2022-07-20 15:34:36 +02:00
hash.h	Use HashWriter where possible	2022-07-20 15:34:36 +02:00
httprpc.cpp	scripted-diff: Use getInt<T> over get_int/get_int64	2022-05-18 19:15:03 +02:00
httprpc.h	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
httpserver.cpp	refactor: move compat.h into compat/	2022-07-20 10:34:46 +01:00
httpserver.h	Merge bitcoin/bitcoin#24681: build: Bump libevent minimum version up to 2.1.8	2022-04-06 13:19:36 +01:00
i2p.cpp	refactor: move compat.h into compat/	2022-07-20 10:34:46 +01:00
i2p.h	refactor: move compat.h into compat/	2022-07-20 10:34:46 +01:00
indirectmap.h
init.cpp	validationcaches: Add and use ValidationCacheSizes	2022-08-03 12:03:27 -04:00
init.h	Move init::SanityCheck to kernel::SanityCheck	2022-06-02 11:42:12 -04:00
key.cpp	refactor: Use Span of std::byte in CExtKey::SetSeed	2022-04-27 19:53:37 +02:00
key.h	refactor: Use Span of std::byte in CExtKey::SetSeed	2022-04-27 19:53:37 +02:00
key_io.cpp	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
key_io.h	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
logging.cpp	logging: fix logging empty threadname	2022-06-02 22:30:30 +08:00
logging.h	logging: add LogPrintfCategory to log unconditionally with category	2022-06-08 14:02:54 +02:00
Makefile.am	validationcaches: Add and use ValidationCacheSizes	2022-08-03 12:03:27 -04:00
Makefile.bench.include	Merge bitcoin/bitcoin#24852: util: optimize HexStr	2022-05-04 20:36:09 +02:00
Makefile.crc32c.include	build: Create .la library for crc32c	2022-04-26 16:25:38 -04:00
Makefile.leveldb.include	Use more specific path when including memenv.h header	2022-06-23 15:33:01 +02:00
Makefile.minisketch.include	build: add minisketch build file and include it	2021-10-21 09:37:30 +08:00
Makefile.qt.include	qt, refactor: Add transactionoverviewwidget.cpp source file	2022-06-14 16:55:22 +02:00
Makefile.qt_locale.include	qt: Pre-branch translation updates for 23.x	2022-02-28 16:59:56 +01:00
Makefile.qttest.include	build: Remove vestigial LIBLEVELDB_SSE42	2022-04-11 16:56:34 -04:00
Makefile.test.include	refactor: Replace BResult with util::Result	2022-08-03 07:33:01 -04:00
Makefile.test_fuzz.include	test/fuzz: Invoke LoadMempool via CChainState	2022-07-15 12:26:00 -04:00
Makefile.test_util.include	build: Don't add unrelated libs to LIBTEST_*	2022-04-11 16:56:34 -04:00
Makefile.univalue.include	Integrate univalue into our buildsystem	2021-10-11 20:46:25 +08:00
mapport.cpp	refactor: move compat.h into compat/	2022-07-20 10:34:46 +01:00
mapport.h
memusage.h
merkleblock.cpp
merkleblock.h	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
net.cpp	addrman: Use system time instead of adjusted network time	2022-07-30 11:04:09 +02:00
net.h	refactor: move compat.h into compat/	2022-07-20 10:34:46 +01:00
net_permissions.cpp	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
net_permissions.h	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
net_processing.cpp	Merge bitcoin/bitcoin#24662: addrman: Use system time instead of adjusted network time	2022-08-05 09:03:33 +01:00
net_processing.h	[net processing] Remove CNode::nServices	2022-07-14 14:50:44 +02:00
net_types.cpp	scripted-diff: Use getInt<T> over get_int/get_int64	2022-05-18 19:15:03 +02:00
net_types.h	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
netaddress.cpp	refactor: use C++11 default initializers	2022-05-17 17:18:58 +01:00
netaddress.h	refactor: move compat.h into compat/	2022-07-20 10:34:46 +01:00
netbase.cpp	refactor: move compat.h into compat/	2022-07-20 10:34:46 +01:00
netbase.h	refactor: move compat.h into compat/	2022-07-20 10:34:46 +01:00
netgroup.cpp	[netgroup] Use nStartByte as offset for the last byte of the group	2022-04-25 15:09:14 +02:00
netgroup.h	[netgroupman] Remove NetGroupManager::GetAsmap()	2022-04-20 14:35:53 +01:00
netmessagemaker.h
noui.cpp	scripted-diff: Avoid incompatibility with CMake AUTOUIC feature	2022-06-14 10:38:51 +02:00
noui.h
outputtype.cpp	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
outputtype.h	refactor: Remove defunct attributes.h includes	2022-05-21 13:54:33 -05:00
pow.cpp
pow.h
prevector.h	Merge bitcoin/bitcoin#24962: prevector: enforce is_trivially_copyable_v	2022-05-16 16:25:47 +02:00
protocol.cpp	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
protocol.h	refactor: Use type-safe std::chrono for addrman time	2022-07-26 11:06:10 +02:00
psbt.cpp	psbt: Implement merge for Taproot fields	2022-06-27 16:47:48 -04:00
psbt.h	psbt: Fix unsigned integer overflow	2022-07-25 18:45:57 +02:00
pubkey.cpp	Use HashWriter where possible	2022-07-20 15:34:36 +02:00
pubkey.h	Add serialization methods to XOnlyPubKey	2022-06-27 16:47:48 -04:00
random.cpp	refactor: move compat.h into compat/	2022-07-20 10:34:46 +01:00
random.h	refactor: Make FEELER_SLEEP_WINDOW type safe (std::chrono)	2022-07-13 15:21:12 +02:00
randomenv.cpp	refactor: move compat.h into compat/	2022-07-20 10:34:46 +01:00
randomenv.h	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
rest.cpp	rest: clean-up for mempool endpoints	2022-08-05 10:28:11 -03:00
rest.h	Handle query string when parsing data format	2022-03-10 12:01:53 +01:00
reverse_iterator.h
scheduler.cpp	refactor: use C++11 default initializers	2022-05-17 17:18:58 +01:00
scheduler.h	Increase threadsafety annotation coverage	2022-05-12 02:25:55 +10:00
serialize.h	Add ChronoFormatter to serialize	2022-07-26 11:05:04 +02:00
shutdown.cpp	scripted-diff: Avoid incompatibility with CMake AUTOUIC feature	2022-06-14 10:38:51 +02:00
shutdown.h	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
signet.cpp	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
signet.h	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
span.h	scripted-diff: rename BytePtr to AsBytePtr	2022-04-26 09:41:45 +01:00
streams.h	streams: Add AutoFile without ser-type and ser-version	2022-06-29 10:31:53 +02:00
sync.cpp	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
sync.h	sync.h: Imply negative assertions when calling LOCK	2022-05-21 01:23:23 +10:00
threadinterrupt.cpp	Expose underlying clock in CThreadInterrupt	2022-07-13 15:20:49 +02:00
threadinterrupt.h	Expose underlying clock in CThreadInterrupt	2022-07-13 15:20:49 +02:00
threadsafety.h
timedata.cpp	scripted-diff: Avoid incompatibility with CMake AUTOUIC feature	2022-06-14 10:38:51 +02:00
timedata.h	addrman: Use system time instead of adjusted network time	2022-07-30 11:04:09 +02:00
tinyformat.h	Use C++17 [[fallthrough]] attribute, and drop -Wno-implicit-fallthrough	2021-07-05 08:59:38 +03:00
torcontrol.cpp	refactor: move compat.h into compat/	2022-07-20 10:34:46 +01:00
torcontrol.h	torcontrol: Query Tor for correct -onion configuration	2022-03-15 01:33:52 +00:00
txdb.cpp	CDiskBlockIndex: rename GetBlockHash() to ConstructBlockHash()	2022-07-22 12:45:07 +02:00
txdb.h	Add missing includes	2022-07-19 14:12:33 +02:00
txmempool.cpp	Remove ::g_max_datacarrier_bytes global	2022-08-02 15:29:16 +02:00
txmempool.h	Remove ::g_max_datacarrier_bytes global	2022-08-02 15:29:16 +02:00
txorphanage.cpp	refactor: log nEvicted message in LimitOrphans then return void	2022-07-28 14:39:45 +08:00
txorphanage.h	refactor: log nEvicted message in LimitOrphans then return void	2022-07-28 14:39:45 +08:00
txrequest.cpp	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00
txrequest.h
uint256.cpp
uint256.h	Use spans of std::byte in serialize	2022-01-02 11:40:31 +01:00
undo.h
validation.cpp	Merge bitcoin/bitcoin#25023: Remove unused SetTip(nullptr) code	2022-08-04 16:48:14 +01:00
validation.h	validationcaches: Use size_t for sizes	2022-08-03 12:03:28 -04:00
validationinterface.cpp	refactor: Remove defunct attributes.h includes	2022-05-21 13:54:33 -05:00
validationinterface.h	scripted-diff: Rename MainSignalsInstance() class to MainSignalsImpl()	2022-05-09 18:35:44 +02:00
version.h
versionbits.cpp	Sanity assert GetAncestor() != nullptr where appropriate	2022-05-05 15:55:44 +02:00
versionbits.h	Increase threadsafety annotation coverage	2022-05-12 02:25:55 +10:00
walletinitinterface.h	Add src/node/* code to node:: namespace	2022-01-06 22:14:16 -05:00
warnings.cpp	scripted-diff: Convert global Mutexes to GlobalMutexes	2022-05-21 01:23:23 +10:00
warnings.h	scripted-diff: Bump copyright headers	2021-12-30 19:36:57 +02:00