In order to make it possible for higher layers to compare transaction quality
(ordering within the implicit total ordering on the mempool), expose a comparison
function and test it.
Before this commit, if a TxGraph::Ref object is destroyed, it becomes impossible
to refer to, but the actual corresponding transaction node in the TxGraph remains,
and remains indefinitely as there is no way to remove it.
Fix this by making the destruction of TxGraph::Ref trigger immediate removal of
the corresponding transaction in TxGraph, both in main and staging if it exists.
In order to make it easy to evaluate proposed changes to a TxGraph, introduce a
"staging" mode, where mutators (AddTransaction, AddDependency, RemoveTransaction)
do not modify the actual graph, but just a staging version of it. That staging
graph can then be commited (replacing the main one with it), or aborted (discarding
the staging).
Move a number of related modifications to TxGraphImpl into a separate
function for removal of transactions. This is preparation for a later
commit where this will be useful in more than one place.
This is a preparation for a next commit where a TxGraph will start representing
potentially two distinct graphs (a main one, and a staging one with proposed
changes).
When transactions are removed from the tail of a cluster, we know the existing
linearization remains acceptable (if it already was), but may just need splitting
and postlinearization, so special case these into separate quality levels.
Chunk-based information (primarily, chunk feerates) are never accessed without
first bringing the relevant Clusters to an "acceptable" quality level. Thus,
while operations are ongoing and Clusters are not acceptable, we can omit
computing the chunkings and chunk feerates for Clusters.
Since m_deps_to_add has been sorted by child Cluster* already, all dependencies
with the same child will be processed consecutively. Take advantage of this by
remember the last partition merged with, and reusing that if applicable.
The m_deps_to_add vector is sorted by child Cluster*, which matches the
order of an_clusters. This means we can walk through m_deps_to_add while
doing the representative lookups for an_clusters, and reuse them.
Instead of leaving the responsibility on higher layers to guarantee that
no connected component within TxGraph (a barely exposed concept, except through
GetCluster()) exceeds the cluster count limit, move this responsibility to
TxGraph itself:
* TxGraph retains a cluster count limit, but it becomes configurable at construction
time (this primarily helps with testing that it is properly enforced).
* It is always allowed to perform mutators on TxGraph, even if they would cause the
cluster count limit to be exceeded. Instead, TxGraph exposes an IsOversized()
function, which queries whether it is in a special "oversize" state.
* During oversize state, many inspectors are unavailable, but mutators remain valid,
so the higher layer can "fix" the oversize state before continuing.
Instead construct a single vector with the list of all clusters in all groups,
and then store per-group offset/range in that list.
For dependencies, reuse m_deps_to_add, and store offset/range into that.
To make testing more powerful, expose a function to perform an internal sanity
check on the state of a TxGraph. This is especially important as TxGraphImpl
contains many redundantly represented pieces of information:
* graph contains clusters, which refer to entries, but the entries refer back
* graph maintains pointers to Ref objects, which point back to the graph.
This lets us make sure they are always in sync.
This adds a simulation fuzz test for txgraph, by comparing with a naive
reimplementation that models the entire graph as a single DepGraph, and
clusters in TxGraph as connected components within that DepGraph.
This adds an initial version of the txgraph module, with the TxGraph class.
It encapsulates knowledge about the fees, sizes, and dependencies between all
mempool transactions, but nothing else.
In particular, it lacks knowledge about txids, inputs, outputs, CTransactions,
... and so forth. Instead, it exposes a generic TxGraph::Ref type to reference
nodes in the TxGraph, which can be passed around and stored by layers on top.
Since cluster_linearize.h does not actually have a Cluster type anymore, it is more
appropriate to rename the index type to DepGraphIndex.
-BEGIN VERIFY SCRIPT-
sed -i 's/Data type to represent transaction indices in clusters./Data type to represent transaction indices in DepGraphs and the clusters they represent./' $(git grep -l 'using ClusterIndex')
sed -i 's|\<ClusterIndex\>|DepGraphIndex|g' $(git grep -l 'ClusterIndex')
-END VERIFY SCRIPT-
This function takes an existing ordering for transactions in a DepGraph, and
makes it a valid linearization for it (i.e., topological). Any topological
prefix of the input remains untouched.
d423fd9ec8 cli, bugfix: for -getinfo, replace IsArgSet() with GetBoolArg() (Jon Atack)
e99e41b307 cli, refactor: simplify public-only classes with structs (Jon Atack)
fdbfd250fb cli, refactor: deduplicate NetworkStringToId() (Jon Atack)
be82139b2a cli, refactor: simplify DetailsRequested() (Jon Atack)
Pull request description:
These have been accumulating over the past few years.
Each is described in its commit message.
ACKs for top commit:
pablomartin4btc:
re-ACK d423fd9ec8
hodlinator:
Code review ACK d423fd9ec8
l0rinc:
ACK d423fd9ec8
ryanofsky:
Code review ACK d423fd9ec8, just running clang-format and updating commit messages since last review
Tree-SHA512: a8e5f7827cef308186d5a7c3a627d2cd8f57437f4465d181986e5d3274ff0e2b9faac252dd55d9257d66a7aa99fca62b3000cdc0988d23385df20ff1f870a046
eb0724f0de doc: banman: reference past vuln due to unbounded banlist (Antoine Poinsot)
ad616b6c01 doc: net: mention past vulnerability as rationale to limit incoming message size (Antoine Poinsot)
4489117c3f doc: txrequest: point to past censorship vulnerability in tx re-request handling (Antoine Poinsot)
68ac9542c4 doc: net_proc: reference past DoS vulnerability in orphan processing (Antoine Poinsot)
c02d9f6dd5 doc: net_proc: reference past defect regarding invalid GETDATA types (Antoine Poinsot)
5e3d9f21df doc: validation: add a reference to historical header spam vulnerability (Antoine Poinsot)
Pull request description:
It is useful when reading code to have context about why it is written or behaves the way it does. Some instances in this PR may seem obvious but i think nonetheless offer important context to anyone willing to change (or review a change to) this code.
ACKs for top commit:
ryanofsky:
Code review ACK eb0724f0de. No changes since last review other than rebase
Tree-SHA512: 271902f45b8130d44153d793bc1096cd22b6ce05494e67c665a5bc45754e3fc72573d303ec8fc7db4098d473760282ddbf0c1cf316947539501dfd8d7d5b8828
7edaf8b64c Benchmark Chainstate::ConnectBlock duration (Eunovo)
Pull request description:
Introduce benchmarks to evaluate ConnectBlock performance for:
- Blocks containing only Schnorr signatures
- Blocks containing both Schnorr and ECDSA signatures
- Blocks containing only ECDSA signatures
The benchmarks in this PR, focus on signature validation. Additional benchmarks may be added in the future to assess other aspects of ConnectBlock.
This is the first step toward implementing Batch Verification of Schnorr Signatures in Core. It provides a way to test and measure the performance improvements of batch verification on Core.
For more details on batch validation, refer to the [batch-verify module on secp](https://github.com/bitcoin-core/secp256k1/pull/1134) and [batch-verify on core](https://github.com/bitcoin/bitcoin/pull/29491).
ACKs for top commit:
josibake:
reACK 7edaf8b64c
fjahr:
utACK 7edaf8b64c
l0rinc:
ACK 7edaf8b64c
Tree-SHA512: 883c8a5e4e4de401ffb9ac9b6789b7fe0737afefbdaf02c6d7e1645392efc4f0d2d28b423ba7e34366a33608e0835793f5e7a1312b5c8063de14446319529cc7
fa7a40d952 contrib: Print deterministic-coverage runs (MarcoFalke)
fa751639fb contrib: Make deterministic-coverage error messages more readable (MarcoFalke)
Pull request description:
This is almost a "refactor" to tidy up the error messages. Apart from the messages, the behavior of the tools is identical.
This was requested in https://github.com/bitcoin/bitcoin/pull/31901#discussion_r1969502508.
Previously, the tool would abort the program early on some errors. Now, the tool propagates an `std::result::Result::Err` up to `main` via an early return. Getting rid of the aborts also allows to drop the `RUST_BACKTRACE` env setting.
ACKs for top commit:
hodlinator:
re-ACK fa7a40d952
janb84:
ACK [fa7a40d](fa7a40d952)
Tree-SHA512: 6c97861306e2fececa14b2d12deafb78995fc2bcf75e4e22773cb0ab4231de78834db9f1f89b30c49d77499433b1c16c1d90b97eb4069c81855bd2a7944b554f
ba82240553 fuzz: split `coinselection` harness (brunoerg)
Pull request description:
This PR splits the `coinselection` fuzz harness into 3 targets (`coinselection_bnb`, `coinselection_knapsack`, `coinselection_srd`). The goal is to be able to fuzz each algorithm separately (to avoid performance issues) and also all of them together.
ACKs for top commit:
janb84:
Tested ACK [ba82240](ba82240553)
maflcko:
review ACK ba82240553👐
marcofleon:
reACK ba82240553
zaidmstrr:
reACK [ba82240](ba82240553)
Tree-SHA512: 277cffd524e57d286dbbbcb2aa0a9f1d720b4c56331dfb0f4425e1666246330616508e47977da23f28a72705aa142bbaf536e2cf7fe4703a2cd2e4b2fd441d9d
63b534f97e fuzz: sanity check hardcoded snapshot in utxo_snapshot target (Antoine Poinsot)
3b85eba83a test util: split up ConnectBlock from MineBlock (Antoine Poinsot)
d1527f6b88 qa: correct off-by-one in utxo snapshot fuzz target (Antoine Poinsot)
Pull request description:
The assumeutxo data for the fuzz target could change and invalidate the hash silently, preventing the fuzz target from reaching some code paths. Fix this by introducing a unit test which would break if the snapshot data the fuzz target relies on were to change.
In implementing this i noticed the height used for coins in the fuzz target is actually off-by-one (as if the first block in the created chain was the genesis but it's block `1`), so fix that too.
ACKs for top commit:
mzumsande:
Code Review ACK 63b534f97e
fjahr:
tACK 63b534f97e
Tree-SHA512: 2399b6e74db9b78aab8efba67c57a405d2d7d880ae3b7d8518a1c96cc6266f61f5e77722cd999adeac5d3e03e73d84cf9ae7bdbcc0afae198cc87049dde4012f
f708498293 torcontrol: Limit reconnect timeout to max seconds and log delay in whole seconds (Eval EXEC)
Pull request description:
I'm reviewing the Tor controller's reconnect-related code and noticed that the reconnect timeout had no limit. This could lead to excessively long delays.
This PR introduces a maximum reconnect timeout of 600 seconds (10 minutes) to prevent excessive delays in reconnection attempts. It also updates the log message to display the retry delay in whole seconds for better readability.
ACKs for top commit:
mabu44:
ACK f708498293
laanwj:
Code review ACK f708498293
luke-jr:
utACK f708498293
Tree-SHA512: 8f18c6c84da6b4e7328638fd74539fbd3dd44f46c5107638de56b72fc079487690861199ceba1197ca34421dcedf79a1ca6bacf2a918a683e71bce9ff710b5d4
fa310cc6f4 test: Fix intermittent issue in p2p_orphan_handling.py (MarcoFalke)
Pull request description:
The test may fail intermittently when the `net` thread is lagging while calling `DeleteNode`. This may result in a split `getdata`, meaning that `peer2.wait_for_parent_requests([int(parent_peekaboo_AB["txid"], 16), int(parent_missing["txid"], 16)])` fails.
Fix it by adding a sync on the `net` thread.
Fixes#31700
ACKs for top commit:
mzumsande:
Code Review ACK fa310cc6f4
Tree-SHA512: e4a58093ab5b9e280c479b845fecb5d228e65519ea3dc2111b393202225fd0feded423e8812452454b6b9348cb37a9c1b01b9d1b1802e9f4aa76b9e56b4b54ef
fa4fb6a8f1 fuzz: Use serial task runner to increase fuzz stability (MarcoFalke)
Pull request description:
Leaking a scheduler with a non-empty queue from the fuzz initialization phase into the fuzz target execution phase is problematic, because it messes with coverage data. This in turn is problematic, because it leads to:
* Decrease in fuzz target execution stability (non-determinism when running the fuzz target).
* Decrease in fuzz input merge stability (non-determinism when selecting a minimum set of fuzz input to reach maximum coverage), which leads to qa-assets bloat.
Fix one such issue. Tracking issue: https://github.com/bitcoin/bitcoin/issues/29018
Can be tested via: `RUST_BACKTRACE=1 cargo run --manifest-path ./contrib/devtools/deterministic-fuzz-coverage/Cargo.toml -- $PWD/bld-cmake $PWD/../b-c-qa-assets/fuzz_corpora/ partially_downloaded_block`.
The failure is non-deterministic (obviously) and will show coverage in validation signals such as `UpdatedBlockTip` before this change and will have this one fixed after this change.
ACKs for top commit:
marcofleon:
ACK fa4fb6a8f1
dergoegge:
Code review ACK fa4fb6a8f1
Tree-SHA512: fd1f66562c1d3c21553c7dd324399cdc16faa2fedfdb8e7544ea6a68b8b356e7c81d81815ecf70e0d334307dab6b275c1889b3b889b6f15eec514beee22c95f4
d190f0facc test, contrib: Fix signer/miner command line escaping (Ryan Ofsky)
0d2eefca8b test, refactor: Add TestNode.binaries to hold binary paths (Ryan Ofsky)
Pull request description:
Add new `TestNode.binaries` object to manage paths to bitcoin binaries.
The `binaries` object makes it possible for the test framework to exercise the bitcoin wrapper executable introduced in https://github.com/bitcoin/bitcoin/pull/31375 and also makes it easier in general to add new binaries, and new options and environment variables controlling how they are invoked, because logic for invoking them that was previously spread out is now consolidated in one place.
These changes were originally part of #31375 but made that PR harder to review because they were unrelated to the other changes there. If this PR can get merged first, python changes in #31375 will be simple, and the test framework changes here should also get a higher quality review.
ACKs for top commit:
maflcko:
re-review-ACK d190f0facc🍓
Sjors:
ACK d190f0facc
vasild:
ACK d190f0facc
Tree-SHA512: 5a6c0553cd2822585810d827ef1c1772cbf3097d3336daf733f8378dd3da79c00fc3721e50ed0f7455908fbd7a509e9739f9be33f588d6bc1aaa400b9d75c650
ffff4a293a bench: Update span-serialize comment (MarcoFalke)
fa4d6ec97b refactor: Avoid false-positive gcc warning (MarcoFalke)
fa942332b4 scripted-diff: Bump copyright headers after std::span changes (MarcoFalke)
fa0c6b7179 refactor: Remove unused Span alias (MarcoFalke)
fade0b5e5e scripted-diff: Use std::span over Span (MarcoFalke)
fadccc26c0 refactor: Make Span an alias of std::span (MarcoFalke)
fa27e36717 test: Fix broken span_tests (MarcoFalke)
fadf02ef8b refactor: Return std::span from MakeUCharSpan (MarcoFalke)
fa720b94be refactor: Return std::span from MakeByteSpan (MarcoFalke)
Pull request description:
`Span` has some issues:
* It does not support fixed-size spans, which are available through `std::span`.
* It is confusing to have it available and in use at the same time with `std::span`.
* It does not obey the standard library iterator build hardening flags. See https://github.com/bitcoin/bitcoin/issues/31272 for a discussion. For example, this allows to catch issues like the one fixed in commit fabeca3458.
Both types are type-safe and can even implicitly convert into each other in most contexts.
However, exclusively using `std::span` seems less confusing, so do it here with a scripted-diff.
ACKs for top commit:
l0rinc:
reACK ffff4a293a
theuni:
ACK ffff4a293a.
Tree-SHA512: 9cc2f1f43551e2c07cc09f38b1f27d11e57e9e9bc0c6138c8fddd0cef54b91acd8b14711205ff949be874294a121910d0aceffe0e8914c4cff07f1e0e87ad5b8
fac3d93c2b fuzz: Speed up *_package_eval fuzz targets a bit (MarcoFalke)
fa40fd043a fuzz: [refactor] Avoid confusing c-style cast (MarcoFalke)
Pull request description:
Each target is at least 10% faster for me when running over the current set of qa-assets, which seems nice.
The changes `outpoints_value` from a map to an unordered map, which is safe, because the element order is not used in the fuzz test and the map is only used for lookup.
(`mempool_outpoints` can't be changed, because the order matters here. Using unordered_set here may result in a non-deterministic fuzz target, given the same fuzz input.)
ACKs for top commit:
l0rinc:
ACK fac3d93c2b
dergoegge:
Code review ACK fac3d93c2b
Tree-SHA512: 8ae5d4e281505aff76a4003d6e9ea388dbb73860e167385bd6a0a201b3acc939db29ee212594952a9e80e85b3cc4cd726ce6dd49551f74013cb4da8a15cbdfb3
c8fab35617 ci: remove -Wno-error=deprecated-declarations from ASAN (fanquake)
a130bbd154 Squashed 'src/leveldb/' changes from 04b5790928..4188247086 (fanquake)
Pull request description:
Cherry-picks two commits from upstream (302786e211, e829478c6a), which remove the usage of `std::aligned_storage/std::aligned_union`.
Note the first cherry-pick is not clean, because due to Google tooling issues, it accidently contained a revert of the prior two commits. See https://github.com/google/leveldb/pull/1249 for more details.
Also see https://issues.chromium.org/issues/388068052, although note that they [reverted the roll to latest leveldb](https://issues.chromium.org/issues/388068052#comment9). I'm guessing due to the acidental reversion issue above.
ACKs for top commit:
l0rinc:
ACK c8fab35617
darosior:
ACK c8fab35617 -- checked it's a clean subtree pull from https://github.com/bitcoin-core/leveldb-subtree/tree/bitcoin-fork
dergoegge:
utACK c8fab35617
Tree-SHA512: 966e61b9ac88af5ae7bf71514bfd5bbdbd8c38c7af65feb6d5e4415062dcff5896dc33fe968ded3462cc599abd921d49ee8336db3e12ed3f59c91ceb949317b7
387385ba1e test: replace assert with assert_equal and assert_greater_than (Chandra Pratap)
Pull request description:
In `test/functional/interface_usdt_net.py`, `assert_equal` is already used to check for equality between objects. Replace `assert.*==` with `assert_equal` and `assert.*>` with `assert_greater_than` to further easify debugging.
Relevant issue: #23119
ACKs for top commit:
maflcko:
lgtm ACK 387385ba1e
0xB10C:
had a quick look, lgtm ACK 387385ba1e
theStack:
utACK 387385ba1e
brunoerg:
code review ACK 387385ba1e
i-am-yuvi:
Great! ACK 387385ba1e
Tree-SHA512: 741a3d98288c9999f62bcbaa3806716b0519ec9b521e1e6e17aa458392245f6eff886af6cb601c66f2147e0265ff1eae57cea3dcfd67af93bef6dff25b056935
cec14ee47d test: switch wallet_crosschain.py to signet (Sjors Provoost)
9c2951541c test: drop testnet4 from wallet_crosschain.py (Sjors Provoost)
Pull request description:
It's sufficient to check only one test network, so this PR reverts the addition of testnet4 from #29775.
Testnet3 is deprecated. Instead of moving to testnet4, which might also be deprecated in the future, use signet.
ACKs for top commit:
fjahr:
utACK cec14ee47d
maflcko:
lgtm ACK cec14ee47d🌰
Tree-SHA512: c5aad6e7d251957f090145eac906f7985fddc3e3ba82df7184d72b961f9c856d324a1065ac98323b75501d136bd7b669fcc2565b9e66b0743eb3f3906ef37570
20fe41e9e8 test: avoid disk space warning for non-regtest (Sjors Provoost)
Pull request description:
`feature_config_args.py` incorrectly assumed that its testnet4 node would not log a disk space warning.
But when #31978 increased `m_assumed_blockchain_size` on testnet4 from 1 to 11 GiB, it triggered this bug on my RAM disk, see https://github.com/bitcoin/bitcoin/tree/master/test#speed-up-test-runs-with-a-ram-disk
This PR fixes the issue by using `-prune` which prevents the warning.
ACKs for top commit:
fjahr:
ACK 20fe41e9e8
maflcko:
lgtm ACK 20fe41e9e8
rkrux:
ACK 20fe41e9e8
Tree-SHA512: f4bbb3ede307e06bf097a3cf7a4099eacc9388e33f551e1d0c4c5f53747bfa593a4b22e5d2e713ce6dd8adf91602fade36fbec9cfc2b250a6b1cf09f11bc8473
In test/functional/interface_usdt_net.py, assert_equal is already
used to check for equality between objects. Replace 'assert.*=='
with 'assert_equal' and 'assert.*>' with 'assert_greater_than'
to further easify debugging.
Pass bitcoin binary command lines from test framework to signet/miner utility
using shell escaping so they are unambigous and don't get mangled if they
contain spaces.
This change is not needed for tests to pass currently, but is a useful change
to avoid CI failures in followup PR
https://github.com/bitcoin/bitcoin/pull/31375 and to avoid other bugs.
21e9d39a37 docs: add release notes for 31603 (brunoerg)
a8b548d75d test: `getdescriptorinfo`/`importdescriptors` with whitespace in pubkeys (brunoerg)
c7afca3d62 test: descriptor: check whitespace into keys (brunoerg)
cb722a3cea descriptor: check whitespace in ParsePubkeyInner (brunoerg)
50856695ef test: fix descriptors in `ismine_tests` (brunoerg)
Pull request description:
Currently, we successfully parse descriptors which contains spaces in the beginning or end of the public/private key within a fragment (e.g. `pk( KEY)`, `pk(KEY )` or `pk( KEY )`). I have noticed that one of the reasons is that the `DecodeBase58` function simply ignore these whitespaces.
This PR changes the `ParsePubkeyInner ` to reject pubkeys that contain a whitespace at the beginning and/or at the end. We will only check the whitespace in some RPCs (e.g. `importdescriptors`), but an already imported descriptor won't be affected by this check, especially because we store descriptors from `ToString`.
For context: https://github.com/brunoerg/bitcoinfuzz/issues/72
ACKs for top commit:
rkrux:
tACK 21e9d39a37
darosior:
re-ACK 21e9d39a37
sipa:
utACK 21e9d39a37
Tree-SHA512: 54f48a89a235517e5cdc29a46dceeb7dabbee93c7616a166288ff3f90131808eb0ece43b0797a11fe827a5f7bd51d65e3e75c16789b0a42020934cabb684cc8f
Add new TestNode.binaries object to manage paths to bitcoin binaries.
Having this object makes it possible for the test framework to exercise the
bitcoin wrapper executable introduced in
https://github.com/bitcoin/bitcoin/pull/31375 and also makes it easier to add
new binaries and options and environment variables controlling how they are
invoked, because logic for invoking them that was previously spread out is now
consolidated in one place.
Co-authored-by: Sjors Provoost <sjors@sprovoost.nl>
1f9b2e150c cmake: Require `zip` only for `deploy` target (Hennadii Stepanov)
0aeff29951 cmake: Check for `makensis` tool before using it (Hennadii Stepanov)
Pull request description:
For `x86_64-w64-mingw32` and `*-apple-darwin` targets, the optional `deploy` target requires dedicated tools: `makensis` and `zip`, respectively.
This PR introduces a uniform checks for those tools when attempting to build the `deploy` target, ensuring they are not required for configuring and building any other targets.
Here is an example of workflow for `x86_64-w64-mingw32`:
```
$ # `nsis` is not installed
$ cmake -B build -G "GNU Makefiles" --toolchain depends/x86_64-w64-mingw32/toolchain.cmake
$ cmake --build build -j $(nproc)
$ cmake --build build -t deploy
Error: NSIS not found.
Please install NSIS and/or ensure that its executable is accessible to the find_program() command—
for example, by setting the MAKENSIS_EXECUTABLE variable or another relevant CMake variable.
Then re-run cmake to regenerate the build system.
Built target deploy
$ sudo apt install nsis
$ cmake -B build
$ cmake --build build -t deploy
...
[100%] Generating bitcoin-win64-setup.exe
[100%] Built target deploy
```
Fixes https://github.com/bitcoin/bitcoin/issues/32018.
ACKs for top commit:
hodlinator:
re-ACK 1f9b2e150c
fanquake:
ACK 1f9b2e150c
Tree-SHA512: 5e2bd28a13bd8fa7c4ba8cf1756d200a4651afe83c463d76ece10027cca343e124eff97012a5368028f761df60f420ab891106b4e33b50045051d57c7464ff98
