fa5674c264 fuzz: Fix off-by-one in package_rbf target (MarcoFalke)
Pull request description:
Running the while loop up to `NUM_ITERS` times may set `iter` to `g_outpoints.size()`, which will then lead to an out-of-bounds read.
There was an assert, which I guess tried to catch this, but the condition in the assert was wrong as well.
Fix all issues by replacing the broken assert with the internal and correct check inside `std::vector::at` and by limiting `iter` to `NUM_ITERS` in the while loop.
Fixes https://github.com/bitcoin/bitcoin/issues/32121
ACKs for top commit:
glozow:
ACK fa5674c264
brunoerg:
code review ACK fa5674c264
Tree-SHA512: 91b849ce969fd25c0ff8c90c2908d3096c77607d8e5fd81201ef33d88a57760199618174b8a6fd634cb5ef2a9068e94703b5c963ca473bd96a14d4bf9ec835cb
https://github.com/bitcoin/bitcoin/pull/30746#discussion_r1817851827 introduced an unsequenced operations with side-effects - which is undefined behavior, i.e. the right hand side can be evaluated before the left hand side, which happens to mutate it.
Tried:
```
clang++ --analyze -std=c++20 -I./src -I./src/test -I./src/test/fuzz src/test/fuzz/base_encode_decode.cpp src/psbt.cpp
```
but it didn't warn about UB.
Grepped for similar ones, but could find any other one in the codebase:
> grep -rnE --include='*.cpp' --include='*.h' '\b(\w+)\(([^)]*\b(\w+)\b[^)]*)\)\s*==\s*\3\.' .
```
./src/test/arith_uint256_tests.cpp:373: BOOST_CHECK(R1L.GetHex() == R1L.ToString());
./src/test/arith_uint256_tests.cpp:374: BOOST_CHECK(R2L.GetHex() == R2L.ToString());
./src/test/arith_uint256_tests.cpp:375: BOOST_CHECK(OneL.GetHex() == OneL.ToString());
./src/test/arith_uint256_tests.cpp:376: BOOST_CHECK(MaxL.GetHex() == MaxL.ToString());
./src/test/fuzz/cluster_linearize.cpp:565: assert(depgraph.FeeRate(best_anc.transactions) == best_anc.feerate);
./src/test/fuzz/cluster_linearize.cpp:646: assert(depgraph.FeeRate(found.transactions) == found.feerate);
./src/test/fuzz/cluster_linearize.cpp:765: assert(depgraph.FeeRate(chunk_info.transactions) == chunk_info.feerate);
./src/test/fuzz/base_encode_decode.cpp:95: assert(DecodeBase64PSBT(psbt, random_string, error) == error.empty());
./src/test/fuzz/key.cpp:102: assert(pubkey.data() == pubkey.begin());
./src/test/skiplist_tests.cpp:42: BOOST_CHECK(vIndex[from].GetAncestor(0) == vIndex.data());
./src/script/signingprovider.cpp:535: ComputeTapbranchHash(node.sub[1]->hash, node.sub[1]->hash) == node.hash) {
./src/pubkey.h:78: return vch.size() > 0 && GetLen(vch[0]) == vch.size();
./src/cluster_linearize.h:881: Assume(elem.inc.feerate.IsEmpty() == elem.pot_feerate.IsEmpty());
```
Hodlinator deduced the UB on Windows in https://github.com/bitcoin/bitcoin/issues/32135#issuecomment-2751723855
Co-authored-by: Hodlinator <172445034+hodlinator@users.noreply.github.com>
226d81f8b7 mining: drop unused -nFees and sigops from CBlockTemplate (Sjors Provoost)
53ad845fb9 test: check fees and sigops in getblocktemplate (Sjors Provoost)
Pull request description:
For the coinbase `vTxFees` used a dummy value of -nFees.
Similarly the first `vTxSigOpsCost` entry was calculated from
the dummy coinbase transaction.
This was introduced in #2115, but the values were never returned by the RPC or used in a test.
Drop 'm and add code comments to prevent confusion.
This PR also adds test coverage for the `fees` and `sigops` fields in `getblocktemplate`, so it closes#32053.
ACKs for top commit:
ismaelsadeeq:
re-ACK 226d81f8b7
ryanofsky:
Code review ACK 226d81f8b7. New test was added since last review, which seems very cleanly written and fixes some missing coverage.
glozow:
ACK 226d81f8b7
Tree-SHA512: 79c534e6bc4810d29114b04dd6db798877732cb473e773bf3cc28f83d14ee3982392587bd0baa39857bd53a79eae3b730d7a7029b08a9b6c3b5c51f86657ca5d
Replace test_rpcwhitelistdefault_0_no_permissions and
test_rpcwhitelistdefault_1_no_permissions with a single
test_rpcwhitelistdefault_permissions function.
Checking for IsArgSet before calling GetArg while providing an arbitrary
default value as fallback is both confusing and fragile.
It is confusing, because the provided fallback is dead code. So it would
be better to just call GetArg without a fallback.
Even better would be to provide the true fallback value and sanitize it
as if it were user-input, but this can be done in a follow-up.
Removing the redundant call to IsArgSet will have to be done either way,
so do it now.
Checking for IsArgSet before calling GetArg while providing the args
default value as fallback is both confusing and fragile.
It is confusing, because the provided fallback is dead code. So it would
be better to just call GetArg without a fallback.
However, ignoring the fallback value is fragile, because it would not be
sanitized.
Fix all issues by sanitizing the fallback value.
288481aabd doc: clarify that testnet min-difficulty is not optional (Sjors Provoost)
Pull request description:
When 20 minutes have gone by on testnet3 or testnet4, the next block `MUST` have difficulty 1. I've seen people be confused about this several times now in recent months. It doesn't help that the code comment is wrong. So fixing that.
The reason is that `nBits` must match exactly:
e568c1dd13/src/validation.cpp (L4212-L4215)
ACKs for top commit:
fjahr:
ACK 288481aabd
kevkevinpal:
ACK [288481a](288481aabd)
Tree-SHA512: 17d426301f386fa5810cceedfdb20a3523ab3ac2f17257ca7a525edd869fa409b150eff4cc258b27adecd0ded1c18ff48a9998fc9caed2faa461e410d4c5a884
6869fb4170 net: Block v2->v1 transport downgrade if !CConnman::fNetworkActive (Hodlinator)
Pull request description:
We might have just set `CNode::fDisconnect` in the first loop because of `!CConnman::fNetworkActive`.
Attempting to reconnect using v1 transport just because `fNetworkActive` was set to `false` at the "right" stage in the v2 handshake does not make sense.
Issue [discovered](https://github.com/bitcoin/bitcoin/pull/31633#discussion_r1930908304) by davidgumberg.
ACKs for top commit:
davidgumberg:
Tested and Reviewed ACK 6869fb4170
mabu44:
ACK 6869fb4170
stratospher:
ACK 6869fb4. I've reviewed the code but don't have strong preference for this branch vs master since only functional change is just a single log not being printed in a low probability scenario (we happen to be attempting v2 connection when P2P network activity is being turned off).
vasild:
ACK 6869fb4170
Tree-SHA512: 54f596e54c5a6546f2c3fec2609aa8d10dec3adcf1001ca16666d8b374b8d79d64397f46c90d9b3915b4e91a5041b6ced3044fd2a5b4fb4aa7282eb51f61296a
a015b7e13d test: Add expected result assertions (yancy)
Pull request description:
~This is a trivial addition to the test suit, however it shouldn't be required to add debug statements and manually run the tests if someone needs to know the results of this test.~
Add an assertion for the values returned. The goal of the test is to show that a minimal weight selection of UTXOs is returned by coin-grinder. Since there are multiple possible solutions, the added assertion shows that coin-grinder finds the solution with the lowest weight. Without this assertion, it's ambiguous whether or not coin-grinder is returning the solution with the lowest weight.
Remove the check that a result is returned since the expected result assertion implies a result.
ACKs for top commit:
janb84:
re ACK [a015b7e](a015b7e13d)
murchandamus:
ACK a015b7e13d
Tree-SHA512: ee3c2688b4a4a07ab209f7655c3956e62a1084419df5e87c27d751a38ff64d4c3457df2317f8077149a6947cdb05b249975de2b8f0e18ca8b17b41f4735fb1c6
2f2ab47bf7 Release notes (Pol Espinasa)
bf194c920c wallet, rpc: deprecate settxfee and paytxfee (Pol Espinasa)
Pull request description:
**Summary**
This PR deprecates the settxfee RPC and paytxfee setting, marking it for removal in Bitcoin Core 31.0.
**Motivation**
The PR was initially motivated by https://github.com/bitcoin/bitcoin/issues/31088. The intention was to create a new function `settxfeerate` to allow users to set a static fee rate in `sat/vB` instead of `btc/kvB`.
The `settxfee` RPC allows users to set a static fee rate for all transactions created by the wallet. However, in a dynamic fee environment, this can lead to poor fee choices, either overpaying when the mempool is empty or underpaying when congestion is high. The preferred approach is to rely on fee estimation, which is designed to adapt to network conditions, and is the one by default. Same argument apply for `paytxfee` setting.
During discussion the consensus was that static fee settings are a footgun and that users should instead specify the fee rate per transaction if they don't want to rely on the fee estimation. Given this, rather than introducing a `settxfeerate` alternative, this PR goes towards removing `settxfee` and `paytxfee` entirely.
**Key Changes**
`settxfee` and `paytxfee` is now deprecated and will be removed in Bitcoin Core 31.0.
Users should rely on fee estimation or explicitly specify a fee rate when constructing transactions.
**Impact on Users**
If users currently use settxfee or paytxfee, they should transition to specifying fees per transaction.
No immediate breakage in 30.0 (must use `-deprecatedrpc=settxfee`), but `settxfee` and `paytxfee` will be removed in 31.0.
**Alternative Approaches Considered**
A settxfeerate alternative (using sat/vB) was initially proposed but ultimately rejected in favor of deprecating static fee setting entirely.
**Notes for removal**
- When removing paytxfee we should also update txconfirmtarget startup option help text.
- Get back the comment from `rpc_deprecated.py` test. [+info](https://github.com/bitcoin/bitcoin/pull/31278#discussion_r1998876768)
ACKs for top commit:
fjahr:
re-ACK 2f2ab47bf7
ismaelsadeeq:
re-ACK 2f2ab47bf7
rkrux:
Concept and utACK 2f2ab47bf7
Tree-SHA512: 0272812cbe5a519737c5d0683acc2072e67559792b4a6472bca8b23426e5ce9e88a3a1eba987feda70a082b8b474b3126893848628d7bf11e1520357b18c8d3e
In order to make it possible for higher layers to compare transaction quality
(ordering within the implicit total ordering on the mempool), expose a comparison
function and test it.
Before this commit, if a TxGraph::Ref object is destroyed, it becomes impossible
to refer to, but the actual corresponding transaction node in the TxGraph remains,
and remains indefinitely as there is no way to remove it.
Fix this by making the destruction of TxGraph::Ref trigger immediate removal of
the corresponding transaction in TxGraph, both in main and staging if it exists.
In order to make it easy to evaluate proposed changes to a TxGraph, introduce a
"staging" mode, where mutators (AddTransaction, AddDependency, RemoveTransaction)
do not modify the actual graph, but just a staging version of it. That staging
graph can then be commited (replacing the main one with it), or aborted (discarding
the staging).
Move a number of related modifications to TxGraphImpl into a separate
function for removal of transactions. This is preparation for a later
commit where this will be useful in more than one place.
This is a preparation for a next commit where a TxGraph will start representing
potentially two distinct graphs (a main one, and a staging one with proposed
changes).
When transactions are removed from the tail of a cluster, we know the existing
linearization remains acceptable (if it already was), but may just need splitting
and postlinearization, so special case these into separate quality levels.
Chunk-based information (primarily, chunk feerates) are never accessed without
first bringing the relevant Clusters to an "acceptable" quality level. Thus,
while operations are ongoing and Clusters are not acceptable, we can omit
computing the chunkings and chunk feerates for Clusters.
Since m_deps_to_add has been sorted by child Cluster* already, all dependencies
with the same child will be processed consecutively. Take advantage of this by
remember the last partition merged with, and reusing that if applicable.
The m_deps_to_add vector is sorted by child Cluster*, which matches the
order of an_clusters. This means we can walk through m_deps_to_add while
doing the representative lookups for an_clusters, and reuse them.
Instead of leaving the responsibility on higher layers to guarantee that
no connected component within TxGraph (a barely exposed concept, except through
GetCluster()) exceeds the cluster count limit, move this responsibility to
TxGraph itself:
* TxGraph retains a cluster count limit, but it becomes configurable at construction
time (this primarily helps with testing that it is properly enforced).
* It is always allowed to perform mutators on TxGraph, even if they would cause the
cluster count limit to be exceeded. Instead, TxGraph exposes an IsOversized()
function, which queries whether it is in a special "oversize" state.
* During oversize state, many inspectors are unavailable, but mutators remain valid,
so the higher layer can "fix" the oversize state before continuing.
Instead construct a single vector with the list of all clusters in all groups,
and then store per-group offset/range in that list.
For dependencies, reuse m_deps_to_add, and store offset/range into that.
To make testing more powerful, expose a function to perform an internal sanity
check on the state of a TxGraph. This is especially important as TxGraphImpl
contains many redundantly represented pieces of information:
* graph contains clusters, which refer to entries, but the entries refer back
* graph maintains pointers to Ref objects, which point back to the graph.
This lets us make sure they are always in sync.
This adds a simulation fuzz test for txgraph, by comparing with a naive
reimplementation that models the entire graph as a single DepGraph, and
clusters in TxGraph as connected components within that DepGraph.
This adds an initial version of the txgraph module, with the TxGraph class.
It encapsulates knowledge about the fees, sizes, and dependencies between all
mempool transactions, but nothing else.
In particular, it lacks knowledge about txids, inputs, outputs, CTransactions,
... and so forth. Instead, it exposes a generic TxGraph::Ref type to reference
nodes in the TxGraph, which can be passed around and stored by layers on top.
Since cluster_linearize.h does not actually have a Cluster type anymore, it is more
appropriate to rename the index type to DepGraphIndex.
-BEGIN VERIFY SCRIPT-
sed -i 's/Data type to represent transaction indices in clusters./Data type to represent transaction indices in DepGraphs and the clusters they represent./' $(git grep -l 'using ClusterIndex')
sed -i 's|\<ClusterIndex\>|DepGraphIndex|g' $(git grep -l 'ClusterIndex')
-END VERIFY SCRIPT-
This function takes an existing ordering for transactions in a DepGraph, and
makes it a valid linearization for it (i.e., topological). Any topological
prefix of the input remains untouched.
Move the comparison to hashWatchedChain inside the while loop.
Although this early return prevents the GetTransactionsUpdated()
call in cases where the tip updates, it's only done to improve
readability. The check itself is very cheap (although a more
useful check might not be).
Also add code comments.
- return null on shutdown instead of the last tip
- ignore timeout value node initialization
This allows consumers of BlockTemplate to safely
assume that a tip is connected, instead of having
to account for startup and early shutdown scenarios.
