687addaf13 test: add BIP-125 rule 5 testcase with default mempool (James O'Beirne)
6120e8e287 test: allow passing sequence through create_self_transfer_multi (James O'Beirne)
Pull request description:
Currently, we only test rule 5 of BIP-125 (replacement transactions cannot evict more than 100 transactions) by changing default mempool parameters to allow for more descendants. The current test works on a single transaction graph that has over 100 descendants.
This patch adds a test to exercise rule 5 using the default mempool parameters. The case is a little more sophisticated: instead of working on a single transaction graph, it uses a replacement transaction to "unite" several UTXOs which join independent transaction graphs. The total number of transactions in these graphs sum to more than the max allowable replacement.
I think the difference in transaction topology makes this a worthwhile testcase to have, setting aside the fact that this testcase works without having to use atypical mempool params.
See also: [relevant discussion from IRC](https://www.erisian.com.au/bitcoin-core-dev/log-2022-05-27.html#l-126)
ACKs for top commit:
laanwj:
Code review ACK 687addaf13
LarryRuane:
ACK 687addaf13
Tree-SHA512: e589aeaf9d6f137d546b7809f8795d6f6043d87b15e97c2efe85b42ce8b49d977ee7d79440c542ca4b0b5ca2de527488029841a1ffc0d96c5771897df4b3f324
MiniWallet's core method for creating txs (`create_self_transfer`)
right now always executes the `testmempoolaccept` RPC to check for
mempool validity or invalidity. In some test cases where we use
MiniWallet to create a huge number of transactions this can lead
to performance issues (e.g. feature_fee_estimation.py where the
execution time after MiniWallet usage almost doubled). Providing
the possibility to skip the mempool checks is a mitigation for
this.
master branch:
$ time ./test/functional/feature_fee_estimation.py
real 3m20.771s
user 2m52.360s
sys 0m39.340s
PR branch:
$ time ./test/functional/feature_fee_estimation.py
real 2m1.386s
user 1m42.510s
sys 0m22.980s
Also explicitly rehash in the cases where we modify a tx after signing
in feature_csv_activation.py. Parts of this test relied on the fact that
rehashing of transactions is done in the course of calculating a block's
merkle root (`calc_merkle_root`), which only works if no hash was
calculated before due to a caching mechanism.
In the following commit the txid in MiniWallet is calculated via
`rehash()`, i.e. this doesn't work anymore and we always have to
explicitely have the right hash before we calculate the merkle root.
2b6dd4e75b test: use MiniWallet for mempool_package_onemore.py (Sebastian Falbesoner)
eb3c5c4ef2 test: MiniWallet: add helper methods `{send,create}_self_transfer_multi` (Sebastian Falbesoner)
Pull request description:
This PR enables one more of the non-wallet functional tests (mempool_package_onemore.py) to be run even with the Bitcoin Core wallet disabled by using the MiniWallet instead, as proposed in https://github.com/bitcoin/bitcoin/issues/20078. For this purpose helper methods `MiniWallet.{create,send}_self_transfer_multi` are introduced which serve as a replacement for `chain_transaction`. With this, it should be also quite straight-forward to change the larger related test `mempool_packages.py` to use MiniWallet.
ACKs for top commit:
MarcoFalke:
ACK 2b6dd4e75b💾
Tree-SHA512: 0c97fa0519ca5eaa6df8953a04678aa8a6a66905a82db6ff40042a675d0c0682aee829a48db84e4e7983d8f766875021f0d39d65e12889342610b8861bc29cd5
2726b60a3a test: use MiniWallet for rpc_createmultisig.py (Ayush Sharma)
Pull request description:
This PR enables one of the non-wallet functional tests (rpc_createmultisig.py) to be run even with the Bitcoin Core wallet disabled by using the MiniWallet instead, as proposed in #20078 .
ACKs for top commit:
danielabrozzoni:
re-ACK 2726b60a3a
Tree-SHA512: fb0ef22d3f1c161ca5963cb19ce76533ac3941f15102fc0aa2286ef3bec48f219e5934d504b41976f9f295fb6ca582b737e0fea896df4eb964cdaba1b2c91650
If no `from_node` parameter is passed explicitely to the
`create_self_transfer` method, the test node passed in the course
of creating the MiniWallet instance is used. This seems to
be the main use-case in most of the current functional
tests, i.e. in many instances the calls can be shortened.
This serves as a replacement for the getnewaddress RPC if no wallet is
available. In addition to the address, it also returns the corresponding
public key and output script (scriptPubKey).
The previous diff touched most files in ./test/, so bump the headers to
avoid having to touch them again for a bump later.
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./test/
-END VERIFY SCRIPT-
The coin selection strategy for MiniWallet is quite straight-forward: simply
pick a single UTXO with the largest value:
self._utxos = sorted(self._utxos, key=lambda k: k['value'])
utxo_to_spend = utxo_to_spend or self._utxos.pop()
If there are several candidates with the same value, however, it is not clear
which one is taken. This can be particularly problematic for coinbase outputs
with fixed block subsidy, since spending could lead to a
'bad-txns-premature-spend-of-coinbase' reject if an UTXO from a too-recent
block is picked. Introduce block height as second criteria (saved in
self._utxos in the methods generate(...) and rescan_utxos(...)), in order to
avoid potential issues with coinbases that are not matured yet.
fac62e6ff5 test: Delete generate* calls from TestNode (MarcoFalke)
fac7f6102f test: Use generate* node RPC, not wallet RPC (MarcoFalke)
faac1cda6e test: Use generate* from TestFramework, not TestNode (MarcoFalke)
Pull request description:
Deleting the methods is needed for #22567 to pave the way to make it easier to implicitly call the `sync_all` member function.
Without the methods being deleted, nothing prevents developers from adding calls to it. As history showed, developers *will* add calls to it. For example, see commit eb02dbba3c from today or the first commit in this pull request.
ACKs for top commit:
stratospher:
Tested ACK fac62e6.
brunoerg:
tACK fac62e6ff5
promag:
Code review ACK fac62e6ff5.
Tree-SHA512: 6d4dea8f95ead954acfef2e6a5d98897ce0c2d02265c5b137bb149d0265543bd51d7e8403e1945b9af75df5524ca50064fe1d2a432b25c8abc71bbb28ed6ed53
429b49378e test: introduce script_util helper for creating P2PK scripts (Sebastian Falbesoner)
Pull request description:
This PR is a follow-up to #22363, which took use of already existing `script_util` helpers to get rid of manual CScript for the P2{PKH,SH,WPKH,WSH} output types, in order to increase readability and maintainability of the test code. Here the same is done for P2PK scripts by introducing a helper `key_to_p2pk_script` and using it. Note that the helper only accepts ECDSA pubkeys (i.e. ones with a size of 33 or 65 bytes), hence it can't be used for scripts in the form of [x-only-pubkey, OP_CHECKSIG].
ACKs for top commit:
brunoerg:
Code review ACK 429b49378e
laanwj:
Code review ACK 429b49378e
rajarshimaitra:
Concept + tACK 429b49378e
Tree-SHA512: 984aea01eba5f38a328d69905d90a3a36f0a02419ca3e5baf3c8095895fb094e3780c7da16fad5851db3847bdb05ce8cda244ab09b79b8aa9602dfb3c5e0414c
fa2ac5881e test: Replace satoshi_round with int() or Decimal() (MarcoFalke)
Pull request description:
satoshi_round will round down. To make the code easier to parse use
Decimal() where possible, which does not round. Or use int(), which
explicitly rounds down.
ACKs for top commit:
lsilva01:
Tested ACK fa2ac5881e on Ubuntu 20.04.
Tree-SHA512: 17795d906aa7652933d43e510e993cdd9cf8926da1febf1c42d463048cb38c92dc518ec08736efe29c0189ffd532b108bc7a715f32b4c2ee58b215df65352eb9
satoshi_round will round down. To make the code easier to parse use
Decimal() where possible, which does not round. Or use int(), which
explicitly rounds down.
With this new method, outputs to an arbitrary scriptPubKey/amount can
be created. Note that the implementation was already present in the
test feature_rbf.py and is just moved to the MiniWallet interface, in
order to enable other tests to also use it.
f680d27155 test: use MiniWallet for make_utxo helper in feature_rbf.py (Sebastian Falbesoner)
0f27524602 test: scale amounts in test_doublespend_tree down by factor 10 (Sebastian Falbesoner)
d1e2481274 test: scale amounts in test_doublespend_chain down by factor 10 (Sebastian Falbesoner)
Pull request description:
This PR aims to further increase MiniWallet usage in the functional test feature_rbf.py by using it in the `make_utxo(...)` helper, which is the only part that needs a wallet for most sub-tests. In order to do that, the amounts for the utxos have to be scaled down in two sub-tests first (`test_doublespend_chain` and `test_doublespend_tree`, see first two commits), since we need amounts passed to `make_utxo` than can be funded by only one input. For creating UTXOs with a value of 50 BTC, we'd need to implement a method for consolidating multiple utxos into one first, which seems to be overkill.
Note that after this PR's change, there is only one sub-test left (`test_rpc`) that needs the wallet compiled into bitcoind.
ACKs for top commit:
MarcoFalke:
review ACK f680d27155🦐
Tree-SHA512: 46c8c245086a9e79855c4ede2f8f412333cf2658136805196b203b3567c89398d77fcb80715c0bb72fdc84331cc67544b2fdc259193a3adcb2fc36e147c26fce
d6d2ab9845 test: MiniWallet: fix fee calculation for P2PK and check tx vsize (Sebastian Falbesoner)
ce024b1c0e test: MiniWallet: force P2PK signature to have fixed size (71 bytes) (Sebastian Falbesoner)
Pull request description:
This PR is a follow-up to #21945. It aims to both fix the fee calculation for P2PK mode transactions and enable its vsize check. Currently, the latter assumes a fixed tx length, which is fine for anyone-can-spend txs but doesn't apply to P2PK output spends due to varying DER signature size; the vsize check is therefore disabled for P2PK mode on master branch.
Creating one million DER signatures with MiniWallet shows the following distribution of sizes (smart people with better math skills probably could deduce the ratios without trying, but hey):
| DER signature size [bytes] | #occurences (ratio) |
| ------------- | ------------- |
| 71 | 498893 (49.89%) |
| 70 | 497244 (49.72%) |
| 69 | 3837 (0.38%) |
| 68 | 22 (0.0022%) |
Note that even smaller signatures are possible (for smaller R and S values with leading zero bytes), it's just that the probability decreases exponentially. Instead of choosing a large vsize check range and hoping that smaller signatures are never created (potentially leading to flaky tests), the proposed solution is ~~to limit the signature size to the two most common sizes 71 and 70 (>99.6% probability) and then accordingly only check for two vsize values; the value to be used for fee calculation is a decimal right between the two possible sizes (167.5 vbytes) and for the vsize check it's rounded down/up integer values are used.~~ to simply grind the signature to a fixed size of 71 bytes (49.89% probability, i.e. on average each call to `sign_tx()`, on average two ECC signing operations are needed).
~~The idea of grinding signatures to a fixed size (similar to https://github.com/bitcoin/bitcoin/pull/13666 which grinds to low-R values) would be counter-productive, as the signature creation in the test suite is quite expensive and this would significantly slow down tests that calculate hundreds of signatures (like e.g. feature_csv_activation.py).~~
For more about transaction sizes on different input/output types, see the following interesting article: https://medium.com/coinmonks/on-bitcoin-transaction-sizes-97e31bc9d816
ACKs for top commit:
MarcoFalke:
Concept ACK d6d2ab9845
Tree-SHA512: 011c70ee0e4adf9ba12902e4b6c411db9ae96bdd8bc810bf1d67713367998e28ea328394503371fc1f5087a819547ddaea56c073b28db893ae1c0031d7927f32
In order to enable exact fee calculation for transactions that spend
P2PK outputs in the MiniWallet, we enforce the created signatures to
have a fixed length (>49.89% probability) by default. With that it is
easier to check the created transactions vsize and avoid flaky tests
that would appear whenever the signatures R- or S-values are smaller
(due to leading zero bytes).
Note that to get the total scriptSig size one has to add another
2 bytes, as there is also the OP_PUSHx instruction on the front and
the sighash type byte on the back, leading to a final scriptSig size
of 73 bytes.
- run mempool_reorg.py even when the wallet is not compiled
- add `locktime` argument to `create_self_transfer` and `send_self_transfer`
- use more logs instead of comments
For the MiniWallet constructor, the two boolean parameters
"raw_script" and "use_p2pk" are replaced by a single parameter of the
newly introduced type MiniWalletMode (derived by enum.Enum), which can
hold the following values:
- ADDRESS_OP_TRUE
- RAW_OP_TRUE
- RAW_P2PK
