fa3e0da06b policy: Treat taproot as always active (MarcoFalke)
Pull request description:
Now that taproot is active, it can be treated as if it was always active for policy for the next major release. This simplifies the code and changes two things:
* Importing `tr` descriptors can be done before the chain is fully synced. This is fine, because the wallet will already generate `tr` descriptors by default (regardless of the taproot status) after commit 47fe7445e7.
* Valid taproot spends won't be rejected from the mempool before taproot is active. This is strictly speaking a bugfix after commit 47fe7445e7, since the wallet may generate taproot spends before the chain is fully synced. For example, a slow node or a purposefully offline node. Currently, the wallet needs the mempool to account for change. See https://github.com/bitcoin/bitcoin/issues/11887.
A similar change was done for segwit v0 in https://github.com/bitcoin/bitcoin/pull/13120 .
This effectively reverts commit c5ec0367d7.
ACKs for top commit:
mjdietzx:
Code Review ACK fa3e0da06b
achow101:
ACK fa3e0da06b
sipa:
utACK fa3e0da06b
gruve-p:
ACK fa3e0da06b
gunar:
Code Review + tACK fa3e0da06
rajarshimaitra:
code review + tACK fa3e0da06b
Tree-SHA512: c6dc7a4e6c345bdec33f256847dc63906ab1696aa683ab9b32a79e715613950884ac3a1a7a44e95f31bb28e58dd64679a616175f7e152b21f5550f3337c8e622
88cc481092 Modify copyright header on Bech32 code (Samuel Dobson)
5599813b80 Add lots of comments to Bech32 (Samuel Dobson)
2eb5792ec7 Add release notes for validateaddress Bech32 error detection (MeshCollider)
42d6a029e5 Refactor and add more tests for validateaddress (Samuel Dobson)
c4979f77c1 Add boost tests for bech32 error detection (MeshCollider)
02a7bdee42 Add error_locations to validateaddress RPC (Samuel Dobson)
b62b67e06c Add Bech32 error location function (Samuel Dobson)
0b06e720c0 More detailed error checking for base58 addresses (Samuel Dobson)
Pull request description:
Addresses (partially) #16779 - no GUI change in this PR
Adds a LocateError function the bech32 library, which is then called by `validateaddress` RPC, (and then eventually from a GUI tool too, future work). I think modifying validateaddress is nicer than adding a separate RPC for this.
Includes tests.
Based on https://github.com/sipa/bech32/blob/master/ecc/javascript/bech32_ecc.js
Credit to sipa for that code
ACKs for top commit:
laanwj:
Code review and manually tested ACK 88cc481092
ryanofsky:
Code review ACK 88cc481092 with caveat that I only checked the new `LocateErrors` code to try to verify it didn't have unsafe or unexpected operations or loop forever or crash. Did not try to verify behavior corresponds to the spec. In the worst case bugs here should just affect error messages not actual decoding of addresses so this seemed ok.
w0xlt:
tACK 88cc481
Tree-SHA512: 9c7fe9745bc7527f80a30bd4c1e3034e16b96a02cc7f6c268f91bfad08a6965a8064fe44230aa3f87e4fa3c938f662ff4446bc682c83cb48c1a3f95cf4186688
e57c0eb865 test: refactor: replace OP_1/OP_TRUE magic numbers by constants (Sebastian Falbesoner)
df5d783aef test: refactor: take use of `create_block` txlist parameter (Sebastian Falbesoner)
ae9df4ef93 test: refactor: take use of `create_block` version parameter (or use default) (Sebastian Falbesoner)
Pull request description:
The helper `create_block` offers two parameters `version` and `txlist` which set the `nVersion` field / extend the `vtx` array of the block, respectively. By taking use of those, we can remove a lot of code, including the recalculation of the merkle root. Both passing txs in string and `CTransaction` format is supported, i.e. we also save potential calls to `tx_from_hex`.
The PR also contains another commit which replaces magic numbers for OP_TRUE/OP_1 (0x51) with the proper constant from the `script` module.
Instances setting the block version of 4 explicitely after calling `create_block` are removed, as this is the default since #16333 got merged (see https://github.com/bitcoin/bitcoin/pull/23521#discussion_r751173671).
ACKs for top commit:
stratospher:
tested ACK e57c0eb.
Tree-SHA512: a56965168d36b40b84e7f334b00472b82c31e8482c9e2651c97a791abd7fee3b40ca15e943a7acafa3acf172066fdace38bb13240084b789fd6ff4f6e510e23a
4868c9f1b3 Extract Taproot internal keyid with GetKeyFromDestination (Andrew Chow)
d8abbe119c Mention bech32m in -addresstype and -changetype help (Andrew Chow)
8fb57845ee Create a tr() descriptor bech32m DescriptorScriptPubKeyMan by default (Andrew Chow)
54b3699862 Store pubkeys in TRDescriptor::MakeScripts (Andrew Chow)
Pull request description:
Make a `tr()` descriptor by default in descriptor wallets so that users will be able to make and use segwit v1 bech32m addresses.
ACKs for top commit:
MarcoFalke:
Concept ACK 4868c9f1b3
Sjors:
re-utACK 4868c9f1b3
gruve-p:
ACK 4868c9f1b3
meshcollider:
Concept + code review ACK 4868c9f1b3
Tree-SHA512: e5896e665b8d559f1d759b6582d1bb24f70d4698a57307684339d9fdcdac28ae9bc17bc946a7efec9cb35c130a95ffc36e3961a335124ec4535d77b8d00e9631
1e86ff7941 test: run rpc-generateblock.py even with wallet disabled (Darius Parvin)
Pull request description:
Run rpc_generateblock.py test even when the wallet is disabled, as discussed in #20078.
This PR picked up from where #20808 left off, started by @ nginocchio. Since then, there have been many improvements to `MiniWallet`, making this PR more straightforward.
L23 makes use of `MiniWallet.rescan_utxos()` to add the pre-mined block utxos (when `self.setup_clean_chain` is not set to `True`), rather than generating new blocks during the test.
ACKs for top commit:
mjdietzx:
Tested ACK 1e86ff7941
Tree-SHA512: 4285f61516dd53a08004eeea26d58f45b4c1c67f5ca4c94ff1bc9fc7e50f486de2e033a8b4aaf67cb4c33d73aa929362e18dc75d5c7951cbf58120b5fb1de555
Also remove the enumerations ("1)", "2)"...) from the test
cases as those potentially hinder maintainability; e.g. if a
new case in inserted in-between, all the remaining
enumerations would need to be adapted.
Passing a list of transactions `txlist` to `create_block` appends
them to the block, hence we don't need to do that manually anymore.
The merkle root calculation can also be removed, since this is done
in the end of the helper.
f1ed30451f test: refactor: simplify `block_submit` in feature_nulldummy.py (Sebastian Falbesoner)
5ba9f1ff59 test: refactor: rename NULLDUMMY-invalidation helper (Sebastian Falbesoner)
e1d4a128e8 test: simplify and document NULLDUMMY-invalidation helper (Sebastian Falbesoner)
Pull request description:
This PR improves the functional test `feature_nulldummy.py` by simplifying the helpers `trueDummy` (renamed to `invalidate_nulldummy_tx`) and `block_submit`. Details can be found in the commit messages.
ACKs for top commit:
laanwj:
Code review ACK f1ed30451f
Tree-SHA512: ad227b31936f53c5dbded823643bced296d86f40b90f2c144a9857db3d00544f9ad5bbce4c7e84b6ece25e78e95c19aafb1d8fb31e610dcd5cbf3da63190de85
fa9c26ab3a test: Force --nosandbox when --valgrind (MarcoFalke)
Pull request description:
The two options are mutually exclusive and will result in a test failure. Fix that.
Can be tested with:
```
$ ./test/functional/wallet_disable.py --valgrind
ACKs for top commit:
fanquake:
ACK fa9c26ab3a
Tree-SHA512: 7d1c36c1b6627ca041757eb0515a0d6cc962a56d783ee4f5647a2ddc2d104491f0586a8ea0b8acebe0a203190f4f5567b349123dfd4c181bcc63361174a8ab63
f1c33ee4ac tests: implement BIP341 test vectors (Pieter Wuille)
ac3037df11 tests: BIP341 test vector generation (Pieter Wuille)
ca83ffc2ea tests: add deterministic signing mode to ECDSA (Pieter Wuille)
c98c53f20c tests: abstract out precomputed BIP341 signature hash elements (Pieter Wuille)
a5bde018b4 tests: give feature_taproot access to sighash preimages (Pieter Wuille)
5140825096 tests: add more fields to TaprootInfo (Pieter Wuille)
2478c6730a Make signing follow BIP340 exactly w.r.t. aux randomness (Pieter Wuille)
Pull request description:
This PR adds code to `test/functional/feature_taproot.py` which runs through a (deterministic) scenario covering several aspects of the wallet side of BIP341 (scriptPubKey computation from keys/scripts, control block computation, key path spending), with the ability to output test vectors in mediawiki format based on this scenario. The generated tests are then also included directly in `src/test/script_tests.cpp` and `src/test/script_standard_tests.cpp`.
I intend to add these test vectors to BIP341 itself: https://github.com/bitcoin/bips/pull/1225
ACKs for top commit:
laanwj:
Code review ACK f1c33ee4ac
Tree-SHA512: fcf7109539cb214d3190516b205cd32d2b1b452f14aa66f4107acfaa8bfc7d368f626857f1935665a4342eabc0b9ee8aba608a7c0a2494bec0b498e723439c9d
fadc4c7272 test: Add txindex migration test (MarcoFalke)
Pull request description:
Test for #22626
ACKs for top commit:
theStack:
Tested ACK fadc4c7272🌁
Tree-SHA512: fc7133ef52826bf0d4fa2ac72c3f1bed4a185ff7492396552ff2cbf6531b053238039211a710cbb949379c56875cd7715f1ed49a514dd3b3f1b46554e3d4bef5
a9872e1478 test: remove unnecessary block rehashing prior to solving (Sebastian Falbesoner)
Pull request description:
Solving a block involves continously rehashing it,
c9dd5c8d6e/test/functional/test_framework/messages.py (L759-L764)
i.e. any extra `rehash` calls before are not necessary and can be dropped.
The instances were identified by searching for all block solving calls via `git grep "solve("` in `./test/functional/`. From 95 instances of `CBlock.solve()` calls, 20 contained an unnecessary rehashing instruction before that are removed in this PR.
ACKs for top commit:
brunoerg:
tACK a9872e1478
rajarshimaitra:
tACK a9872e1478
Tree-SHA512: 160092be717d0d250778b8ab091ebd77cc6865d2754ef150cf3b4d4ac7304d4bf3d2ebb61ec6b04a55040c8895b9fb4d28653ea4b099d56d90776c9111cf173f
The `create_block` helper accepts a list of txs that it includes in the
created block, hence we don't have to do that manually. Also, rehashing
before solving the block is not needed and can be removed.
The function `trueDummy` in feature_nulldummy.py is currently more
complicated than it needs to be. Rather than converting the scriptSig to
a CScript and looping through it to build a new scriptSig with the
modified push, simply directly replace the push of null (OP_0) with a
push of one (OP_TRUE/OP_1).
Note that on master, actually an element with the value of 0x51 is
pushed (0x0151...) -- this was very likely not intended, as 0x51 is the
script op-code for OP_TRUE, and also the function's name suggests that
the "true" value shall be pushed.
This does the following:
* Adds a rfc6979 argument to test_framework/key.py's sign_ecdsa to
select (deterministic) RFC6979-based nonce generation.
* Add a flag in feature_taproot.py's framework called "deterministic".
* Make the Schnorr signing in feature_taproot.py randomized by default,
reverting to the old deterministic (aux_rnd=0x0000...00) behavior
if the deterministic context flag is set.
* Make the ECDSA signing in feature_taproot.py use RFC6979-based nonces
when the deterministic context flag is set (keeping the old randomized
behavior otherwise).
22b44fc696 p2p: improve checkaddrman logging with duration in milliseconds (Jon Atack)
ec65bed00e log, timer: add LOG_TIME_MILLIS_WITH_CATEGORY_MSG_ONCE macro (Jon Atack)
325da75a53 log, timer: allow not repeating log message on completion (Jon Atack)
Pull request description:
This patch:
- updates the `logging/timer.h::Timer` class to allow not repeating the log message on completion
- adds a `LOG_TIME_MILLIS_WITH_CATEGORY_MSG_ONCE` macro that prints the descriptive message when logging the start but not when logging the completion
- updates the checkaddrman logging to log the duration, and renames the function like the `-checkaddrman` configuration option in order to prefix every log message with `CheckAddrman` instead of the longer, less pleasant, and different-from-checkaddrman `ForceCheckAddrman` (the Doxygen documentation on the function already makes clear that it is unaffected by `m_consistency_check_ratio`).
before
```
2021-09-21T18:42:50Z [opencon] Addrman checks started: new 64864, tried 1690, total 66554
2021-09-21T18:42:50Z [opencon] Addrman checks completed successfully
```
after
```
2021-09-21T18:42:50Z [opencon] CheckAddrman: new 64864, tried 1690, total 66554 started
2021-09-21T18:42:50Z [opencon] CheckAddrman: completed (76.21ms)
```
To test, build and run bitcoind with `-debug=addrman -checkaddrman=<n>` for a value of `n` in the range of, say, 10 to 40.
ACKs for top commit:
laanwj:
Code review ACK 22b44fc696
Tree-SHA512: 658c0dfaaa9d07092e2418f2d05007c58cc35be6593f22b3c592ce793334a885dd92dacc46bdeddc9d37939cf11174660a094c07c0fa117fbb282953aa45a94d
aa1a4c9204 Add file validation to savemempool RPC test (lsilva01)
871e64d22f Add filename to savemempool RPC result (lsilva01)
Pull request description:
Currently, if the user calls the `savemempool` RPC method, there is no way to know
where the file was created (unless the user knows internal implementation details).
This PR adds a return message stating the file name and path where the mempool was saved and changes `mempool_persist.py` to validate this new return message.
ACKs for top commit:
laanwj:
Code review ACK aa1a4c9204
Tree-SHA512: e8b1dd0a8976e5eb15f7476c9651e492d2c621a67e0b726721fa7a2ae0ddd272ee28b87a2d0c650bd635e07fa96bdefe77bece4deb6486ef3ee9a4f83423a840
The previous diff touched most files in ./test/, so bump the headers to
avoid having to touch them again for a bump later.
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./test/
-END VERIFY SCRIPT-
facc352648 test: Implicitly sync after generate*, unless opted out (MarcoFalke)
Pull request description:
The most frequent failure in functional tests are intermittent races. Fixing such bugs is cumbersome because it involves:
* Noticing the failure
* Fetching and reading the log to determine the test case that failed
* Adding a `self.sync_all()` where it was forgotten
* Spamming out a pr and waiting for review, which is already sparse
Also, writing a linter to catch those is not possible, nor is review effective in finding these bugs prior to merge.
Fix all future intermittent races caused by a missing sync_block call by calling `sync_all` implicitly after each `generate*`, unless opted out. This ensures that the code is race-free (with regards to blocks) when the tests pass once, instead of our current approach where the code can never be guaranteed to be race-free.
There are some scripted-diff cleanups (see https://github.com/bitcoin/bitcoin/pull/22567), but they will be submitted in a follow-up to reduce the conflicts in this pull.
ACKs for top commit:
lsilva01:
tACK facc352 on Ubuntu 20.04
brunoerg:
tACK facc352648 on MacOS 11.6
Tree-SHA512: 046a40a066b4a3bd28a3077bd654fa8887442dd1f0ec6fd11671865809ef02376f126eb667a1320ebd67b6e372c78c00dbf8bd25d86ed86f1d9a25363103ed97
420695c193 contrib: recognize CJDNS seeds as such (Vasil Dimov)
f9c28330a0 net: take the first 4 random bits from CJDNS addresses in GetGroup() (Vasil Dimov)
29ff79c0a2 net: relay CJDNS addresses even if we are not connected to CJDNS (Vasil Dimov)
d96f8d304c net: don't skip CJDNS from GetNetworkNames() (Vasil Dimov)
c2d751abba net: take CJDNS into account in CNetAddr::GetReachabilityFrom() (Vasil Dimov)
9b43b3b257 test: extend feature_proxy.py to test CJDNS (Vasil Dimov)
508eb258fd test: remove default argument of feature_proxy.py:node_test() (Vasil Dimov)
6387f397b3 net: recognize CJDNS addresses as such (Vasil Dimov)
e6890fcb44 net: don't skip CJDNS from GetNetworksInfo() (Vasil Dimov)
e9d90d3c11 net: introduce a new config option to enable CJDNS (Vasil Dimov)
78f456c576 net: recognize CJDNS from ParseNetwork() (Vasil Dimov)
de01e312b3 net: use -proxy for connecting to the CJDNS network (Vasil Dimov)
aedd02ef27 net: make it possible to connect to CJDNS addresses (Vasil Dimov)
Pull request description:
CJDNS overview
=====
CJDNS is like a distributed, shared VPN with multiple entry points where every participant can reach any other participant. All participants use addresses from the `fc00::/8` network (reserved IPv6 range). Installation and configuration is done outside of applications, similarly to VPN (either in the host/OS or on the network router).
Motivation
=====
Even without this PR it is possible to connect two Bitcoin Core nodes through CJDNS manually by using e.g. `-addnode` in environments where CJDNS is set up. However, this PR is necessary for address relay to work properly and automatic connections to be made to CJDNS peers. I.e. to make CJDNS a first class citizen network like IPv4, IPv6, Tor and I2P.
Considerations
=====
An address from the `fc00::/8` network, could mean two things:
1. Part of a local network, as defined in RFC 4193. Like `10.0.0.0/8`. Bitcoin Core could be running on a machine with such address and have peers with those (e.g. in a local network), but those addresses are not relayed to other peers because they are not globally routable on the internet.
2. Part of the CJDNS network. This is like Tor or I2P - if we have connectivity to that network then we could reach such peers and we do relay them to other peers.
So, Bitcoin Core needs to be able to tell which one is it when it encounters a bare `fc00::/8` address, e.g. from `-externalip=` or by looking up the machine's own addresses. Thus a new config option is introduced `-cjdnsreacable`:
* `-cjdnsreacable=0`: it is assumed a `fc00::/8` address is a private IPv6 (1.)
* `-cjdnsreacable=1`: it is assumed a `fc00::/8` address is a CJDNS one (2.)
After setting up CJDNS outside of Bitcoin Core, a node operator only needs to enable this option.
Addresses from P2P relay/gossip don't need that because they are properly tagged as IPv6 or as CJDNS.
For testing
=====
```
[fc32:17ea:e415:c3bf:9808:149d:b5a2:c9aa]:8333
[fc68:7026:cb27:b014:5910:e609:dcdb:22a2]:8333
[fcb3:dc50:e1ae:7998:7dc0:7fa6:4582:8e46]:8333
[fcc7:be49:ccd1:dc91:3125:f0da:457d:8ce]:8333
[fcf2:d9e:3a25:4eef:8f84:251b:1b4d:c596]:8333
```
ACKs for top commit:
dunxen:
ACK 420695c
jonatack:
re-ACK 420695c193 per `git range-diff 23ae793 4fbff39 420695c`
laanwj:
Code review ACK 420695c193
Tree-SHA512: 21559886271aa84671d52b120fa3fa5a50fdcf0fcb26e5b32049c56fab0d606438d19dd366a9c8ce612d3894237ae6d552ead3338b326487e3534399b88a317a
80dc829be7 tests: Calculate fees more similarly to CFeeRate::GetFee (Andrew Chow)
ce2cc44afd tests: Test for assertion when feerate is rounded down (Andrew Chow)
0fbaef9676 fees: Always round up fee calculated from a feerate (Andrew Chow)
Pull request description:
When calculating the fee for a feerate, it is possible that the final calculation will have fractional satoshis. Currently those are ignored via truncation which results in the absolute fee being rounded down. Rounding down is problematic because it results in a feerate that is slightly lower than the feerate represented by the `CFeeRate` object. A slightly lower feerate particularly causes issues for coin selection as it can trigger an assertion error. To avoid potentially underpaying the feerate (and the assertion), always round up the calculated fee.
A test is added for the assertion, along with a comment explaining what happens.
It is unlikely that a user can trigger this as it requires a very specific set of rounding errors to occur as well as the transaction not needing any change and being right on the lower bound of the exact match window. However I was able to trigger the assertion while running coin selection simulations, albeit after thousands of transactions and with some weird feerates.
ACKs for top commit:
ryanofsky:
Code review ACK 80dc829be7
promag:
Tested ACK 80dc829be7.
lsilva01:
tACK 80dc829
meshcollider:
utACK 80dc829be7
Tree-SHA512: fe26684c60f236cab48ea6a4600c141ce766dbe59504ec77595dcbd7fd0b34559acc617007f4f499c9155d8fda0a336954413410ba862b19c765c0cfac79d642
The default bool argument makes it harder to read because the last but
one argument is also bool. Pass all of them as named arguments to
increase readability.
Another bool argument will be added to indicate whether to test CJDNS.
Co-authored-by: Jon Atack <jon@atack.com>
