This makes it more generalistic than just having the miniscripts since
we are going to have Taproot descriptors with (multiple) miniscripts in
them too.
- Remove usage of the internal wait_until_helper function
- Use framework self.no_op instead of new no_sync function
co-authored-by: Andrew Chow <github@achow101.com>
0f83ab407e test: display abrupt shutdown errors in console output (furszy)
Pull request description:
Making it easier to debug errors in the CI environment,
particularly in scenarios where it's not immediately clear
what happened nor which node crashed (or shutdown abruptly).
A bit of context:
Currently, the test framework redirects each node's stderr output
stream to a different temporary file inside each node's data directory.
While this is sufficient for storing the error, it isn't very helpful for
understanding what happened just by reading the CI console output.
Most of the time, reading the stderr file in the CI environment is not
possible, because people don't have access to it.
Testing Note:
The displayed error difference can be observed by cherry-picking this
commit 9cc5393c0f on top of this branch and running any
functional test.
ACKs for top commit:
maflcko:
lgtm ACK 0f83ab407e
theStack:
ACK 0f83ab407e
Tree-SHA512: 83ce4d21d5316e8cb16a17d3fbe77b8649fced9e09410861d9674c233f6e9c34bcf573504e387e4f439c2841b2ee9855d0d35607fa13aa89eafe0080c45ee82d
5b878be742 [doc] add release note for submitpackage (glozow)
7a9bb2a2a5 [rpc] allow submitpackage to be called outside of regtest (glozow)
5b9087a9a7 [rpc] require package to be a tree in submitpackage (glozow)
e32ba1599c [txpackages] IsChildWithParentsTree() (glozow)
b4f28cc345 [doc] parent pay for child in aggregate CheckFeeRate (glozow)
Pull request description:
Permit (restricted topology) submitpackage RPC outside of regtest. Suggested in https://github.com/bitcoin/bitcoin/pull/26933#issuecomment-1510851570
This RPC should be safe but still experimental - interface may change, not all features (e.g. package RBF) are implemented, etc. If a miner wants to expose this to people, they can effectively use "package relay" before the p2p changes are implemented. However, please note **this is not package relay**; transactions submitted this way will not relay to other nodes if the feerates are below their mempool min fee. Users should put this behind some kind of rate limit or permissions.
ACKs for top commit:
instagibbs:
ACK 5b878be742
achow101:
ACK 5b878be742
dergoegge:
Code review ACK 5b878be742
ajtowns:
ACK 5b878be742
ariard:
Code Review ACK 5b878be742. Though didn’t manually test the PR.
Tree-SHA512: 610365c0b2ffcccd55dedd1151879c82de1027e3319712bcb11d54f2467afaae4d05dca5f4b25f03354c80845fef538d3938b958174dda8b14c10670537a6524
fa071aeb61 wallet: No BDB creation, unless -deprecatedrpc=create_bdb (MarcoFalke)
Pull request description:
With BDB being removed soon, it seems confusing and harmful to allow users to create fresh BDB wallets going forward, as it would load them with an additional burden of having to migrate them soon after.
Also, it would be good to allow for one release for test (and external) scripts to adapt.
Fix all issues by introducing the `-deprecatedrpc=create_bdb` setting.
ACKs for top commit:
Sjors:
tACK fa071aeb61
achow101:
ACK fa071aeb61
furszy:
utACK fa071aeb
Tree-SHA512: 37a4c3e4ba659e0ebe2382e71d9c80e42a895d9ad743f5dda7c110fbbb7d2a36f46769982552a9ac0c3a57203379ef164be97aa8033eb7674d6b4da030ba8f9b
a9ef702a87 assumeutxo: change getchainstates RPC to return a list of chainstates (Ryan Ofsky)
Pull request description:
Current `getchainstates` RPC returns "normal" and "snapshot" fields which are not ideal because it requires new "normal" and "snapshot" terms to be defined, and the definitions are not really consistent with internal code. (In the RPC interface, the "snapshot" chainstate becomes the "normal" chainstate after it is validated, while in internal code there is no "normal chainstate" and the "snapshot chainstate" is still called that temporarily after it is validated).
The current `getchainstates` RPC is also awkward to use if you to want information about the most-work chainstate, because you have to look at the "snapshot" field if it exists, and otherwise fall back to the "normal" field.
Fix these issues by having `getchainstates` just return a flat list of chainstates ordered by work, and adding a new chainstate "validated" field alongside the existing "snapshot_blockhash" field so it is explicit if a chainstate was originally loaded from a snapshot, and whether the snapshot has been validated.
This change was motivated by comment thread in https://github.com/bitcoin/bitcoin/pull/28562#discussion_r1344154808
ACKs for top commit:
Sjors:
re-ACK a9ef702a87
jamesob:
re-ACK a9ef702
achow101:
ACK a9ef702a87
Tree-SHA512: b364e2e96675fb7beaaee60c4dff4b69e6bc2d8a30dea1ba094265633d1cddf9dbf1c5ce20c07d6e23222cf1e92a195acf6227e4901f3962e81a1e53a43490aa
afd9a673c4 test: roundtrip wallet backwards compat downgrade (Andrew Chow)
bbf43c63b9 test: Add 25.0 to wallet backwards compatibiilty test (Andrew Chow)
538939ec39 test: Run upgrade test on all nodes (Andrew Chow)
6d4699028b test: Run downgrade test on descriptor wallets (Andrew Chow)
f158573be1 test: Add 0.21 tr() incompatibility test (Andrew Chow)
f41215c3f0 test: add logging 0.17 incompatibilities in wallet back compat (Andrew Chow)
71c03aeff7 test: Refactor v19 addmultisigaddress test to be distinct (Andrew Chow)
53f35d02cb test: Remove w1_v18 from wallet backwards compatibility (Andrew Chow)
313d665437 test: Fix 0.16 wallet paths and downgrade test (Andrew Chow)
5d8469362a test: Add helper functions for checking node versions (Andrew Chow)
Pull request description:
It was somewhat surprising to me that wallet_backwards_compatibility.py did not catch #27915 since the purpose of the test is to find downgrade issues such as that. It turns out the test was deficient in several places when it came to testing descriptor wallets, as well as deficient in addition to failing to correctly test some releases.
This PR fixes these test cases, adds more informative logging, slightly refactors the entire test in order to better test future versions, and adds a 25.0 node to the test.
Notable changes:
* The compatibility test with 0.16 should not have been passing. The wallets were being copied incorrectly for 0.16 and resulting in 0.16 creating new wallets rather than testing the target wallets.
* The downgrade test will actually be run on descriptor wallets and it will test that downgrades are successful, and a subsequent upgrade is also successful. This catches #27915.
* The upgrade and downgrade test will be run on all versions up to master, rather than just 0.16, 0.17, and 0.19.
ACKs for top commit:
Sjors:
re-ACK afd9a673c4
furszy:
ACK afd9a67
Tree-SHA512: dd2d85cab29a636da93020681c533534af4a9cda18d8550c9db9d8937719b3a225025966981c5d4d2f30486448a772b760f0e723a25ea6bc49df80387dc7b8b0
fa28f5a381 test: Bump walletpassphrase timeouts to avoid intermittent issues (MarcoFalke)
Pull request description:
This bumps all timeouts for all `walletpassphrase` to avoid intermittent issues in `valgrind` (or other sanitizers).
As an idea for a follow-up, `walletpassphrase` could be changed to treat `0` as "no timeout" instead of "instant timeout".
Example failure:
```
node0 2023-09-03T22:44:38.374955Z [httpworker.3] [rpc/server.cpp:594] [RPCRunLater] [rpc] queue run of timer lockwallet(w6) in 60 seconds (using HTTP)
test 2023-09-03T22:44:40.173000Z TestFramework.bitcoincli (DEBUG): Running bitcoin-cli ['-rpcwallet=w6', 'getnewaddress', '', 'legacy']
node0 2023-09-03T22:44:59.810893Z [http] [httpserver.cpp:255] [http_request_cb] [http] Received a POST request for /wallet/w6 from 127.0.0.1:48928
node0 2023-09-03T22:44:59.813132Z [httpworker.1] [rpc/request.cpp:181] [parse] [rpc] ThreadRPCServer method=getnewaddress user=__cookie__
node0 2023-09-03T22:44:59.837183Z [httpworker.1] [wallet/sqlite.cpp:53] [TraceSqlCallback] [/ci_container_base/ci/scratch/test_runner/test_runner_₿_🏃_20230903_183350/wallet_createwallet_171/node0/regtest/w6/wallet.dat] SQLite Statement: INSERT or REPLACE into main values(?, ?)
node0 2023-09-03T22:44:59.929735Z [httpworker.1] [wallet/sqlite.cpp:53] [TraceSqlCallback] [/ci_container_base/ci/scratch/test_runner/test_runner_₿_🏃_20230903_183350/wallet_createwallet_171/node0/regtest/w6/wallet.dat] SQLite Statement: INSERT or REPLACE into main values(?, ?)
node0 2023-09-03T22:44:59.934484Z [httpworker.1] [wallet/sqlite.cpp:53] [TraceSqlCallback] [/ci_container_base/ci/scratch/test_runner/test_runner_₿_🏃_20230903_183350/wallet_createwallet_171/node0/regtest/w6/wallet.dat] SQLite Statement: INSERT or REPLACE into main values(?, ?)
node0 2023-09-03T22:44:59.935467Z [httpworker.1] [wallet/sqlite.cpp:53] [TraceSqlCallback] [/ci_container_base/ci/scratch/test_runner/test_runner_₿_🏃_20230903_183350/wallet_createwallet_171/node0/regtest/w6/wallet.dat] SQLite Statement: INSERT or REPLACE into main values(?, ?)
test 2023-09-03T22:45:02.328000Z TestFramework.bitcoincli (DEBUG): Running bitcoin-cli ['-rpcwallet=w6', 'signmessage', 'mqatqH4VQmrZ81nxUfrnfcLnxgbzhZb4PC', 'test']
node0 2023-09-03T22:45:20.269375Z [http] [httpserver.cpp:255] [http_request_cb] [http] Received a POST request for /wallet/w6 from 127.0.0.1:44618
node0 2023-09-03T22:45:20.270670Z [httpworker.2] [rpc/request.cpp:181] [parse] [rpc] ThreadRPCServer method=signmessage user=__cookie__
test 2023-09-03T22:45:23.490000Z TestFramework.bitcoincli (DEBUG): Running bitcoin-cli ['-rpcwallet=w6', 'keypoolrefill', '1']
node0 2023-09-03T22:45:40.244603Z [http] [httpserver.cpp:255] [http_request_cb] [http] Received a POST request for /wallet/w6 from 127.0.0.1:32854
node0 2023-09-03T22:45:40.293021Z [httpworker.0] [rpc/request.cpp:181] [parse] [rpc] ThreadRPCServer method=keypoolrefill user=__cookie__
test 2023-09-03T22:45:41.852000Z TestFramework (ERROR): JSONRPC error
Traceback (most recent call last):
File "/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 131, in main
self.run_test()
File "/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/wallet_createwallet.py", line 156, in run_test
w6.keypoolrefill(1)
File "/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_node.py", line 732, in __call__
return self.cli.send_cli(self.command, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_node.py", line 795, in send_cli
raise JSONRPCException(dict(code=int(code), message=message))
test_framework.authproxy.JSONRPCException: Error: Please enter the wallet passphrase with walletpassphrase first. (-13)
ACKs for top commit:
achow101:
ACK fa28f5a381
Tree-SHA512: 58caa569cec39acc121d4cc038a4190937af34e85d2696272ed4f2792fd386469b0cfefd2cb564438fedded97b21b23d8bf46ba27b5633671a277ed4679f0d5d
Current getchainstates RPC returns "normal" and "snapshot" fields which are not
ideal because it requires new "normal" and "snapshot" terms to be defined, and
the definitions are not really consistent with internal code. (In the RPC
interface, the "snapshot" chainstate becomes the "normal" chainstate after it
is validated, while in internal code there is no "normal chainstate" and the
"snapshot chainstate" is still called that temporarily after it is validated).
The current getchainstatees RPC is also awkward to use if you to want
information about the most-work chainstate because you have to look at the
"snapshot" field if it exists, and otherwise fall back to the "normal" field.
Fix these issues by having getchainstates just return a flat list of
chainstates ordered by work, and adding new chainstate "validated" field
alongside the existing "snapshot_blockhash" so it is explicit if a chainstate
was originally loaded from a snapshot, and whether the snapshot has been
validated.
Making it easier to debug errors in the CI environment,
particularly in scenarios where it's not immediately clear
what happened nor which node crashed (or shutdown abruptly).
d27b9a2248 test: fix feature_init.py file perturbation (Martin Zumsande)
ad66ca1e47 init: abort loading of blockindex in case of missing height. (Martin Zumsande)
Pull request description:
When the block index database is non-contiguous due to file corruption (i.e. it contains indexes of height `x-1` and `x+1`, but not `x`), bitcoind can currently crash with an assert in `BuildSkip()` / `GetAncestor()` during `BlockManager::LoadBlockIndex()`:
```
bitcoind: chain.cpp:112: const CBlockIndex* CBlockIndex::GetAncestor(int) const: Assertion `pindexWalk->pprev' failed.
```
This PR changes it such that we instead return an `InitError` to the user.
I stumbled upon this because I noticed that the file perturbation in `feature_init.py` wasn't working as intended, which is fixed in the second commit:
* Opening the file twice in one `with` statement would lead to `tf_read` being empty, so the test wouldn't perturb anything but replace the file with a new one. Fixed by first opening for read, then for write.
* We need to restore the previous state after perturbations, so that only the current perturbation is active and not a mix of the current and previous ones.
* I also added `checkblocks=200` to the startup parameters so that corruption in earlier blocks of `blk00000.dat` is detected during init verification and not ignored.
After fixing `feature_init.py` like that I'd run into the `assert` mentioned above (so running the testfix from the second commit without the first one is a way to reproduce it).
ACKs for top commit:
achow101:
ACK d27b9a2248
furszy:
Code ACK d27b9a224
fjahr:
Code review ACK d27b9a2248
Tree-SHA512: 2e54da6030c5813c86bd58f816401e090bb43c5b834764a5e3c0e55dbfe09e423f88042cab823db3742088204b274d4ad2abf58a3832a4b18328b11a30bf7094
352d5eb2a9 test: getrawaddrman RPC (0xb10c)
da384a286b rpc: getrawaddrman for addrman entries (0xb10c)
Pull request description:
Inspired by `getaddrmaninfo` (#27511), this adds a hidden/test-only `getrawaddrman` RPC. The RPC returns information on all addresses in the address manager new and tried tables. Addrman table contents can be used in tests and during development.
The RPC result encodes the `bucket` and `position`, the internal location of addresses in the tables, in the address object's string key. This allows users to choose to consume or to ignore the location information. If the internals of the address manager implementation change, the location encoding might change too.
```
getrawaddrman
EXPERIMENTAL warning: this call may be changed in future releases.
Returns information on all address manager entries for the new and tried tables.
Result:
{ (json object)
"table" : { (json object) buckets with addresses in the address manager table ( new, tried )
"bucket/position" : { (json object) the location in the address manager table (<bucket>/<position>)
"address" : "str", (string) The address of the node
"port" : n, (numeric) The port number of the node
"network" : "str", (string) The network (ipv4, ipv6, onion, i2p, cjdns) of the address
"services" : n, (numeric) The services offered by the node
"time" : xxx, (numeric) The UNIX epoch time when the node was last seen
"source" : "str", (string) The address that relayed the address to us
"source_network" : "str" (string) The network (ipv4, ipv6, onion, i2p, cjdns) of the source address
},
...
},
...
}
Examples:
> bitcoin-cli getrawaddrman
> curl --user myusername --data-binary '{"jsonrpc": "1.0", "id": "curltest", "method": "getrawaddrman", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:8332/
```
ACKs for top commit:
willcl-ark:
reACK 352d5eb2a9
amitiuttarwar:
reACK 352d5eb2a9
stratospher:
reACK 352d5eb.
achow101:
ACK 352d5eb2a9
Tree-SHA512: cc462666b5c709617c66b0e3e9a17c4c81e9e295f91bdd9572492d1cb6466fc9b6d48ee805ebe82f9f16010798370effe5c8f4db15065b8c7c0d8637675d615e
fa6e6a3f03 doc: Remove confusing assert linter (MarcoFalke)
Pull request description:
The `assert()` documentation and linter are redundant and confusing:
* The source code already refuses to compile with `assert()` disabled.
* They violate the assumptions about `Assert()`, which *requires* side effects.
* The existing linter doesn't enforce the guideline, only checking for `++` and `--` side effects.
Fix all issues by removing the docs and the linter. See also https://github.com/bitcoin/bitcoin/pull/26684#discussion_r1287370102
Going forward everyone is free to use whatever code in this regard they think is the easiest to read. Also, everyone is still free to share style-nits, if they think it is a good use of their time and of the pull request author. Finally, the author is still free to dismiss or ignore this style-nit, or any other style-nit.
ACKs for top commit:
hebasto:
ACK fa6e6a3f03, I have reviewed the code and it looks OK.
theStack:
ACK fa6e6a3f03
Tree-SHA512: 686738d71e1316cc95e5d3f71869b55a02bfb137c795cc0875057f4410e564bc8eff03c985a2087b007fb08fc84551c7da1e8b30c7a9c3f2b14e5e44a5970236
edbed31066 chainparams: add signet assumeutxo param at height 160_000 (Sjors Provoost)
b8cafe3871 chainparams: add testnet assumeutxo param at height 2_500_000 (Sjors Provoost)
99839bbfa7 doc: add note about confusing HaveTxsDownloaded name (James O'Beirne)
7ee46a755f contrib: add script to demo/test assumeutxo (James O'Beirne)
42cae39356 test: add feature_assumeutxo functional test (James O'Beirne)
0f64bac603 rpc: add getchainstates (James O'Beirne)
bb05857794 refuse to activate a UTXO snapshot if mempool not empty (James O'Beirne)
ce585a9a15 rpc: add loadtxoutset (James O'Beirne)
62ac519e71 validation: do not activate snapshot if behind active chain (James O'Beirne)
9511fb3616 validation: assumeutxo: swap m_mempool on snapshot activation (James O'Beirne)
7fcd21544a blockstorage: segment normal/assumedvalid blockfiles (James O'Beirne)
4c3b8ca35c validation: populate nChainTx value for assumedvalid chainstates (James O'Beirne)
49ef778158 test: adjust chainstate tests to use recognized snapshot base (James O'Beirne)
1019c39982 validation: pruning for multiple chainstates (James O'Beirne)
373cf91531 validation: indexing changes for assumeutxo (James O'Beirne)
1fffdd76a1 net_processing: validationinterface: ignore some events for bg chain (James O'Beirne)
fbe0a7d7ca wallet: validationinterface: only handle active chain notifications (James O'Beirne)
f073917a9e validationinterface: only send zmq notifications for active (James O'Beirne)
4d8f4dcb45 validation: pass ChainstateRole for validationinterface calls (James O'Beirne)
1e59acdf17 validation: only call UpdatedBlockTip for active chainstate (James O'Beirne)
c6af23c517 validation: add ChainstateRole (James O'Beirne)
9f2318c76c validation: MaybeRebalanceCaches when chain leaves IBD (James O'Beirne)
434495a8c1 chainparams: add blockhash to AssumeutxoData (James O'Beirne)
c711ca186f assumeutxo: remove snapshot during -reindex{-chainstate} (James O'Beirne)
c93ef43e4f bugfix: correct is_snapshot_cs in VerifyDB (James O'Beirne)
b73d3bbd23 net_processing: Request assumeutxo background chain blocks (Suhas Daftuar)
Pull request description:
- Background and FAQ: https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/proposal
- Prior progress/project: https://github.com/bitcoin/bitcoin/projects/11
- Replaces https://github.com/bitcoin/bitcoin/pull/15606, which was closed due to Github slowness. Original description and commentary can be found there.
---
This changeset finishes the first phase of the assumeutxo project. It makes UTXO snapshots loadable via RPC (`loadtxoutset`) and adds `assumeutxo` parameters to chainparams. It contains all the remaining changes necessary to both use an assumedvalid snapshot chainstate and do a full validation sync in the background.
This may look like a lot to review, but note that
- ~200 lines are a (non-essential) demo shell script
- Many lines are functional test, documentation, and relatively dilute RPC code.
So it shouldn't be as burdensome to review as the linecount might suggest.
- **P2P**: minor changes are made to `init.cpp` and `net_processing.cpp` to make simultaneous IBD across multiple chainstates work.
- **Pruning**: implement correct pruning behavior when using a background chainstate
- **Blockfile separation**: to prevent "fragmentation" in blockfile storage, have background chainstates use separate blockfiles from active snapshot chainstates to avoid interleaving heights and impairing pruning.
- **Indexing**: some `CValidationInterface` events are given with an additional parameter, ChainstateRole, and all indexers ignore events from ChainstateRole::ASSUMEDVALID so that indexation only happens sequentially.
- Have `-reindex` properly wipe snapshot chainstates.
- **RPC**: introduce RPC commands `loadtxoutset` and (hidden) `getchainstates`.
- **Release docs & first assumeutxo commitment**: add notes and a particular assumeutxo hash value for first AU-enabled release.
- This will complete the project and allow use of UTXO snapshots for faster node bootstrap.
The next phase, if it were to be pursued, would be coming up with a way to distribute the UTXO snapshots over the P2P network.
---
### UTXO snapshots
Create your own with `./contrib/devtools/utxo_snapshot.sh`, e.g.
```shell
./contrib/devtools/utxo_snapshot.sh 788000 utxo.dat ./src/bitcoin-cli -datadir=$(pwd)/testdata`)
```
or use the pre-generated ones listed below.
- Testnet: **2'500'000** (Sjors):
- torrent: `magnet:?xt=urn:btih:511e09f4bf853aefab00de5c070b1e031f0ecbe9&dn=utxo-testnet-2500000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969`
- sha256: `79db4b025448cc0ac388d8589a28eab02de53055d181e34eb47391717aa16388`
- Signet: **160'000** (Sjors):
- torrent: `magnet:?xt=urn:btih:9da986cb27b3980ea7fd06b21e199b148d486880&dn=utxo-signet-160000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969`
- sha256: `eeeca845385ba91e84ef58c09d38f98f246a24feadaad57fe1e5874f3f92ef8c`
- Mainnet: **800'000** (Sjors):
- Note: this needs the following commit cherry-picked in: 24deb2022b
- torrent: `magnet:?xt=urn:btih:50ee955bef37f5ec3e5b0df4cf0288af3d715a2e&dn=utxo-800000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969`
### Testing
#### For fun (~5min)
If you want to do a quick test, you can run `./contrib/devtools/test_utxo_snapshots.sh` and follow the instructions. This is mostly obviated by the functional tests, though.
#### For real (longer)
If you'd like to experience a real usage of assumeutxo, you can do that too.
I've cut a new snapshot at height 788'000 (http://img.jameso.be/utxo-788000.dat - but you can do it yourself with `./contrib/devtools/utxo_snapshot.sh` if you want). Download that, and then create a datadir for testing:
```sh
$ cd ~/src/bitcoin # or whatever
# get the snapshot
$ curl http://img.jameso.be/utxo-788000.dat > utxo-788000.dat
# you'll want to do this if you like copy/pasting
$ export AU_DATADIR=/home/${USER}/au-test # or wherever
$ mkdir ${AU_DATADIR}
$ vim ${AU_DATADIR}/bitcoin.conf
dbcache=8000 # or, you know, something high
blockfilterindex=1
coinstatsindex=1
prune=3000
logthreadnames=1
```
Obtain this branch, build it, and then start bitcoind:
```sh
$ git remote add jamesob https://github.com/jamesob/bitcoin
$ git fetch jamesob assumeutxo
$ git checkout jamesob/assumeutxo
$ ./configure $conf_args && make # (whatever you like to do here)
# start 'er up and watch the logs
$ ./src/bitcoind -datadir=${AU_DATADIR}
```
Then, in some other window, load the snapshot
```sh
$ ./src/bitcoin-cli -datadir=${AU_DATADIR} loadtxoutset $(pwd)/utxo-788000.dat
```
You'll see some log messages about headers retrieval and waiting to see the snapshot in the headers chain. Once you get the full headers chain, you'll spend a decent amount of time (~10min) loading the snapshot, checking it, and flushing it to disk. After all that happens, you should be syncing to tip in pretty short order, and you'll see the occasional `[background validation]` log message go by.
In yet another window, you can check out chainstate status with
```sh
$ ./src/bitcoin-cli -datadir=${AU_DATADIR} getchainstates
```
as well as usual favorites like `getblockchaininfo`.
ACKs for top commit:
achow101:
ACK edbed31066
Tree-SHA512: 6086fb9a38dc7df85fedc76b30084dd8154617a2a91e89a84fb41326d34ef8e7d7ea593107afba01369093bf8cc91770621d98f0ea42a5b3b99db868d2f14dc2
Test that the getrawaddrman returns the addresses in the new and tried
tables. We can't check the buckets and positions as these are not
deterministic (yet).
380130d9d7 test: add coverage to feature_addrman.py (kevkevin)
Pull request description:
I added two new tests that will cover the nNew and nTried tests which add coverage to the if block by checking values larger than our range since we only check for negative values now
adding coverage to these lines
https://github.com/bitcoin/bitcoin/blob/master/src/addrman.cpp#L273https://github.com/bitcoin/bitcoin/blob/master/src/addrman.cpp#L280
our test seem to only cover the `nTried < 0` and `nNew < 0` scenarios
ACKs for top commit:
ismaelsadeeq:
ACK 380130d9d7, code looks good to me 🍃 .
0xB10C:
Re-ACK 380130d9d7
Tree-SHA512: a063bd9ca4d2d536a27c8c22a28fb13759a96f19cd8ba6cb8879cf7f65046d4ff6e8f70df17feaffd0d0d08ef914cb18a11258d313a4841c811a7e7ae4df6d5b
782701ce7d test: Test loading wallets with conflicts without a chain (Andrew Chow)
4660fc82a1 wallet: Check last block and conflict height are valid in MarkConflicted (Andrew Chow)
Pull request description:
`MarkConflicted` assumes that `m_last_block_processed_height` is always valid. However it may not be valid when a chain is not attached, as happens in the wallet tool and during migration. In such situations, when the conflicting height is also negative (which occurs on loading when no chain is available), the calculation of the number of conflict confirms results in a non-negative value which passes the existing check for valid values. This will subsequently hit an assertion in `GetTxDepthInMainChain`.
Furthermore, `MarkConflicted` is also only called on loading a transaction whose parent has a stored state of `TxStateConflicted` and was loaded before the child transaction. This depends on the loading order, which for both sqlite and bdb depends on the txids.
We can avoid this by explicitly checking that both `m_last_block_processed_height` and `conflicting_height` are non-negative. Both `tool_wallet.py` and `wallet_migration.py` are updated to create wallets with a state that triggers the assertion.
Fixes#28510
ACKs for top commit:
ryanofsky:
Code review ACK 782701ce7d. Nice catch, and clever test (grinding the txid)
furszy:
ACK 782701ce
Tree-SHA512: 1344e0279ec5413a43a2819d101fb571fbf4821de2d13958a0fdffc99f57082ef3243ec454c8343f97dc02ed1fce8c8b0fd89388420ab2e55618af42ad5630a9
2ab7952bda test: add bip157 coverage for (start height > stop height) disconnect (Sebastian Falbesoner)
63e90e1d3f test: check for specific disconnect reasons in p2p_blockfilters.py (Sebastian Falbesoner)
Pull request description:
This PR checks for specific disconnect reasons using `assert_debug_log` in the functional test `p2p_blockfilters.py`. With that we ensure that the disconnect happens for the expected reason and also makes it easier to navigate between implementation and test code, i.e. both the questions "do we have test coverage for this disconnect cause?" (from an implementation reader's perspective) and "where is the code handling this disconnect cause?" (from a test reader's perspective) can be answered simply by grep-ping the corresponding debug message.
Also, based on that, missing coverage for the (start height > stop height) disconnect case is added:
b7138252ac/src/net_processing.cpp (L3050-L3056)
ACKs for top commit:
MarcoFalke:
lgtm ACK 2ab7952bda
furszy:
Looks good, code ACK 2ab7952b
Tree-SHA512: 0581cb569d5935aaa004a95a6f16eeafe628b9d816ebb89232f2832e377049df878a1e74c369fb46931b94e1a3a5e3f4aaa21a007c0a488f4ad2cda0919c605d
f9047771d6 lint: fix custom mypy cache dir setting (Fabian Jahr)
Pull request description:
fixes#28183
The custom cache dir for `mypy` can only be set via an environment variable, setting the `MYPY_CACHE_DIR` variable in the program is not sufficient. This error was introduced while translating the shell script to python.
See also the mypy documentation: https://mypy.readthedocs.io/en/stable/config_file.html#confval-cache_dir
ACKs for top commit:
MarcoFalke:
lgtm ACK f9047771d6
Tree-SHA512: 7e8fb0cd06688129bd46d1afb8647262eb53d0f60b1ef6f288fedaa122d906fb62c9855e8bb0d6c6297d41a87a47d3cec7a00df55a7d033947937dfe23d07ba7
fa40b3ee22 test: Avoid test failure on Linux root without cap-add LINUX_IMMUTABLE (MarcoFalke)
Pull request description:
This turns a test failure on Linux when running the test as `root`, but without the `LINUX_IMMUTABLE` capability, into an early return, with a suggestion to turn on `LINUX_IMMUTABLE` next time (if possible).
ACKs for top commit:
pinheadmz:
utACK fa40b3ee22
jonatack:
ACK fa40b3ee22
Tree-SHA512: d986ff8aeae5f8267c21a23d5be16f7c5a4d4d3be045a6999d8b39c7b8672cfe915dedde762cc9965cdc4970940bffc4b0d1412833d8036d4425450eb6181f67
I added two new tests that will cover the nNew and nTried tests which
add coverage to the if block by checking values larger than our range
since we only check for negative values now
Co-authored-by: ismaelsadeeq <ask4ismailsadiq@gmail.com>
Add the script to the shellcheck exception list since the
quoted variables rule needs to be violated in order to get
bitcoind to pick up on $CHAIN_HACK_FLAGS.
96b3f2dbe4 test: add unit test coverage for Python ECDSA implementation (Sebastian Falbesoner)
Pull request description:
This PR adds missing unit test coverage for the Python ECDSA implementation, which should be useful for detecting potential problems early whenever changes in the test framework's Python implementation of secp256k1 are made (e.g. #26222). Note that right now we don't call `ECPubKey.verify_ecdsa` anywhere in our tests, so we wouldn't notice if it is broken at some point.
To keep it simple, the already existing unit test for Schnorr signatures is extended to also check ECDSA signatures. For that purpose, the dictionary storing private-key/public-key entries use their legacy types `ECKey/ECPubKey` instead of bare byte-arrays, and for Schnorr signing/verification the necessary conversions (ECKey -> bare private key, ECPubKey -> x-only pubkey) is done later when needed. To avoid code duplication, a helper function `random_bitflip` for damaging signatures is introduced.
The unit test can be run by either calling it for this single module:
`$ python3 -m unittest ./test/functional/test_framework/key.py`
or simply running `$ ./test/functional/test_runner.py` which calls all test framework module's unit tests at the start (see TEST_FRAMEWORK_MODULES list).
ACKs for top commit:
achow101:
ACK 96b3f2dbe4
sipa:
utACK 96b3f2dbe4
stratospher:
tested ACK 96b3f2d.
Tree-SHA512: b993f25b843fa047376addda4ce4b0f15750ffba926528b5cca4c5f99b9af456206f4e8af885d25a017dddddf382ddebf38765819b3d16a3f28810d03b010808
b5a962564e tests: Use manual bumps instead of bumpfee for resendwallettransactions (Andrew Chow)
Pull request description:
Bumpfee will try to increase the entire package to the target feerate, which causes repeated bumpfees to quickly shoot up in fees, causing intermittent failures when the fee is too large. We don't care about this property, just that the child is continuously replaced until we observe it's position in mapWallet is before its parent. Instead of using bumpfee, we can create raw transactions which have only pay (just above) the additional incremental relay fee, thus avoiding this problem.
Fixes#28491
ACKs for top commit:
kevkevinpal:
ACK [b5a9625](b5a962564e)
mzumsande:
Code review ACK b5a962564e
pablomartin4btc:
ACK b5a962564e -> adding the `try_rpc` to avoid (skip) any possible failure around the manual bump fee (if we ever reach it as [explained](https://github.com/bitcoin/bitcoin/pull/28540#issuecomment-1737648048)) makes a lot of sense as the spirit of the test is the tx (child before parent) sort in the `mapWallet` (as also [explained](https://github.com/bitcoin/bitcoin/issues/28491#issuecomment-1736161363)).
MarcoFalke:
lgtm ACK b5a962564e
Tree-SHA512: f184f11c73be0c30753181901f51a3b4b9c4135e0c4681e9f4ca94692c49bac15c91683c85266a2124333c8593e9919bfd9102724616faab299740f2eb98741f
b3db8c9d5c rpc: bumpfee, improve doc for 'reduce_output' arg (furszy)
Pull request description:
Fixes#28180. Resulted from discussions with S3RK, achow101, and Murch.
The current argument name and description are dangerous as it don't
describe the case where the user selects the recipient output as the
change address. This one could end up been increased by the inputs
minus outputs remainder. Which, when `bumpfee` adds new inputs
to the transaction, leads the process to send more coins to the
recipient. Which is not what the user would expect from a
'reduce_output' param naming.
ACKs for top commit:
S3RK:
ACK b3db8c9d5c
achow101:
ACK b3db8c9d5c
murchandamus:
ACK b3db8c9d5c
Tree-SHA512: 91f607e2f5849041d7c099afdddae11af8bed5b1ac90c9d22921267f272e21b44e107d6968e037f05f958a61fe29e94e5fb44b224fb3606f197f83ec4ba3b1e7
Bumpfee will try to increase the entire package to the target feerate,
which causes repeated bumpfees to quickly shoot up in fees, causing
intermittent failures when the fee is too large. We don't care about
this property, just that the child is continuously replaced until we
observe it's position in mapWallet is before its parent. Instead of
using bumpfee, we can create raw transactions which have only pay the
additional incremental relay fee, thus avoiding this problem.
Loading a wallet with conflicts without a chain (e.g. wallet tool and
migration) would previously result in an assertion due to -1 being both
a valid number of conflict confirmations, and the indicator that that
member has not been set yet.
The current argument name and description are dangerous as it don't
describe the case where the user selects the recipient output as the
change address. This one could end up been increased by the inputs
minus outputs remainder. Which, when bumpfee adds new inputs
to the transaction, leads the process to send more coins to the
recipient. Which is not what the user would expect from a
'reduce_output' param naming.
Co-authored-by: Murch <murch@murch.one>
a99e9e655a doc: add release note (ismaelsadeeq)
2b4edf889a test: check `descriptorprocesspsbt` return hex encoded tx (ismaelsadeeq)
c405207a18 rpc: `descriptorprocesspsbt` return hex encoded tx (ismaelsadeeq)
Pull request description:
Coming from [#28414 comment](https://github.com/bitcoin/bitcoin/pull/28414#pullrequestreview-1618684391) Same thing also for `descriptorprocesspsbt`.
Before this PR `descriptorprocesspsbt` returns a boolean `complete` which indicates that the psbt is final, users then have to call `finalizepsbt` to get the hex encoded network transaction.
In this PR if the psbt is complete the return object also has the hex encoded network transaction ready for broadcast with `sendrawtransaction`.
This save users calling `finalizepsbt` with the descriptor, if it is already complete.
ACKs for top commit:
achow101:
ACK a99e9e655a
pinheadmz:
ACK a99e9e655a
ishaanam:
ACK a99e9e655a
Tree-SHA512: c3f1b1391d4df05216c463127cd593f8703840430a99febb54890bc66fadabf9d9530860605f347ec54c1694019173247a0e7a9eb879d3cbb420f9e8d9839b75
eb8f58f5e4 Add functional test to catch too large vsize packages (Greg Sanders)
1a579f9d01 Handle over-sized (in virtual bytes) packages with no in-mempool ancestors (Greg Sanders)
bc013fe8e3 Bugfix: Pass correct virtual size to CheckPackageLimits (Luke Dashjr)
533660c58a Replace MAX_PACKAGE_SIZE with MAX_PACKAGE_WEIGHT to avoid vbyte confusion (Greg Sanders)
Pull request description:
(Alternative) Minimal subset of https://github.com/bitcoin/bitcoin/pull/28345 to:
1) Replace MAX_PACKAGE_SIZE with MAX_PACKAGE_WEIGHT which accounts for additional WU necessary to not exclude default chain limit transactions that would have been accepted individually. Avoids sigops vbyte confusion.
2) pass correct vsize to chain limit evaluations in package context
3) stop overly-large packages that have no existing mempool ancestors (also a bugfix by itself if someone sets non-standard chain limits)
This should fix the known issues while not blocking additional refactoring later.
ACKs for top commit:
achow101:
ACK eb8f58f5e4
ariard:
Re-Code ACK eb8f58f5e
glozow:
reACK eb8f58f5e4
Tree-SHA512: 1b5cca1a526207e25d387fcc29a776a3198c3a013dc2b35c6275b9d5a64db2476c154ebf52e3a1aed0b9924c75613f21a946577aa760de28cadf0c9c7f68dc39
d05be124db test: added coverage to estimatefee (kevkevin)
Pull request description:
Added a assert for an rpc error when we try to estimate fee for the max conf_target
Line I am adding coverage to https://github.com/bitcoin/bitcoin/blob/master/src/rpc/fees.cpp#LL71C52-L71C52
ACKs for top commit:
MarcoFalke:
lgtm ACK d05be124db
Tree-SHA512: dfab075989446e33d1a5ff1a308f1ba1b9f80cce3848fbe4231f69212ceef456a3f2b19365a42123e0397c31893fd9f1fd9973cc00cfbb324386e12ed0e6bccc
