7de7685372 test, assumeutxo: Use assert_debug_log for error details (pablomartin4btc)
Pull request description:
This is a follow-up on the invalid hash dump fix#28698, [suggested](https://github.com/bitcoin/bitcoin/pull/28698#pullrequestreview-1698178157) by theStack and agreed by Sjors and ryanofsky.
ACKs for top commit:
Sjors:
ACK 7de7685372
maflcko:
lgtm ACK 7de7685372
Tree-SHA512: 036b3cef3084e3ead8923e8dcabe4fa7ebe97fb514d223aa38bc38df10337e3fe3113e42322178b58fb03fcd4511af4b5b56bceecbb7ded5b9758842c70db3f2
faa2ad88bc test: Add missing wait for version to be sent in add_outbound_p2p_connection (MarcoFalke)
Pull request description:
Can be tested with:
```diff
diff --git a/test/functional/test_framework/p2p.py b/test/functional/test_framework/p2p.py
index b1ed97b794..eb4f72c6b6 100755
--- a/test/functional/test_framework/p2p.py
+++ b/test/functional/test_framework/p2p.py
@@ -205,6 +205,7 @@ class P2PConnection(asyncio.Protocol):
assert not self._transport
logger.debug("Connected & Listening: %s:%d" % (self.dstaddr, self.dstport))
self._transport = transport
+ import time;time.sleep(.1);
if self.on_connection_send_msg:
self.send_message(self.on_connection_send_msg)
self.on_connection_send_msg = None # Never used again
```
Found and reported by mzumsande in https://github.com/bitcoin/bitcoin/pull/28782#pullrequestreview-1718560252
ACKs for top commit:
mzumsande:
ACK faa2ad88bc
Tree-SHA512: 863f06125dec40cccaa852d0d8ca2e2b9c0b74610205e9fd6c9c279bdf36801ff475e3d873fd1b18172eb8220e17b2caff60069ce63512e569934a43f27d03fd
0420f99f42 Create net_peer_connection unit tests (Jon Atack)
4b834f6499 Allow unit tests to access additional CConnman members (Jon Atack)
34b9ef443b net/rpc: Makes CConnman::GetAddedNodeInfo able to return only non-connected address on request (Sergi Delgado Segura)
94e8882d82 rpc: Prevents adding the same ip more than once when formatted differently (Sergi Delgado Segura)
2574b7e177 net/rpc: Check all resolved addresses in ConnectNode rather than just one (Sergi Delgado Segura)
Pull request description:
## Rationale
Currently, `addnode` has a couple of corner cases that allow it to either connect to the same peer more than once, hence wasting outbound connection slots, or add redundant information to `m_added_nodes`, hence making Bitcoin iterate through useless data on a regular basis.
### Connecting to the same node more than once
In general, connecting to the same node more than once is something we should try to prevent. Currently, this is possible via `addnode` in two different ways:
1. Calling `addnode` more than once in a short time period, using two equivalent but distinct addresses
2. Calling `addnode add` using an IP, and `addnode onetry` after with an address that resolved to the same IP
For the former, the issue boils down to `CConnman::ThreadOpenAddedConnections` calling `CConnman::GetAddedNodeInfo` once, and iterating over the result to open connections (`CConman::OpenNetworkConnection`) on the same loop for all addresses.`CConnman::ConnectNode` only checks a single address, at random, when resolving from a hostname, and uses it to check whether we are already connected to it.
An example to test this would be calling:
```
bitcoin-cli addnode "127.0.0.1:port" add
bitcoin-cli addnode "localhost:port" add
```
And check how it allows us to perform both connections some times, and some times it fails.
The latter boils down to the same issue, but takes advantage of `onetry` bypassing the `CConnman::ThreadOpenAddedConnections` logic and calling `CConnman::OpenNetworkConnection` straightaway. A way to test this would be:
```
bitcoin-cli addnode "127.0.0.1:port" add
bitcoin-cli addnode "localhost:port" onetry
```
### Adding the same peer with two different, yet equivalent, addresses
The current implementation of `addnode` is pretty naive when checking what data is added to `m_added_nodes`. Given the collection stores strings, the checks at `CConnman::AddNode()` basically check wether the exact provided string is already in the collection. If so, the data is rejected, otherwise, it is accepted. However, ips can be formatted in several ways that would bypass those checks.
Two examples would be `127.0.0.1` being equal to `127.1` and `[::1]` being equal to `[0:0:0:0:0:0:0:1]`. Adding any pair of these will be allowed by the rpc command, and both will be reported as connected by `getaddednodeinfo`, given they map to the same `CService`.
This is less severe than the previous issue, since even tough both nodes are reported as connected by `getaddednodeinfo`, there is only a single connection to them (as properly reported by `getpeerinfo`). However, this adds redundant data to `m_added_nodes`, which is undesirable.
### Parametrize `CConnman::GetAddedNodeInfo`
Finally, this PR also parametrizes `CConnman::GetAddedNodeInfo` so it returns either all added nodes info, or only info about the nodes we are **not** connected to. This method is used both for `rpc`, in `getaddednodeinfo`, in which we are reporting all data to the user, so the former applies, and to check what nodes we are not connected to, in `CConnman::ThreadOpenAddedConnections`, in which we are currently returning more data than needed and then actively filtering using `CService.fConnected()`
ACKs for top commit:
jonatack:
re-ACK 0420f99f42
kashifs:
> > tACK [0420f9](0420f99f42)
sr-gi:
> > > tACK [0420f9](0420f99f42)
mzumsande:
Tested ACK 0420f99f42
Tree-SHA512: a3a10e748c12d98d439dfb193c75bc8d9486717cda5f41560f5c0ace1baef523d001d5e7eabac9fa466a9159a30bb925cc1327c2d6c4efb89dcaf54e176d1752
6559e4d27a tests: Increase wallet_miniscript.py rpc timeout to 90 seconds (Andrew Chow)
Pull request description:
The signing test for the large miniscript can sometimes take longer than the 30 second timeout, depending on the load on my system. Increasing it to 90 seconds seems to be good enough.
ACKs for top commit:
kevkevinpal:
but increasing seems fine ACK [6559e4d](6559e4d27a)
maflcko:
lgtm ACK 6559e4d27a
Tree-SHA512: 1b7bf94c77f85a0deddb1384aacbeb934205d0a630fecc8e75a4a98d1946d77d9bca36692fb6c1ab8e9276392f617281aafc4c685c248a8d3b0c77f896cda624
fa02598469 test: Add missing sync on send_version in peer_connect (MarcoFalke)
Pull request description:
Without the sync, the logic will be racy. For example, `p2p_sendtxrcncl.py` is failing locally (and on CI occasionally), because non-version messages will be sent before the version message:
```py
self.log.info('SENDTXRCNCL with version=0 triggers a disconnect')
sendtxrcncl_low_version = create_sendtxrcncl_msg()
sendtxrcncl_low_version.version = 0
peer = self.nodes[0].add_p2p_connection(PeerNoVerack(), send_version=True, wait_for_verack=False)
with self.nodes[0].assert_debug_log(["txreconciliation protocol violation"]):
peer.send_message(sendtxrcncl_low_version)
peer.wait_for_disconnect()
```
```
test 2023-11-02T08:15:19.620000Z TestFramework (INFO): SENDTXRCNCL with version=0 triggers a disconnect
test 2023-11-02T08:15:19.621000Z TestFramework.p2p (DEBUG): Connecting to Bitcoin Node: 127.0.0.1:11312
test 2023-11-02T08:15:19.624000Z TestFramework.p2p (DEBUG): Connected & Listening: 127.0.0.1:11312
test 2023-11-02T08:15:19.798000Z TestFramework.p2p (DEBUG): Send message to 127.0.0.1:11312: msg_sendtxrcncl(version=0, salt=2)
test 2023-11-02T08:15:19.799000Z TestFramework.p2p (DEBUG): Send message to 127.0.0.1:11312: msg_version(nVersion=70016 nServices=9 nTime=Thu Nov 2 08:15:19 2023 addrTo=CAddress(nServices=1 net=IPv4 addr=127.0.0.1 port=11312) addrFrom=CAddress(nServices=1 net=IPv4 addr=0.0.0.0 port=0) nNonce=0x369AC031CDA96022 strSubVer=/python-p2p-tester:0.0.3/ nStartingHeight=-1 relay=1)
node0 2023-11-02T08:15:19.804409Z [net] [net.cpp:3676] [CNode] [net] Added connection peer=0
node0 2023-11-02T08:15:19.805256Z [net] [net.cpp:1825] [CreateNodeFromAcceptedSocket] [net] connection from 127.0.0.1:55964 accepted
node0 2023-11-02T08:15:19.809861Z [msghand] [net_processing.cpp:3356] [ProcessMessage] [net] received: sendtxrcncl (12 bytes) peer=0
node0 2023-11-02T08:15:19.810297Z [msghand] [net_processing.cpp:3582] [ProcessMessage] [net] non-version message before version handshake. Message "sendtxrcncl" from peer=0
node0 2023-11-02T08:15:19.810928Z [msghand] [net_processing.cpp:3356] [ProcessMessage] [net] received: version (111 bytes) peer=0
...
test 2023-11-02T09:35:20.166000Z TestFramework.utils (ERROR): wait_until() failed. Predicate: ''''
def test_function():
if check_connected:
assert self.is_connected
return test_function_in()
'''
test 2023-11-02T09:35:20.187000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
File "/ci_container_base/ci/scratch/build/bitcoin-s390x-linux-gnu/test/functional/test_framework/test_framework.py", line 132, in main
self.run_test()
File "/ci_container_base/ci/scratch/build/bitcoin-s390x-linux-gnu/test/functional/p2p_sendtxrcncl.py", line 188, in run_test
peer.wait_for_disconnect()
File "/ci_container_base/ci/scratch/build/bitcoin-s390x-linux-gnu/test/functional/test_framework/p2p.py", line 478, in wait_for_disconnect
self.wait_until(test_function, timeout=timeout, check_connected=False)
File "/ci_container_base/ci/scratch/build/bitcoin-s390x-linux-gnu/test/functional/test_framework/p2p.py", line 470, in wait_until
wait_until_helper_internal(test_function, timeout=timeout, lock=p2p_lock, timeout_factor=self.timeout_factor)
File "/ci_container_base/ci/scratch/build/bitcoin-s390x-linux-gnu/test/functional/test_framework/util.py", line 275, in wait_until_helper_internal
raise AssertionError("Predicate {} not true after {} seconds".format(predicate_source, timeout))
AssertionError: Predicate ''''
def test_function():
if check_connected:
assert self.is_connected
return test_function_in()
''' not true after 4800.0 seconds
ACKs for top commit:
mzumsande:
ACK fa02598469
Tree-SHA512: 78871f603d387e2df8c0acbdfa95441fa186f80e94593021bb219bbf1bc9dc7efc4e266bd254b5cc41114c38227ff3b7f6172335d9bb828427f0a2acffde752d
The signing test for the large miniscript can sometimes take longer than
the 30 second timeout, depending on the load on my system. Increasing it
to 90 seconds seems to be good enough.
376dc2cfb3 test: add coverage to rpc_blockchain.py (kevkevin)
Pull request description:
Included a test that checks the functionality of setting
the first param of getnetworkhashps to negative value returns
the average network hashes per second from the last difficulty change.
ACKs for top commit:
jlopp:
tACK 376dc2cfb3
achow101:
ACK 376dc2cfb3
ismaelsadeeq:
Tested ACK 376dc2cfb3
pablomartin4btc:
tACK 376dc2cfb3
Tree-SHA512: 02d52f622e9cb7a1240c5d124510dd75d03f696f119b2625b0befd60b004ec50ff1a2d5515e0e227601adeecd837e0778ed131ee2a8c5f75f1b824be711213a7
9cfc1c9440 test: check that we don't send a getaddr msg to an inbound peer (Martin Zumsande)
88c33c6748 test: make python p2p not send getaddr messages when it's being connected to (Martin Zumsande)
Pull request description:
`bitcoind` nodes send `getaddr` messages only to outbound nodes (and ignore `getaddr` received by outgoing connections).
The python p2p node should mirror this behavior by not sending a `getaddr` message when it is not the initiator of the connection.
This is currently causing several unnecessary messages being sent and then ignored (`Ignoring "getaddr" from outbound-full-relay connection.`) in tests like `p2p_add_connections.py`.
ACKs for top commit:
pinheadmz:
concept ACK 9cfc1c9440
pablomartin4btc:
re ACK 9cfc1c9440
BrandonOdiwuor:
re ACK 9cfc1c9440
Tree-SHA512: 812bec5d8a4828b4384d4cdd4362d6eec09acb2363e888f2b3e3bf8b925e0e17f15e13dc297d6b616c68b93ace9ede7245b07b405d3f5f8eada98350f74230dc
02a4f1a385 addrman: log AS only when using asmap (brunoerg)
Pull request description:
This PR changes the log to just print the ASN when using asmap, same logic presented in other logs:
afa081a39b/src/net_processing.cpp (L3552-L3556)afa081a39b/src/net_processing.cpp (L3598-L3604)
ACKs for top commit:
naumenkogs:
ACK 02a4f1a385
mzumsande:
Code Review ACK 02a4f1a385
Tree-SHA512: adad5904ab163660d47554b32dc2dc3dfdff8dd64b94e5320ad11706381264d1e338654fa8239430eed4ccbebc8f6670698b4278895794055c37fc4bcefe71bc
Currently it is possible to add the same node twice when formatting IPs in
different, yet equivalent, manner. This applies to both ipv4 and ipv6, e.g:
127.0.0.1 = 127.1 | [::1] = [0:0:0:0:0:0:0:1]
`addnode` will accept both and display both as connected (given they translate to
the same IP). This will not result in multiple connections to the same node, but
will report redundant info when querying `getaddednodeinfo` and populate `m_added_nodes`
with redundant data.
This can be avoided performing comparing the contents of `m_added_addr` and the address
to be added as `CServices` instead of as strings.
811067ca1c test: add coverage for snapshot chainstate not matching AssumeUTXO parameters (pablomartin4btc)
4a5be10b92 assumeutxo, blockstorage: prevent core dump on invalid hash (pablomartin4btc)
Pull request description:
While reviewing #27596 (ran `loadtxoutset` in `mainnet` before `m_assumeutxo_data` is empty as [currently](434495a8c1/src/kernel/chainparams.cpp (L175-L177)) in master - back to 1b1d711), got a `core dumped`, so it seems there's a potential issue if new releases ever remove snapshot details or a semi-experienced user performs a `loadtxoutset` on a different "customised" binary version (not sure if this is a real use case).
```
2023-10-18T17:42:52Z [init] Using obfuscation key for /tmp/.test_utxo_2/blocks/index: 0000000000000000
node/blockstorage.cpp:390 LoadBlockIndex: Assertion `GetParams().AssumeutxoForBlockhash(*snapshot_blockhash)' failed.
Aborted (core dumped)
```
<details>
<summary>This is also happening before IBD is completed (<code>background validation</code> still being performed as it can be seen in rpc <code>getchainstates</code>)</summary>
```
/src/bitcoin-cli -datadir=${AU_DATADIR} getchainstates
{
"headers": 813097,
"chainstates": [
{
"blocks": 368249,
"bestblockhash": "00000000000000000b7a08224a1cb00d337100ba7a46c03d04b2c2d8964efc37",
"difficulty": 52278304845.59168,
"verificationprogress": 0.086288278873286,
"coins_db_cache_bytes": 7969177,
"coins_tip_cache_bytes": 14908338995,
"validated": true
},
{
"blocks": 813097,
"bestblockhash": "0000000000000000000270c9fdce7b17db64cca91f90106964b58e33a4d91089",
"difficulty": 61030681983175.59,
"verificationprogress": 0.999997140098457,
"coins_db_cache_bytes": 419430,
"coins_tip_cache_bytes": 784649420,
"snapshot_blockhash": "00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054",
"validated": false
}
]
}
```
</details>
<details>
<summary>Steps to reproduce the core dump error and its output:</summary>
1. Perform a `loadtxoutset` in `mainnet` on compiled `bitcoind` adding the block hash from Sjors's [commit](24deb2022b).
2. Once step 1 finishes, remove the added code from step 1 and compile again or just compile `master` without any changes on top.
3. Run `bitcoind`, soon it'll crash with:
```
2023-10-18T17:42:52Z [init] init message: Loading block index…
2023-10-18T17:42:52Z [init] Assuming ancestors of block 00000000000000000001a0a448d6cf2546b06801389cc030b2b18c6491266815 have valid signatures.
2023-10-18T17:42:52Z [init] Setting nMinimumChainWork=000000000000000000000000000000000000000052b2559353df4117b7348b64
2023-10-18T17:42:52Z [init] Prune configured to target 3000 MiB on disk for block and undo files.
2023-10-18T17:42:52Z [init] [snapshot] detected active snapshot chainstate (/tmp/.test_utxo_2/chainstate_snapshot) - loading
2023-10-18T17:42:52Z [init] [snapshot] switching active chainstate to Chainstate [snapshot] @ height -1 (null)
2023-10-18T17:42:52Z [init] Opening LevelDB in /tmp/.test_utxo_2/blocks/index
2023-10-18T17:42:52Z [init] Opened LevelDB successfully
2023-10-18T17:42:52Z [init] Using obfuscation key for /tmp/.test_utxo_2/blocks/index: 0000000000000000
node/blockstorage.cpp:390 LoadBlockIndex: Assertion `GetParams().AssumeutxoForBlockhash(*snapshot_blockhash)' failed.
Aborted (core dumped)
```
</details>
<details>
<summary>After original change, error message output:</summary>
```
2023-10-20T15:49:12Z [init] init message: Loading block index…
2023-10-20T15:49:12Z [init] Assuming ancestors of block 00000000000000000001a0a448d6cf2546b06801389cc030b2b18c6491266815 have valid signatures.
2023-10-20T15:49:12Z [init] Setting nMinimumChainWork=000000000000000000000000000000000000000052b2559353df4117b7348b64
2023-10-20T15:49:12Z [init] Prune configured to target 3000 MiB on disk for block and undo files.
2023-10-20T15:49:12Z [init] [snapshot] detected active snapshot chainstate (/tmp/.test_utxo_2/chainstate_snapshot) - loading
2023-10-20T15:49:12Z [init] [snapshot] switching active chainstate to Chainstate [snapshot] @ height -1 (null)
2023-10-20T15:49:12Z [init] Opening LevelDB in /tmp/.test_utxo_2/blocks/index
2023-10-20T15:49:12Z [init] Opened LevelDB successfully
2023-10-20T15:49:12Z [init] Using obfuscation key for /tmp/.test_utxo_2/blocks/index: 0000000000000000
2023-10-20T15:49:13Z [init] *** Assumeutxo data not found for the given blockhash '00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054'.
2023-10-20T15:49:13Z [init] Error: Assumeutxo data not found for the given blockhash '00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054'.
Error: Assumeutxo data not found for the given blockhash '00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054'.
2023-10-20T15:49:13Z [init] Shutdown requested. Exiting.
2023-10-20T15:49:13Z [init] Shutdown: In progress...
2023-10-20T15:49:13Z [scheduler] scheduler thread exit
2023-10-20T15:49:13Z [shutoff] Flushed fee estimates to fee_estimates.dat.
2023-10-20T15:49:13Z [shutoff] Shutdown: done
```
</details>
<details>
<summary>Alternative on error handling using <code>return error()</code> instead of <code>return FatalError()</code> used in this PR, which produces a different output and perhaps confusing:</summary>
```
2023-10-20T21:45:58Z [init] Using obfuscation key for /tmp/.test_utxo_2/blocks/index: 0000000000000000
2023-10-20T21:45:59Z [init] ERROR: Assumeutxo data not found for the given blockhash '00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054'.
2023-10-20T21:45:59Z [init] : Error loading block database.
Please restart with -reindex or -reindex-chainstate to recover.
: Error loading block database.
Please restart with -reindex or -reindex-chainstate to recover.
2023-10-20T21:45:59Z [init] Aborted block database rebuild. Exiting.
2023-10-20T21:45:59Z [init] Shutdown: In progress...
2023-10-20T21:45:59Z [scheduler] scheduler thread exit
2023-10-20T21:45:59Z [shutoff] Flushed fee estimates to fee_estimates.dat.
2023-10-20T21:45:59Z [shutoff] Shutdown: done
```
</details>
<details>
<summary>Current state (including ryanofsky <a href="https://github.com/bitcoin/bitcoin/pull/28698#discussion_r1368635965">suggestion</a>), after code change, error message output:</summary>
```
2023-10-25T02:29:57Z [init] Using obfuscation key for /home/pablo/.test_utxo_2/regtest/blocks/index: 0000000000000000
2023-10-25T02:29:57Z [init] *** Assumeutxo data not found for the given blockhash 'f09b5835f3f8b39481f2af3257bbc2e82845552d4d2d6d31cf520fc24263ed5b'.
2023-10-25T02:29:57Z [init] Error: A fatal internal error occurred, see debug.log for details
Error: A fatal internal error occurred, see debug.log for details
2023-10-25T02:29:57Z [init] Shutdown requested. Exiting.
2023-10-25T02:29:57Z [init] Shutdown: In progress...
2023-10-25T02:29:57Z [scheduler] scheduler thread exit
2023-10-25T02:29:57Z [shutoff] Flushed fee estimates to fee_estimates.dat.
2023-10-25T02:29:57Z [shutoff] Shutdown: done
```
</details>
ACKs for top commit:
naumenkogs:
ACK 811067ca1c
theStack:
ACK 811067ca1c
ryanofsky:
Code review ACK 811067ca1c.
Tree-SHA512: cfc137b0a4f638b99fd7dac2c35cc729ef71ae1166a2a8960a91055ec90841cb33aed589834012cfe0e157937e2a76a88d1020ea1df2bc98e1114eb1fc8eaae4
fe3ac3700d test: replace random_bytes with randbytes #28720 (ns-xvrn)
Pull request description:
With Python upgraded to 3.9 replaced the `random_bytes` function in util of functional tests and replaced it's usage with `random.randbytes`.
Closes#28720.
ACKs for top commit:
maflcko:
lgtm ACK fe3ac3700d
BrandonOdiwuor:
ACK fe3ac3700d
stickies-v:
ACK fe3ac3700d, thanks for picking this up
kristapsk:
utACK fe3ac3700d
Tree-SHA512: f65a75e73ebd840c2936eb133d42bccd552f25b717c8ca25c18d06e0593e12f292389cfcc0a0b0759004b67a46ea0c8ac237973ef90f246139778230be1e64e1
50d1ac1207 test: remove unused `find_output` helper (Sebastian Falbesoner)
73a339abc3 test: refactor: support sending funds with outpoint result (Sebastian Falbesoner)
Pull request description:
In wallet-related functional tests we often want to send funds to an address and use the resulting (non-change) UTXO directly after as input for another transaction. Doing that is currently tedious, as it involves finding the index part of the outpoint manually by calling helpers like `find_vout_for_address` or `find_output` first. This results in two different txid/vout variables which then again have to be combined to a single dictionary `{"txid": ..., "vout": ...}` in order to be specified as input for RPCs like `createrawtransaction` or `createpsbt`. For example:
```
txid1 = node1.sendtoaddress(addr1, value1)
vout1 = find_vout_for_address(node1, txid1, addr1)
txid2 = node2.sendtoaddress(addr2, value2)
vout2 = find_vout_for_address(node2, txid2, addr2)
node.createrawtransaction([{'txid': txid1, 'vout': vout1}, {'txid': txid2, 'vout': vout2}], .....)
```
This PR introduces a helper `create_outpoints` to immediately return the outpoint as
UTXO dictionary in the common format, making the tests more readable and avoiding unnecessary duplication:
```
utxo1 = self.create_outpoints(node1, outputs=[{addr1: value1}])[0]
utxo2 = self.create_outpoints(node2, outputs=[{addr2: value2}])[0]
node.createrawtransaction([utxo1, utxo2], .....)
```
Tests are switched to work with UTXO-objects rather than two individual txid/vout variables accordingly.
The `find_output` helper is removed, as it seems generally a bad idea to search for an outpoint only based on the output value. If that's really ever needed in the future, it makes probably more sense to add it as an additional parameter to `find_vout_of_address`. Note that `find_output` supported specifying a block-hash for where to look for the transaction (being passed on to the `getrawtransaction` RPC). This seems to be unneeded, as txids are always unique and for the only test that used that parameter (rpc_psbt.py) there was no observed difference in run-time, so it was not reintroduced in the new helper.
There are still some `find_vout_of_address` calls remaining, used for detecting change outputs or for whenever the sending happens via `sendrawtransaction` instead, so this PR tackles not all, but the most common case.
ACKs for top commit:
achow101:
ACK 50d1ac1207
BrandonOdiwuor:
ACK 50d1ac1207
maflcko:
ACK 50d1ac1207 🖨
Tree-SHA512: af2bbf13a56cc840fefc1781390cf51625f1e41b3c030f07fc9abb1181b2d414ddbf795e887db029e119cbe45de14f7c987c0cba72ff0b8953080ee218a7915a
fa25e8b0a1 doc: Recommend lint image build on every call (MarcoFalke)
faf70c1f33 Bump python minimum version to 3.9 (MarcoFalke)
fa8996b930 ci: Bump i686_multiprocess.sh to latest Ubuntu LTS (MarcoFalke)
Pull request description:
All supported operating systems ship with python 3.9 (or later), so bumping the minimum should not cause any issues. A bump will allow new code to use new python 3.9 features.
For reference:
* https://packages.debian.org/bullseye/python3
* https://packages.ubuntu.com/focal/python3.9
* FreeBSD 12/13 also ships with 3.9
* CentOS-like 8/9 also ships with 3.9 (and 3.11)
* OpenSuse Leap also ships with 3.9 (and 3.11) https://software.opensuse.org/package/python311-base
This is for Bitcoin Core 27.0 in 2024 (next year), not the soon upcoming 26.0 next month.
ACKs for top commit:
Sjors:
ACK fa25e8b0a1
jamesob:
ACK fa25e8b0a1 ([`jamesob/ackr/28211.1.MarcoFalke.bump_python_minimum_supp`](https://github.com/jamesob/bitcoin/tree/ackr/28211.1.MarcoFalke.bump_python_minimum_supp))
Tree-SHA512: 86c9f6ac4b5ba94a62ee6a6062dd48a8295d8611a39cdb5829f4f0dbc77aaa1a51edccc7a99275bf699143ad3a6fe826de426d413e5a465e3b0e82b86d10c32e
5a0688a20d test: enable reindex readonly test on *BSD and macOS as root (Matthew Zipkin)
Pull request description:
see https://github.com/bitcoin/bitcoin/pull/27850#discussion_r1349505585
OpenBSD and FreeBSD don't have `chattr` but they do have `chflags`, use that method to make the block file immutable for the reindex_readonly test.
Written and tested on a VPS running FreeBSD:
```
FreeBSD freebsd-13-1 13.2-RELEASE-p4 FreeBSD 13.2-RELEASE-p4 GENERIC amd64
```
ACKs for top commit:
maflcko:
re-cr-lgtm-ACK 5a0688a20d
jonatack:
ACK 5a0688a20d tested on macOS only
theStack:
ACK 5a0688a20d
Tree-SHA512: 8c88d282d09c00355d22c4c504b779f60e420327a5e07bcf80fa77b97fefcb04952af9ceaf439d9033a0a2448cb26a02663fe6bddcd4a74792857cfbaf1c5162
This commit introduces a helper `create_outpoints` to execute the
`send` RPC and immediately return the target address outpoints as UTXO
dictionary in the common format, making the tests more readable and
avoiding unnecessary duplication.
4814e4063e test: Check tx metadata is migrated to watchonly (Andrew Chow)
d616d30ea5 wallet: Reload watchonly and solvables wallets after migration (Andrew Chow)
118f2d7d70 wallet: Copy all tx metadata to watchonly wallet (Andrew Chow)
9af87cf348 test: Check that a failed wallet migration is cleaned up (Andrew Chow)
Pull request description:
Some incomplete/incorrect state as a result of migration can be mitigated/cleaned up by simply restarting the migrated wallets. We already do this for a wallet when it is migrated, but we do not for the new watchonly and solvables wallets that may be created. This PR introduces this behavior, in addition to creating those wallets initially without an attached chain.
While implementing this, I noticed that not all `CWalletTx` metadata was being copied over to the watchonly wallet and so some data, such as time received, was being lost. This PR fixes this as a side effect of not having a chain attached to the watchonly wallet. A test has also been added.
ACKs for top commit:
ishaanam:
light code review ACK 4814e4063e
ryanofsky:
Code review ACK 4814e4063e. Just implemented the suggested orderpos, copyfrom, and path set comments since last review
furszy:
ACK 4814e406
Tree-SHA512: 0b992430df9f452cb252c2212df8e876613f43564fcd1dc00c6c31fa497adb84dfff6b5ef597590f9b288c5f64cb455f108fcc9b6c9d1fe9eb2c39e7f2c12a89
The legacy serialization was vulnerable to maleation and is fixed by
adopting the same serialization procedure as was already in use for
MuHash.
This also includes necessary test fixes where the hash_serialized2 was
hardcoded as well as correction of the regtest chainparams.
Co-authored-by: Sebastian Falbesoner <sebastian.falbesoner@gmail.com>
When migrating, create the watchonly and solvables wallets without a
context. Then unload and reload them after migration completes, as we do
for the actual wallet.
There is also additional handling for a failed reload.
004903ebad test: Add Wallet Unlock Context Manager (Brandon Odiwuor)
Pull request description:
Fixes#28601, see https://github.com/bitcoin/bitcoin/pull/28403#discussion_r1325426430
Add Context Manager to manage the locking and unlocking of locked wallets with a passphrase during testing.
ACKs for top commit:
kevkevinpal:
lgtm ACK [004903e](004903ebad)
maflcko:
lgtm ACK 004903ebad
Tree-SHA512: ab234c167e71531df0d974ff9a31d444f7ce2a1d05aba5ea868cc9452f139845eeb24ca058d88f058bc02482b762adf2d99e63a6640b872cc71a57a0068abfe8
fa4c6836c9 test: Fix failing time check in rpc_net.py (MarcoFalke)
Pull request description:
This check fails on slow runners, such as s390x qemu.
Fix it by using mocktime.
See https://github.com/bitcoin/bitcoin/pull/28523#discussion_r1357980527
ACKs for top commit:
0xB10C:
ACK fa4c6836c9
pinheadmz:
ACK fa4c6836c9
brunoerg:
crACK fa4c6836c9
Tree-SHA512: 83fb534682e11e97537dc89de8c16f206f38af1a892a2d5970c02684c05eaea8fc9adba3159f16b2440ca0b3871d513a0562a6f3a38f19a5574a47be0919e42f
621db2f004 test: assumeutxo file with unknown block hash (Fabian Jahr)
Pull request description:
Takes care of one of the open Todos in the assumeutxo functional test. Since an unknown block could be any hash, I simply chose one placeholder, it could also be a random string though.
ACKs for top commit:
maflcko:
lgtm ACK 621db2f004
pablomartin4btc:
cr ACK 621db2f004
theStack:
ACK 621db2f004
ryanofsky:
Code review ACK 621db2f004
Tree-SHA512: ee0438ce619f7348c6f88e39b0ea7ddddb8832956d9034ecc795c6033d5d905c09d11b7d0d5afc38231b2fd091ea7c1bd0a0be99d9c32c4e6357a25d76294142
9620cb4493 assumeutxo: fail early if snapshot block hash doesn't match AssumeUTXO parameters (Sebastian Falbesoner)
Pull request description:
Right now the `loadtxoutset` RPC call treats literally all files with a minimum size of 40 bytes (=size of metadata) as potential valid snapshot candidates and the waiting loop for seeing the metadata block hash in the headers chain is always entered, e.g.:
```
$ ./src/bitcoin-cli loadtxoutset ~/.vimrc
<wait>
bitcoind log:
...
2023-10-15T14:55:45Z [snapshot] waiting to see blockheader 626174207465730a7265626d756e207465730a656c62616e65207861746e7973 in headers chain before snapshot activation
...
```
There is no point in doing any further action though if we already know from the start that the UTXO snapshot loading won't be successful. This PR adds an assumeutxo parameter check immediately after the metadata is read in, so we can fail immediately on a mismatch:
```
$ ./src/bitcoin-cli loadtxoutset ~/.vimrc
error code: -32603
error message:
Unable to load UTXO snapshot, assumeutxo block hash in snapshot metadata not recognized (626174207465730a7265626d756e207465730a656c62616e
65207861746e7973)
```
This way, users who mistakenly try to load files that are not snapshots don't have to wait 10 minutes (=the block header waiting timeout) anymore to get a negative response. If a file is loaded which is a valid snapshot (referencing to an existing block hash), but one which doesn't match the parameters, the feedback is also faster, as we don't have to wait anymore to see the hash in the headers chain before getting an error.
This is also partially fixes#28621.
ACKs for top commit:
maflcko:
lgtm ACK 9620cb4493
ryanofsky:
Code review ACK 9620cb4493. This should fix an annoyance and bad UX.
pablomartin4btc:
tACK 9620cb4493
Tree-SHA512: f88b865e9d46254858e57c024463f389cd9d8760a7cb30c190aa1723a931e159987dfc2263a733825d700fa612e7416691e4d8aab64058f1aeb0a7fa9233ac9c
Two recently added tests (PR #28625 / commit 2e31250027
and PR #28634 / commit 3bb51c29df)
introduced a bug by wrongly using the `assert_debug_log` helper.
Instead of passing the expected debug string in a list as expected, it
was passed as bare string, which is then interpretered as a list of
characters, very likely leading the debug log assertion pass even if the
intended message is not appearing.
In order to avoid bugs like this in the future, enforce that the
`{un}expected_msgs` parameters are lists.
4077e43bf6 test: fix usdt undeclared function errors on mantis (willcl-ark)
Pull request description:
This is one way to fix#28600
Recently usage of undeclared functions became an error rather than a warning, in C2x. https://reviews.llvm.org/D122983?id=420290
This change has migrated into the build tools of Ubuntu 23.10 which now causes the USDT tests to fail to compile, see
https://github.com/bitcoin/bitcoin/issues/28600
I think there are various potential fixes:
1. Manually declare the functions we use
2. Fix imports so that manual declarations aren't needed
3. Revert the new C2X behaviour and don't error on implicit function declarations
I would have preferred solution 2, but I believe this will require changes to the upstream bcc package. Having played with the imports I can get things working in a standalone C program, using system headers, but when building the program from a python context as we do in the test it uses its own headers (bundled with the python lib) rather than the system ones, and manually importing (some) system headers results in definition mismatches. I also investigated explicitly importing required headers from the package, which use paths like `#import </virtual/bcc/bcc_helpers.h>`, but this seems more obtuse and brittle than simply ignoring the warning.
Therefore I think that until the upstream python pacakge fixes their declarations, we should fix this by setting `-Wno-error=implicit-function-declaration` for the tracing programs.
cc maflcko 0xB10C
ACKs for top commit:
maflcko:
lgtm ACK 4077e43bf6
Tree-SHA512: 8368bb1155e920a95db128dc893267f8dab64f1ae53f6d63c6d9294e2e4e92bef8515e3697e9113228bedc51c0afdbc5bbcf558c119bf0eb3293dc2ced86b435
850670e3d6 test: don't run old binaries under valgrind (Sjors Provoost)
Pull request description:
Some, but not all, backward compatibility tests fail for me and it seems useless to run old release binaries under valgrind anyway.
Can be tested by running `test/functional/feature_txindex_compatibility.py --valgrind --timeout-factor=10` with and without this PR.
—
The previous version of this PR disabled these test entirely under valgrind. The current version does run the test, but starts the old binaries without valgrind.
ACKs for top commit:
maflcko:
lgtm ACK 850670e3d6
Tree-SHA512: ebdf461083f1292528e6619963b910f486b60b4f6b183f0aea2c8bfcafa98caeb204d138700cd288450643bcec5e49e12b89f2f7537fccdf495a2a33acd9cea0
