Commit graph

39368 commits

Author SHA1 Message Date
Anthony Towns
6e9e4e6130 Use ParamsWrapper for witness serialization 2023-11-14 08:45:30 +10:00
fanquake
5800c558eb
Merge bitcoin/bitcoin#28580: guix: update time-machine
92d12f1c89 guix: update time-machine to 77386bdbfe6b0c649c05ab37f08051d1ab3e5074 (fanquake)

Pull request description:

  python-altgraph (0.17.4) has been upstreamed, see: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=0c6198319a61d85cd8925af418466dcdccf3daff, so we can use it, and drop our package definition.

  Also includes:
  * GCC 10.4.0 -> 10.5.0: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=2fbb5398a39bf18e41235891a0740fa0bc4d7a4d.
  * Linux Kernel Headers 6.1 -> 6.1.61
  * LLVM 16 & LLVM 17 become available.

ACKs for top commit:
  hebasto:
    ACK 92d12f1c89.
  laanwj:
    LGTM ACK 92d12f1c89

Tree-SHA512: e362890ebf44d0fa6b276e023f431ce02c7a451dc8472d0ad729f72a76a8001c8c02cec322bd17680e039a1f55e654eccc4466e24a6eeccd50f0076328b3cedd
2023-11-13 16:41:21 +00:00
Andrew Chow
d232e36abd
Merge bitcoin/bitcoin#28207: mempool: Persist with XOR
fa6b053b5c mempool: persist with XOR (MarcoFalke)

Pull request description:

  Currently the `mempool.dat` file stores data received from remote peers as-is. This may be problematic when a program other than Bitcoin Core tries to interpret them by accident. For example, an anti-virus program or other program may scan the file and move it into quarantine, or delete it, or corrupt it.

  While the local wallet is expected to re-submit any pending transactions, unrelated transactions may be missing from the mempool after a restart. This may cause fee estimates to be off, or may cause block relay to be slower.

  Fix this, similar to https://github.com/bitcoin/bitcoin/pull/6650, by rolling a random XOR pattern over the dat file when writing or reading it.

  Obviously this can only protect against programs that accidentally and unintentionally are trying to mess with the dat file. Any program that intentionally wants to mess with the dat file can still trivially do so.

ACKs for top commit:
  achow101:
    re-ACK fa6b053b5c
  glozow:
    reACK fa6b053b5c
  ismaelsadeeq:
    ACK fa6b053b5c

Tree-SHA512: ded2ce3d81bc944b828263534e3178a1e45a914fe8e024f4a14c6561a73e301820944ecc75dd704b3d4221a7a3a5c0597ccab79546250c1197609ee981fe324e
2023-11-13 11:28:15 -05:00
fanquake
6342348072
Merge bitcoin/bitcoin#28076: util: Replace std::filesystem with util/fs.h
bbbbdb0cd5 ci: Add filesystem lint check (MarcoFalke)
fada2f9110 refactor: Replace <filesystem> with <util/fs.h> (MarcoFalke)

Pull request description:

  Using `std::filesystem` is problematic:

  * There is a `fs` namespace wrapper for it. So having two ways to achieve the same is confusing.
  * Not using the `fs` wrapper is dangerous and buggy, because it disables known bugs by deleting problematic functions.

  Fix all issues by removing use of it and adding a linter to avoid using it again in the future.

ACKs for top commit:
  TheCharlatan:
    ACK  bbbbdb0cd5
  fanquake:
    ACK bbbbdb0cd5 🦀

Tree-SHA512: 0e2d49742b08eb2635e6fce41485277cb9c40fe20b81017c391d3472a43787db1278a236825714ca1e41c9d2f59913865cfb0c649e3c8ab1fb598c849f80c660
2023-11-13 14:10:54 +00:00
fanquake
29c2c90362
Merge bitcoin/bitcoin#28721: multiprocess compatibility updates
3b70f7b615 doc: fix broken doc/design/multiprocess.md links after #24352 (Ryan Ofsky)
6d43aad742 span: Make Span template deduction guides work in SFINAE context (Ryan Ofsky)
8062c3bdb9 util: Add ArgsManager SetConfigFilePath method (Ryan Ofsky)
441d00c60f interfaces: Rename CalculateBumpFees methods to be compatible with capn'proto (Ryan Ofsky)
156f49d682 interfaces: Change getUnspentOutput return type to avoid multiprocess segfault (Ryan Ofsky)
4978754c00 interfaces: Add schedulerMockForward method so mockscheduler RPC can work across processes (Ryan Ofsky)
924327eaf3 interfaces: Fix const virtual method that breaks multiprocess support (Ryan Ofsky)
82a379eca8 streams: Add SpanReader ignore method (Russell Yanofsky)

Pull request description:

  This is a collection of small changes to interfaces and code which were needed as part of multiprocess PR #10102, but have been moved here to make that PR smaller.

  All of these changes are refactoring changes which do not affect behavior of current code

  ---

  This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/issues/28722).

ACKs for top commit:
  achow101:
    ACK 3b70f7b615
  naumenkogs:
    ACK 3b70f7b615
  maflcko:
    re-ACK 3b70f7b615  🎆

Tree-SHA512: 2368772b887056ad8a9f84c299cfde76ba45943770e3b5353130580900afa9611302195b899ced7b6e303b11f053ff204cae7c28ff4e12c55562fcc81119ba4c
2023-11-13 12:32:55 +00:00
fanquake
e862bceb17
Merge bitcoin/bitcoin#27935: fuzz: call lookup functions before calling Ban
fca0a8938e ci: remove "--exclude banman" for fuzzing in mac (brunoerg)
f9b286353f fuzz: call lookup functions before calling `Ban` (brunoerg)

Pull request description:

  Fixes #27924

  To not have any discrepancy, it's required to call lookup functions before calling `Ban`. If we don't do it, the assertion `assert(banmap == banmap_read);` may fail because `BanMapFromJson` will call `LookupSubNet` and cause the discrepancy between the banned and the loaded one. It happens especially in MacOS (#27924).

  Also, calling lookup functions before banning is what RPC `setban` does.

ACKs for top commit:
  maflcko:
    lgtm ACK fca0a8938e
  dergoegge:
    ACK fca0a8938e

Tree-SHA512: a3d635088a556df4507e65542157f10b41d4f87dce42927b58c3b812f262f4544b6b57f3384eef1097ffdd7c32b8dd1556aae201254960cbfbf48d45551200f7
2023-11-13 10:57:01 +00:00
fanquake
dd5f5713bc
Merge bitcoin/bitcoin#28391: refactor: Simplify CTxMempool/BlockAssembler fields, remove some external mapTx access
4dd94ca18f [refactor] remove access to mapTx in validation_block_tests (TheCharlatan)
d0cd2e804e [refactor] rewrite BlockAssembler inBlock and failedTx as sets of txids (glozow)
55b0939cab scripted-diff: rename vTxHashes to txns_randomized (TheCharlatan)
a03aef9cec [refactor] rewrite vTxHashes as a vector of CTransactionRef (glozow)
938643c3b2 [refactor] remove access to mapTx in validation.cpp (glozow)
333367a940 [txmempool] make CTxMemPoolEntry::lockPoints mutable (glozow)
1bf4855016 [refactor] use CheckPackageLimits for checkChainLimits (glozow)
dbc5bdbf59 [refactor] remove access to mapTx.find in mempool_tests.cpp (glozow)
f80909e7a3 [refactor] remove access to mapTx in blockencodings_tests.cpp (glozow)
8892d6b744 [refactor] remove access to mapTx from rpc/mempool.cpp (glozow)
fad61aa561 [refactor] get wtxid from entry instead of vTxHashes (glozow)
9cd8cafb77 [refactor] use exists() instead of mapTx.find() (glozow)
14804699e5 [refactor] remove access to mapTx from policy/rbf.cpp (glozow)
1c6a73abbd [refactor] Add helper for retrieving mempool entry (TheCharlatan)
453b4813eb [refactor] Add helper for iterating through mempool entries (stickies-v)

Pull request description:

  Motivation
  * It seems preferable to use stdlib data structures instead of boost if they can achieve close to the same thing.
  * Code external to mempool should ideally use its public helper methods instead of accessing `mapTx` or its iterators directly.
  * Reduce the number of complex boost multi index type interactions
  * Also see #28335 for further context/motivation. This PR together with #28385 simplifies that one.

  Overview of things done in this PR:
  * Make `vTxHashes` a vector of transaction references instead of a pair of transaction hash and iterator. The trade off here is that the data is retrieved on the fly with `GetEntry` instead of being cached in `vTxHashes`.
  * Introduce `GetEntry` helper method to replace the more involved `GetIter` where applicable
  * Replace `mapTx` access with `CTxMemPool` helper methods
  * Simplify `checkChainLimits` call in `node/interfaces.cpp`
  * Make `CTxMemPoolEntry`s `lockPoints`mutable such that they can be changed with a const iterator directly instead of going through `mapTx`
  * Make `BlockAssembler`'s `inBlock` and `failedTx` sets of transaction hashes.

ACKs for top commit:
  glozow:
    reACK 4dd94ca
  maflcko:
    re-ACK 4dd94ca18f 👝
  stickies-v:
    re-ACK 4dd94ca18f

Tree-SHA512: c4d043f2186e4fde337591883fac66cade3058173987b49502bd65cecf69207a3df1077f6626809652ab63230013167b7f39a2b39f1c5166959e5495df57065f
2023-11-13 10:51:41 +00:00
fanquake
e11b7587a1
Merge bitcoin/bitcoin#28831: test: Avoid intermittent failures in feature_init
44445ae8f1 test: Avoid intermittent failures in feature_init (MarcoFalke)

Pull request description:

  The code not only modifies block dat files, but also leveldb files, which may be of smaller size. Such corruption may not force leveldb to abort, according to the intermittent test failures.

  Fix the intermittent test failures by reverting 5ab6419f38 .

ACKs for top commit:
  kevkevinpal:
    lgtm ACK [44445ae](44445ae8f1)
  fjahr:
    ACK 44445ae8f1
  theStack:
    ACK 44445ae8f1

Tree-SHA512: 8084e4aeb8a976c1706a1898d7854c55d0c4ec7b5a08f65f97ffc173c935f9b0e0c1caef7be1538a458e4c018f7bd1948173349ec76ca48bc4013a63f284bb0f
2023-11-13 10:12:34 +00:00
fanquake
9c4b74fa92
Merge bitcoin/bitcoin#28777: doc: update docs for CHECK_ATOMIC macro
ebc7063c80 doc: update docs for CHECK_ATOMIC macro (fanquake)

Pull request description:

  Clarify that supported versions of GCC are not affected, and that Clang
  prior to version 15 still requires the explicit `-latomic` linking, when
  compiling for 32-bit.

ACKs for top commit:
  hebasto:
    ACK ebc7063c80.

Tree-SHA512: 6044dc28547431cfde7e89b663b5f9a86a4cb801212a21c3dbb18a1c41a53640480c3e4e944050dc3ec4cded9bc4c1f8eae8dbb60596289fef49bb13a8b53b76
2023-11-13 10:00:43 +00:00
fanquake
92d12f1c89
guix: update time-machine to 77386bdbfe6b0c649c05ab37f08051d1ab3e5074
python-altgraph (0.17.4) has been upstreamed. See:
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=0c6198319a61d85cd8925af418466dcdccf3daff

Also includes:
GCC 10.4.0 -> 10.5.0:
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=2fbb5398a39bf18e41235891a0740fa0bc4d7a4d.
Linux Kernel Headers 6.1.46 -> 6.1.61
LLVM 16 & 17 become available.
2023-11-13 09:44:56 +00:00
fanquake
95a3934cf3
Merge bitcoin/bitcoin#28786: guix: switch to 6.1 kernel headers over 5.15
380e365563 guix: switch to 6.1 kernel headers over 5.15 (fanquake)

Pull request description:

  6.1 is the current longterm release: https://kernel.org/.

  Note that using an older version of the kernel headers inside Guix, is not a "hack" for compatibility, and is explicitly recommended against by glibc:

  https://sourceware.org/glibc/wiki/FAQ#What_version_of_the_Linux_kernel_headers_should_be_used.3F.

  so using the latest version of the longterm headers seems appropriate.

  The last time we changed this was when we consolidated all builds to 5.15, in #25006.

ACKs for top commit:
  TheCharlatan:
    ACK 380e365563

Tree-SHA512: 78eb601e10261d99afd030dd7d039d962c106c48a57f16deb1c65b68fee4831e1070e4c35201f567fd24bbdab30a2b00804ddd118e1fee1dc8cdac7a3fb32ac5
2023-11-13 09:41:20 +00:00
fanquake
8243762700
Merge bitcoin/bitcoin#28849: test: fix node index bug when comparing peerinfo
22e38080ea test: fix node index bug when comparing peerinfo (Kashif Smith)

Pull request description:

  fix node index bug when comparing peerinfo in test/functional/p2p_v2_transport.py

ACKs for top commit:
  theStack:
    ACK 22e38080ea
  mzumsande:
    ACK 22e38080ea, good find!

Tree-SHA512: 9ee336eea999c61fb9f8704cc6361cf289fd3a361ab636c97695121ca3bcb8b38fbbfb55484311c17faa76d02065d91d190c489e1f3defd628216bf80a93f1fe
2023-11-11 15:57:11 +00:00
Kashif Smith
22e38080ea test: fix node index bug when comparing peerinfo 2023-11-10 16:25:18 -05:00
TheCharlatan
4dd94ca18f
[refactor] remove access to mapTx in validation_block_tests
Use the helper function instead of reaching into the mapTx member
object.
2023-11-10 16:44:47 +01:00
glozow
d0cd2e804e
[refactor] rewrite BlockAssembler inBlock and failedTx as sets of txids 2023-11-10 16:44:45 +01:00
TheCharlatan
55b0939cab
scripted-diff: rename vTxHashes to txns_randomized
-BEGIN VERIFY SCRIPT-
git grep -l "vTxHashesIdx" src | xargs sed -i "s/vTxHashesIdx/idx_randomized/g"
git grep -l "vTxHashes" src | xargs sed -i "s/vTxHashes/txns_randomized/g"
-END VERIFY SCRIPT-
2023-11-10 16:44:44 +01:00
glozow
a03aef9cec
[refactor] rewrite vTxHashes as a vector of CTransactionRef
vTxHashes exposes a complex mapTx iterator type that its external users
don't need. Directly populate it with CTransactionRef instead.
2023-11-10 16:44:42 +01:00
glozow
938643c3b2
[refactor] remove access to mapTx in validation.cpp 2023-11-10 16:44:40 +01:00
glozow
333367a940
[txmempool] make CTxMemPoolEntry::lockPoints mutable
Allows calling UpdateLockPoints() with a (const) txiter. Note that this
was already possible for caller using mapTx.modify(txiter). The point
here is to not be accessing mapTx when doing so.
2023-11-10 16:44:39 +01:00
glozow
1bf4855016
[refactor] use CheckPackageLimits for checkChainLimits
The behavior is the same as CalculateMemPoolAncestors. The only
difference is the string returned, and the string is discarded anyway
since checkChainLimits only cares about pass/fail.
2023-11-10 16:44:37 +01:00
glozow
dbc5bdbf59
[refactor] remove access to mapTx.find in mempool_tests.cpp 2023-11-10 16:44:35 +01:00
glozow
f80909e7a3
[refactor] remove access to mapTx in blockencodings_tests.cpp 2023-11-10 16:44:33 +01:00
glozow
8892d6b744
[refactor] remove access to mapTx from rpc/mempool.cpp 2023-11-10 16:44:32 +01:00
glozow
fad61aa561
[refactor] get wtxid from entry instead of vTxHashes 2023-11-10 16:44:30 +01:00
glozow
9cd8cafb77
[refactor] use exists() instead of mapTx.find() 2023-11-10 16:44:29 +01:00
glozow
14804699e5
[refactor] remove access to mapTx from policy/rbf.cpp 2023-11-10 16:44:27 +01:00
TheCharlatan
1c6a73abbd
[refactor] Add helper for retrieving mempool entry
In places where the iterator is only needed for accessing the actual
entry, it should not be required to first retrieve the iterator.
2023-11-10 16:44:25 +01:00
stickies-v
453b4813eb
[refactor] Add helper for iterating through mempool entries
Instead of reaching into the mapTx data structure, use a helper method
that provides the required vector of CTxMemPoolEntry pointers.
2023-11-10 16:44:20 +01:00
fanquake
1fdd832842
Merge bitcoin/bitcoin#28835: test: Check error details with assert_debug_log on the assumeutxo invalid hash dump - follow-up #28698
7de7685372 test, assumeutxo: Use assert_debug_log for error details (pablomartin4btc)

Pull request description:

  This is a follow-up on the invalid hash dump fix #28698, [suggested](https://github.com/bitcoin/bitcoin/pull/28698#pullrequestreview-1698178157) by theStack and agreed by Sjors and ryanofsky.

ACKs for top commit:
  Sjors:
    ACK 7de7685372
  maflcko:
    lgtm ACK 7de7685372

Tree-SHA512: 036b3cef3084e3ead8923e8dcabe4fa7ebe97fb514d223aa38bc38df10337e3fe3113e42322178b58fb03fcd4511af4b5b56bceecbb7ded5b9758842c70db3f2
2023-11-10 09:55:56 +00:00
pablomartin4btc
7de7685372 test, assumeutxo: Use assert_debug_log for error details
This is a follow-up on the invalid hash dump fix PR #28698.

https://github.com/bitcoin/bitcoin/pull/28698#pullrequestreview-1698178157
2023-11-09 18:54:27 -03:00
MarcoFalke
fa6b053b5c
mempool: persist with XOR 2023-11-09 19:44:50 +01:00
MarcoFalke
44445ae8f1
test: Avoid intermittent failures in feature_init 2023-11-09 14:30:02 +01:00
brunoerg
fca0a8938e ci: remove "--exclude banman" for fuzzing in mac 2023-11-09 10:11:59 -03:00
brunoerg
f9b286353f fuzz: call lookup functions before calling Ban
Also, compare banmaps only if there are no invalid
entries.
2023-11-09 10:11:51 -03:00
fanquake
b3898e946c
Merge bitcoin/bitcoin#28826: ci: Switch IWYU to clang_17 branch
9f208c0171 ci: Switch IWYU to `clang_17` branch (Hennadii Stepanov)

Pull request description:

  The IWYU version [0.21](https://github.com/include-what-you-use/include-what-you-use/releases/tag/0.21) has been tagged, and the `clang_17` branch is available now.

ACKs for top commit:
  maflcko:
    lgtm ACK 9f208c0171

Tree-SHA512: 8b8f8743d1c2719b6383b5a6a48356ac02a301d1ce9cee77f93cc04c12de22e9ac6b59e23550a589540e68292cfac0d85bacedc9ca26f6b589011d36ee1d38cf
2023-11-09 13:07:05 +00:00
Hennadii Stepanov
9f208c0171
ci: Switch IWYU to clang_17 branch 2023-11-09 12:04:17 +00:00
fanquake
88c3b100f0
Merge bitcoin/bitcoin#28829: ci: win64 task does use boost:process
5f0bf2ef69 ci: win64 task does use boost:process (fanquake)

Pull request description:

  It passes `--enable-external-signer`.

ACKs for top commit:
  maflcko:
    lgtm ACK 5f0bf2ef69

Tree-SHA512: 789877aac0d36429f31256adc07812d1914a8a059a43ef22416be97270b083902c253ff0561b3de28e76db005387f14b2712bfcfb1334f69b293c39ce0e7467c
2023-11-09 11:21:21 +00:00
fanquake
5f0bf2ef69
ci: win64 task does use boost:process
It passes `--enable-external-signer`.
2023-11-09 10:50:32 +00:00
fanquake
c2da8c583f
Merge bitcoin/bitcoin#28822: test: Add missing wait for version to be sent in add_outbound_p2p_connection
faa2ad88bc test: Add missing wait for version to be sent in add_outbound_p2p_connection (MarcoFalke)

Pull request description:

  Can be tested with:

  ```diff
  diff --git a/test/functional/test_framework/p2p.py b/test/functional/test_framework/p2p.py
  index b1ed97b794..eb4f72c6b6 100755
  --- a/test/functional/test_framework/p2p.py
  +++ b/test/functional/test_framework/p2p.py
  @@ -205,6 +205,7 @@ class P2PConnection(asyncio.Protocol):
           assert not self._transport
           logger.debug("Connected & Listening: %s:%d" % (self.dstaddr, self.dstport))
           self._transport = transport
  +        import time;time.sleep(.1);
           if self.on_connection_send_msg:
               self.send_message(self.on_connection_send_msg)
               self.on_connection_send_msg = None  # Never used again
  ```

  Found and reported by mzumsande in https://github.com/bitcoin/bitcoin/pull/28782#pullrequestreview-1718560252

ACKs for top commit:
  mzumsande:
    ACK faa2ad88bc

Tree-SHA512: 863f06125dec40cccaa852d0d8ca2e2b9c0b74610205e9fd6c9c279bdf36801ff475e3d873fd1b18172eb8220e17b2caff60069ce63512e569934a43f27d03fd
2023-11-09 10:26:56 +00:00
glozow
d60ebea597
Merge bitcoin/bitcoin#28808: refactor: Miniminer package linearization followups
b4b01d3fb4 [refactor] updating miniminer comments to be more accurate (kevkevin)
83933eff00 [refactor] Miniminer var cached_descendants to descendants (kevkevin)
43423fd834 [refactor] Change MiniMinerMempoolEntry order (kevkevin)

Pull request description:

  ### Motivation
  In https://github.com/bitcoin/bitcoin/pull/28762 there were some post merge comments which are being addressed in this PR with the following commits

  ### [8d4c46f](8d4c46f54d) Reorganizing `MiniMinerMempoolEntry` to match the order we have elsewhere
  * https://github.com/bitcoin/bitcoin/pull/28762#discussion_r1381775670

  ### [7505ec2](7505ec2054) Renaming `cached_descendants` to `descendants` for simpler variable naming
  * https://github.com/bitcoin/bitcoin/pull/28762#discussion_r1381819567

  ### [b21f2f2](b21f2f2f55) Code comment modifications to be more accurate to what is actually happening
  * https://github.com/bitcoin/bitcoin/pull/28762#discussion_r1381902909 and
  * https://github.com/bitcoin/bitcoin/pull/28762#discussion_r1382002278 and
  * https://github.com/bitcoin/bitcoin/pull/28762#discussion_r1383041819

ACKs for top commit:
  murchandamus:
    reACK b4b01d3fb4
  theStack:
    LGTM ACK b4b01d3fb4

Tree-SHA512: 54f044a578fb203d8a3c1aa0bcd1fc4bcdff0bc9b024351925a4caf0ccece7c7736b0694ad1168c3cbb447bdb58a91f4cac365f46114da29a889fbc8ea595b82
2023-11-09 09:30:58 +00:00
kevkevin
b4b01d3fb4
[refactor] updating miniminer comments to be more accurate 2023-11-08 14:45:18 -06:00
Andrew Chow
3d7544b481
Merge bitcoin/bitcoin#28823: ci: remove note re M1 usage
8cbb619691 ci: remove note re M1 usage (fanquake)

Pull request description:

  M1 is now available in GitHub CI, but we don't currently have a plan to use it, so remove the comment.

ACKs for top commit:
  maflcko:
    lgtm ACK 8cbb619691
  achow101:
    ACK 8cbb619691
  hebasto:
    ACK 8cbb619691.

Tree-SHA512: 13bbd4ad2358b0df6781031d6bdba456ffe706f30bf273a317ea8031f28276ef5821b5f767e4fb47d444b4e9ad7b8b7f67563927838552d275aca481d0e2fc2f
2023-11-08 13:20:51 -05:00
Andrew Chow
19d1ba1b41
Merge bitcoin/bitcoin#28787: init: completely remove -zapwallettxes (remaining hidden option)
5039c346ca init: completely remove `-zapwallettxes` (remaining hidden option) (Sebastian Falbesoner)

Pull request description:

  The `-zapwallettxes` functionality has been removed in v0.21.0 (see commit 3340dbadd3 / PR #19671), with the parameter being kept as hidden option, to inform users via an exit error that `abandontransaction` should be used instead.

  As any guides that still suggest to use `-zapwallettxes` would refer to a Bitcoin Core version that is EOL since many years (i.e. <= v0.20.x), it is highly unlikely that the error caused by the option is still relevant for any user, hence it seems fine to remove it now.

ACKs for top commit:
  achow101:
    ACK 5039c346ca
  BrandonOdiwuor:
    ACK 5039c346ca
  fanquake:
    ACK 5039c346ca

Tree-SHA512: e3ccc6918e0f8fa68dbd1a7ec4999cc2a44e28038711919fcddaf0727648c73a9ba0fb77674317147592a113fad20755d4e727f48176bc17b048fbdebad2d6c9
2023-11-08 10:53:42 -05:00
fanquake
8cbb619691
ci: remove note re M1 usage
M1 is now available in GitHub CI, but we don't currently have a plan to
use it, so remove the comment.
2023-11-08 15:07:17 +00:00
fanquake
f1f3f2d9cc
Merge bitcoin/bitcoin#28815: fuzz: Avoid timeout and bloat in fuzz targets
fabb5046a7 fuzz: Avoid timeout and bloat in fuzz targets (MarcoFalke)

Pull request description:

  If the fuzz input contains invalid data *in a loop*, abort early. This will teach the fuzz engine to look for useful data and avoids bloating the fuzz input folder with useless (repeated) data.

ACKs for top commit:
  dergoegge:
    utACK fabb5046a7
  brunoerg:
    crACK fabb5046a7

Tree-SHA512: 26da100d7558ae6fdd5292fb146d8858b2af8f78c546ca2509b9d27b33a33e9462ecb6035de142f9f36dd5de32f8cbad099d6c7a697902d23e1bb621cd27dc88
2023-11-08 14:19:35 +00:00
glozow
9ad19fc7c7
Merge bitcoin/bitcoin#28155: net: improves addnode / m_added_nodes logic
0420f99f42 Create net_peer_connection unit tests (Jon Atack)
4b834f6499 Allow unit tests to access additional CConnman members (Jon Atack)
34b9ef443b net/rpc: Makes CConnman::GetAddedNodeInfo able to return only non-connected address on request (Sergi Delgado Segura)
94e8882d82 rpc: Prevents adding the same ip more than once when formatted differently (Sergi Delgado Segura)
2574b7e177 net/rpc: Check all resolved addresses in ConnectNode rather than just one (Sergi Delgado Segura)

Pull request description:

  ## Rationale

  Currently, `addnode` has a couple of corner cases that allow it to either connect to the same peer more than once, hence wasting outbound connection slots, or add redundant information to `m_added_nodes`, hence making Bitcoin iterate through useless data on a regular basis.

  ### Connecting to the same node more than once

  In general, connecting to the same node more than once is something we should try to prevent. Currently, this is possible via `addnode` in two different ways:

  1. Calling `addnode` more than once in a short time period, using two equivalent but distinct addresses
  2. Calling `addnode add` using an IP, and `addnode onetry` after with an address that resolved to the same IP

  For the former, the issue boils down to `CConnman::ThreadOpenAddedConnections` calling `CConnman::GetAddedNodeInfo` once, and iterating over the result to open connections (`CConman::OpenNetworkConnection`) on the same loop for all addresses.`CConnman::ConnectNode` only checks a single address, at random, when resolving from a hostname, and uses it to check whether we are already connected to it.

  An example to test this would be calling:

  ```
  bitcoin-cli addnode "127.0.0.1:port" add
  bitcoin-cli addnode "localhost:port" add
  ```

  And check how it allows us to perform both connections some times, and some times it fails.

  The latter boils down to the same issue, but takes advantage of `onetry` bypassing the `CConnman::ThreadOpenAddedConnections` logic and calling `CConnman::OpenNetworkConnection` straightaway. A way to test this would be:

  ```
  bitcoin-cli addnode "127.0.0.1:port" add
  bitcoin-cli addnode "localhost:port" onetry
  ```

  ### Adding the same peer with two different, yet equivalent, addresses

  The current implementation of `addnode` is pretty naive when checking what data is added to `m_added_nodes`. Given the collection stores strings, the checks at `CConnman::AddNode()` basically check wether the exact provided string is already in the collection. If so, the data is rejected, otherwise, it is accepted. However, ips can be formatted in several ways that would bypass those checks.

  Two examples would be `127.0.0.1` being equal to `127.1` and `[::1]` being equal to `[0:0:0:0:0:0:0:1]`. Adding any pair of these will be allowed by the rpc command, and both will be reported as connected by `getaddednodeinfo`, given they map to the same `CService`.

  This is less severe than the previous issue, since even tough both nodes are reported as connected by `getaddednodeinfo`, there is only a single connection to them (as properly reported by `getpeerinfo`). However, this adds redundant data to `m_added_nodes`, which is undesirable.

  ### Parametrize `CConnman::GetAddedNodeInfo`
  Finally, this PR also parametrizes `CConnman::GetAddedNodeInfo` so it returns either all added nodes info, or only info about the nodes we are **not** connected to. This method is used both for `rpc`, in `getaddednodeinfo`, in which we are reporting all data to the user, so the former applies, and to check what nodes we are not connected to, in `CConnman::ThreadOpenAddedConnections`, in which we are currently returning more data than needed and then actively filtering using `CService.fConnected()`

ACKs for top commit:
  jonatack:
    re-ACK 0420f99f42
  kashifs:
    > > tACK [0420f9](0420f99f42)
  sr-gi:
    > > > tACK [0420f9](0420f99f42)
  mzumsande:
    Tested ACK 0420f99f42

Tree-SHA512: a3a10e748c12d98d439dfb193c75bc8d9486717cda5f41560f5c0ace1baef523d001d5e7eabac9fa466a9159a30bb925cc1327c2d6c4efb89dcaf54e176d1752
2023-11-08 11:31:36 +00:00
MarcoFalke
faa2ad88bc
test: Add missing wait for version to be sent in add_outbound_p2p_connection 2023-11-08 11:36:24 +01:00
fanquake
d690f89b57
Merge bitcoin/bitcoin#28785: validation: return more helpful results for reconsiderable fee failures and skipped transactions
1147e00e59 [validation] change package-fee-too-low, return wtxid(s) and effective feerate (glozow)
10dd9f2441 [test] use CheckPackageMempoolAcceptResult in previous tests (glozow)
3979f1afcb [validation] add TxValidationResult::TX_RECONSIDERABLE, TX_UNKNOWN (glozow)
5c786a026a [refactor] use Wtxid for m_wtxids_fee_calculations (glozow)

Pull request description:

  Split off from #26711 (suggested in https://github.com/bitcoin/bitcoin/pull/26711#issuecomment-1786392253). This is part of #27463.

  - Add 2 new TxValidationResults
    - `TX_RECONSIDERABLE` helps us encode transactions who have failed fee checks that can be bypassed using package validation. This is distinguished from `TX_MEMPOOL_POLICY` so that we re-validate a transaction if and only if it is eligible for package CPFP. In the future, we will have a separate cache for reconsiderable rejects so these transactions don't go in `m_recent_rejects`.
    - `TX_UNKNOWN` helps us communicate that we aborted package validation and didn't finish looking at this transaction: it's not valid but it's also not invalid (i.e. don't cache it as a rejected tx)
  - Return effective feerate and the wtxids of transactions used to calculate that effective feerate when the error is `TX_SINGLE_FAILURE`. Previously, we would only provide this information if the transaction passed. Now that we have package validation, it's much more helpful to the caller to know how the failing feerate was calculated. This can also be used to improve our submitpackage RPC result (which is currently a bit unhelpful when things fail).
  - Use the newly added `CheckPackageMempoolAcceptResult` for existing package validation tests. This increases test coverage and helps test the changes made in this PR.

ACKs for top commit:
  instagibbs:
    reACK 1147e00e59
  achow101:
    ACK 1147e00e59
  murchandamus:
    reACK 1147e00e59
  ismaelsadeeq:
    ACK 1147e00e59

Tree-SHA512: ac1cd73c2b487a1b99d329875d39d8107c91345a5b0b241d54a6a4de67faf11be69a2721cc732c503024a9cca381dac33d61e187957279e3c82653bea118ba91
2023-11-08 10:17:05 +00:00
fanquake
059f131314
Merge bitcoin/bitcoin#28820: tests: Increase wallet_miniscript.py rpc timeout to 90 seconds
6559e4d27a tests: Increase wallet_miniscript.py rpc timeout to 90 seconds (Andrew Chow)

Pull request description:

  The signing test for the large miniscript can sometimes take longer than the 30 second timeout, depending on the load on my system. Increasing it to 90 seconds seems to be good enough.

ACKs for top commit:
  kevkevinpal:
    but increasing seems fine ACK [6559e4d](6559e4d27a)
  maflcko:
    lgtm ACK 6559e4d27a

Tree-SHA512: 1b7bf94c77f85a0deddb1384aacbeb934205d0a630fecc8e75a4a98d1946d77d9bca36692fb6c1ab8e9276392f617281aafc4c685c248a8d3b0c77f896cda624
2023-11-08 09:56:49 +00:00
fanquake
1162d046ec
Merge bitcoin/bitcoin#28782: test: Add missing sync on send_version in peer_connect
fa02598469 test: Add missing sync on send_version in peer_connect (MarcoFalke)

Pull request description:

  Without the sync, the logic will be racy. For example, `p2p_sendtxrcncl.py` is failing locally (and on CI occasionally), because non-version messages will be sent before the version message:

  ```py
          self.log.info('SENDTXRCNCL with version=0 triggers a disconnect')
          sendtxrcncl_low_version = create_sendtxrcncl_msg()
          sendtxrcncl_low_version.version = 0
          peer = self.nodes[0].add_p2p_connection(PeerNoVerack(), send_version=True, wait_for_verack=False)
          with self.nodes[0].assert_debug_log(["txreconciliation protocol violation"]):
              peer.send_message(sendtxrcncl_low_version)
              peer.wait_for_disconnect()
  ```

  ```
   test  2023-11-02T08:15:19.620000Z TestFramework (INFO): SENDTXRCNCL with version=0 triggers a disconnect
   test  2023-11-02T08:15:19.621000Z TestFramework.p2p (DEBUG): Connecting to Bitcoin Node: 127.0.0.1:11312
   test  2023-11-02T08:15:19.624000Z TestFramework.p2p (DEBUG): Connected & Listening: 127.0.0.1:11312
   test  2023-11-02T08:15:19.798000Z TestFramework.p2p (DEBUG): Send message to 127.0.0.1:11312: msg_sendtxrcncl(version=0, salt=2)
   test  2023-11-02T08:15:19.799000Z TestFramework.p2p (DEBUG): Send message to 127.0.0.1:11312: msg_version(nVersion=70016 nServices=9 nTime=Thu Nov  2 08:15:19 2023 addrTo=CAddress(nServices=1 net=IPv4 addr=127.0.0.1 port=11312) addrFrom=CAddress(nServices=1 net=IPv4 addr=0.0.0.0 port=0) nNonce=0x369AC031CDA96022 strSubVer=/python-p2p-tester:0.0.3/ nStartingHeight=-1 relay=1)
   node0 2023-11-02T08:15:19.804409Z [net] [net.cpp:3676] [CNode] [net] Added connection peer=0
   node0 2023-11-02T08:15:19.805256Z [net] [net.cpp:1825] [CreateNodeFromAcceptedSocket] [net] connection from 127.0.0.1:55964 accepted
   node0 2023-11-02T08:15:19.809861Z [msghand] [net_processing.cpp:3356] [ProcessMessage] [net] received: sendtxrcncl (12 bytes) peer=0
   node0 2023-11-02T08:15:19.810297Z [msghand] [net_processing.cpp:3582] [ProcessMessage] [net] non-version message before version handshake. Message "sendtxrcncl" from peer=0
   node0 2023-11-02T08:15:19.810928Z [msghand] [net_processing.cpp:3356] [ProcessMessage] [net] received: version (111 bytes) peer=0
  ...
   test  2023-11-02T09:35:20.166000Z TestFramework.utils (ERROR): wait_until() failed. Predicate: ''''
                                             def test_function():
                                                 if check_connected:
                                                     assert self.is_connected
                                                 return test_function_in()
                                     '''
   test  2023-11-02T09:35:20.187000Z TestFramework (ERROR): Assertion failed
                                     Traceback (most recent call last):
                                       File "/ci_container_base/ci/scratch/build/bitcoin-s390x-linux-gnu/test/functional/test_framework/test_framework.py", line 132, in main
                                         self.run_test()
                                       File "/ci_container_base/ci/scratch/build/bitcoin-s390x-linux-gnu/test/functional/p2p_sendtxrcncl.py", line 188, in run_test
                                         peer.wait_for_disconnect()
                                       File "/ci_container_base/ci/scratch/build/bitcoin-s390x-linux-gnu/test/functional/test_framework/p2p.py", line 478, in wait_for_disconnect
                                         self.wait_until(test_function, timeout=timeout, check_connected=False)
                                       File "/ci_container_base/ci/scratch/build/bitcoin-s390x-linux-gnu/test/functional/test_framework/p2p.py", line 470, in wait_until
                                         wait_until_helper_internal(test_function, timeout=timeout, lock=p2p_lock, timeout_factor=self.timeout_factor)
                                       File "/ci_container_base/ci/scratch/build/bitcoin-s390x-linux-gnu/test/functional/test_framework/util.py", line 275, in wait_until_helper_internal
                                         raise AssertionError("Predicate {} not true after {} seconds".format(predicate_source, timeout))
                                     AssertionError: Predicate ''''
                                             def test_function():
                                                 if check_connected:
                                                     assert self.is_connected
                                                 return test_function_in()
                                     ''' not true after 4800.0 seconds

ACKs for top commit:
  mzumsande:
    ACK fa02598469

Tree-SHA512: 78871f603d387e2df8c0acbdfa95441fa186f80e94593021bb219bbf1bc9dc7efc4e266bd254b5cc41114c38227ff3b7f6172335d9bb828427f0a2acffde752d
2023-11-08 09:55:53 +00:00