The current behavior of seednode fetching is pretty eager: we do it as the first
step under `ThreadOpenNetworkConnections` even if some peers may be queryable
from our addrman. This poses two potential issues:
- First, if permanently set (e.g. running with seednode in a config file) we'd
be signaling such seed every time we restart our node
- Second, we will be giving the seed node way too much influence over our addrman,
populating the latter even with data from the former even when unnecessary
This changes the behavior to only add seednodes to `m_addr_fetch` if our addrman
is empty, or little by little after we've spent some time trying addresses from
our addrman. Also, seednodes are added to `m_addr_fetch` in random order, to avoid
signaling the same node in case more than one seed is added and we happen to try
them over multiple restarts
1bc9f64bee contrib: assume binary existence in sec/sym checks (fanquake)
51d8f435c9 contrib: simplify ELF test-security-check (fanquake)
1810e20677 contrib: simplify PE test-security-check (fanquake)
6c9746ff92 contrib: simplify MACHO test-security-check (fanquake)
Pull request description:
The current `test-security-check` script is hard to understand, and change (i.e https://github.com/bitcoin/bitcoin/pull/29987/files#diff-52aa0cda44721f089e53b128cb1232a876006ef257b211655456b17dfb2ec712); tests are also not done in isolation (when-possible). Fix that, and add missing checks. Simplifies future toolchain/security/hardening changes.
ACKs for top commit:
hebasto:
ACK 1bc9f64bee (assuming my Guix hashes match; I'll provide them shortly).
TheCharlatan:
ACK 1bc9f64bee
Tree-SHA512: 1885d0ce63a94ffa61345327f919da20b63de6dd4148d6db3ee8bad4485253a36e8ab0dbee48cecc02ea35d139edfed75453af45fc364bcbef6fe16b6823bc7a
c85accecaf [refactor] delete EraseTxNoLock, just use EraseTx (glozow)
6ff84069a5 remove obsoleted TxOrphanage::m_mutex (glozow)
61745c7451 lock m_recent_confirmed_transactions using m_tx_download_mutex (glozow)
723ea0f9a5 remove obsoleted hashRecentRejectsChainTip (glozow)
18a4355250 update recent_rejects filters on ActiveTipChange (glozow)
36f170d879 add ValidationInterface::ActiveTipChange (glozow)
3eb1307df0 guard TxRequest and rejection caches with new mutex (glozow)
Pull request description:
See #27463 for full project tracking.
This contains the first few commits of #30110, which require some thinking about thread safety in review.
- Introduce a new `m_tx_download_mutex` which guards the transaction download data structures including `m_txrequest`, the rolling bloom filters, and `m_orphanage`. Later this should become the mutex guarding `TxDownloadManager`.
- `m_txrequest` doesn't need to be guarded using `cs_main` anymore
- `m_recent_confirmed_transactions` doesn't need its own lock anymore
- `m_orphanage` doesn't need its own lock anymore
- Adds a new `ValidationInterface` event, `ActiveTipChanged`, which is a synchronous callback whenever the tip of the active chainstate changes.
- Flush `m_recent_rejects` and `m_recent_rejects_reconsiderable` on `ActiveTipChanged` just once instead of checking the tip every time `AlreadyHaveTx` is called. This should speed up calls to that function (no longer comparing a block hash each time) and removes the need to lock `cs_main` every time it is called.
Motivation:
- These data structures need synchronization. While we are holding `m_tx_download_mutex`, these should hold:
- a tx hash in `m_txrequest` is not also in `m_orphanage`
- a tx hash in `m_txrequest` is not also in `m_recent_rejects` or `m_recent_confirmed_transactions`
- In the future, orphan resolution tracking should also be synchronized. If a tx has an entry in the orphan resolution tracker, it is also in `m_orphanage`, and not in `m_txrequest`, etc.
- Currently, `cs_main` is used to e.g. sync accesses to `m_txrequest`. We should not broaden the scope of things it locks.
- Currently, we need to know the current chainstate every time we call `AlreadyHaveTx` so we can decide whether we should update it. Every call compares the current tip hash with `hashRecentRejectsChainTip`. It is more efficient to have a validation interface callback that updates the rejection filters whenever the chain tip changes.
ACKs for top commit:
instagibbs:
reACK c85accecaf
dergoegge:
Code review ACK c85accecaf
theStack:
Light code-review ACK c85accecaf
hebasto:
ACK c85accecaf, I have reviewed the code and it looks OK.
Tree-SHA512: c3bd524b5de1cafc9a10770dadb484cc479d6d4c687d80dd0f176d339fd95f73b85cb44cb3b6b464d38a52e20feda00aa2a1da5a73339e31831687e4bd0aa0c5
a517029646 depends: switch to building expat with CMake (fanquake)
Pull request description:
Switch to building Expat with CMake, instead of Autotools.
ACKs for top commit:
hebasto:
re-ACK a517029646.
Tree-SHA512: ca040545dd83fb81a8b209aa24cae6e22eaeff04f44bdabc4454adf6ea63d34f4ae27bd5980c65db2d2542e23eb2712102719023c262ab63a933c90b5999c11e
09ce3501fa fix: Make TxidFromString() respect string_view length (Hodlinator)
01e314ce0a refactor: Change base_blob::SetHex() to take std::string_view (Hodlinator)
2f5577dc2e test: uint256 - Garbage suffixes and zero padding (Hodlinator)
f11f816800 refactor: Make uint256_tests no longer use deprecated BOOST_CHECK() (Hodlinator)
f0eeee2dc1 test: Add test for TxidFromString() behavior (Ryan Ofsky)
Pull request description:
### Problem
Prior to this, `TxidFromString()` was passing `string_view::data()` into `uint256S()` which meant it would only receive the a naked `char*` pointer and potentially scan past the `string_view::length()` until it found a null terminator (or some other non-hex character).
Appears to have been a fully dormant bug as callers were either passing a string literal or `std::string` directly to `TxidFromFromString()`, meaning a null terminator always existed at `pointer[length()]`. Bug existed since original merge of `TxidFromString()`.
### Solution
Make `uint256S()` (and `base_blob::SetHex()`) take and operate on `std::string_view` instead of `const char*` and have `TxidFromString()` pass that in.
(PR was prompted by comment in https://github.com/bitcoin/bitcoin/pull/30377#issuecomment-2208857200 (referring to https://github.com/bitcoin/bitcoin/pull/28922#discussion_r1404437378)).
ACKs for top commit:
maflcko:
re-ACK 09ce3501fa🕓
paplorinc:
ACK 09ce3501fa
ryanofsky:
Code review ACK 09ce3501fa. I think the current code changes are about as small as you could make to fix the bug without introducing a string copy, and the surrounding test improvements are all very nice and welcome.
Tree-SHA512: c2c10551785fb6688d1e2492ba42a8eee4c19abbe8461bb0774d56a70c23cd6b0718d2641632890bee880c06202dee148126447dd2264eaed4f5fee7e1bcb581
29eafd5733 rpc: doc: use "output script" terminology consistently in "asm"/"hex" results (Sebastian Falbesoner)
Pull request description:
The wording "public key script" was likely chosen as a human-readable form of the technical term `scriptPubKey`, but it doesn't seem to be really widespread. Replace it by the more (probably most?) common term "output script" instead. Note that the argument for the `decodescript` RPC is not necessarily an output script (it could e.g. be also a redeem script), so in this case we just stay generic and use "script".
See also the draft BIP "Terminology for Transaction Components" (https://github.com/murchandamus/bips/blob/2022-04-tx-terminology/bip-tx-terminology.mediawiki) from murchandamus which suggests to use "output script" as well.
Affects the help text of the following RPCs:
- decodepsbt
- decoderawtransaction
- decodescript
- getblock (if verbosity=3)
- getrawtransaction (if verbosity=2,3)
- gettxout
ACKs for top commit:
maflcko:
ACK 29eafd5733
achow101:
ACK 29eafd5733
BrandonOdiwuor:
ACK 29eafd5733
tdb3:
ACK 29eafd5733
Tree-SHA512: 62eb92d42bc44e36dc3090df7b248a123868a74af253d2046de02086e688bf6ff98307b927ba2fee3d599f85e073aeb8eca90ed15105ca63b648b6796cfa340b
d63ef73800 test: Add loadtxoutset test with tip on snapshot block (Fabian Jahr)
c2f86d4bcb test: Remove already resolved assumeutxo todo comments (Fabian Jahr)
Pull request description:
The first commit removes three Todos that have been addressed previously (see commit message for details).
The second message resolves another todo by adding the missing test case. This is a special case of "the tip has more work than the snapshot" where the tip is the same block as the snapshot base block.
Related to #28648.
ACKs for top commit:
jrakibi:
ACK [d63ef73](d63ef73800)
achow101:
ACK d63ef73800
maflcko:
ACK d63ef73800
alfonsoromanz:
Re ACK d63ef73800
Tree-SHA512: 8d5a25fc0b26531db3a9740132694138f2103b7b42eeb1d4a64095bfc901c1372e23601c0855c7def84c8a4e185d10611e4e830c4e479f1b663ae6ed53abb130
Prior to this, passing string_view::data() into uint256S() meant the latter would only receive the a naked char* pointer and potentially scan past the string_view::length() until it found a null terminator (or some other non-hex character).
Appears to have been a fully dormant bug as callers were either passing a string literal or std::string directly to TxidFromFromString(), meaning null terminator always existed at pointer[length()]. Bug existed since original merge of TxidFromString(), discussed in https://github.com/bitcoin/bitcoin/pull/28922#discussion_r1404437378.
6a5e9e40e1 doc: use proper doxygen formatting for CTxMemPool::cs (Vasil Dimov)
Pull request description:
Having `@par title` followed by an empty line renders improperly in Doxygen - it results in a paragraph with a title but without a body.
https://www.doxygen.nl/manual/commands.html#cmdpar
This also results in a compiler warning (or error) with Clang 19:
```
./txmempool.h:368:34: error: empty paragraph passed to '@par' command [-Werror,-Wdocumentation]
368 | * @par Consistency guarantees
| ~~~~~~~~~~~~~~~~~~~~~~~~~~^
1 error generated.
```
ACKs for top commit:
maflcko:
review ACK 6a5e9e40e1
tdb3:
ACK 6a5e9e40e1
Tree-SHA512: 2c4c9e5fd4bd44754800a9bcfff74df101afc060b84451c45aa098e4ceb05a47f28a36f8473b31222552fad6339b752a148e6b1c7d41c2003f515b3eb4060902
Having `@par title` followed by an empty line renders improperly in
Doxygen - it results in a paragraph with a title but without a body.
https://www.doxygen.nl/manual/commands.html#cmdpar
This also results in a compiler warning (or error) with Clang 19:
```
./txmempool.h:368:34: error: empty paragraph passed to '@par' command [-Werror,-Wdocumentation]
368 | * @par Consistency guarantees
| ~~~~~~~~~~~~~~~~~~~~~~~~~~^
1 error generated.
```
fa33a63bd9 fuzz: Speed up PickValue in txorphan (MarcoFalke)
Pull request description:
`PickValue` will advance a begin iterator on the `outpoints` set, which is expensive, because it only has a `++` operator. As it is called in a loop of `num_in` (~`outpoints.size()`), the runtime is `O(outpoints.size() ^ 2)`.
Fix it by making the runtime linear.
ACKs for top commit:
glozow:
ACK fa33a63bd9, thanks for taking the suggestion
dergoegge:
utACK fa33a63bd9
Tree-SHA512: 33f440d97c6834d907d43a8d29e4fb2c995f0d244460bd079af100f13d3607a53e44a0db52f4eb5c487d98df0ff4f2f6d987bf94b922ae9f4506f1295ad6214c
8c935e625e depends: Fix CMake-generated `libevent*.pc` files (Hennadii Stepanov)
Pull request description:
Broken out of #30454. This is a backport of the merged upstream PR: https://github.com/libevent/libevent/pull/1622.
Note that after #29835 we might end up dropping pkg-config and using the installed CMake files directly, but that depends on whether or not enough distros actually ship those files.
Either way, having fixed up .pc files won't hurt.
ACKs for top commit:
hebasto:
ACK 8c935e625e.
fanquake:
ACK 8c935e625e
Tree-SHA512: 259c2ad78fb9e90370a7205dc71c40acda1a872f6509435133bc1c4c2c3de57366e80679aa083e13ed85e7966883dc470c0147ee171a2ed0171a18cd5ffc99b3
fa7bee13bf lint: Use git clone --depth=1 (MarcoFalke)
fadb7c2a91 lint: Add missing docker.io prefix to ci/lint_imagefile (MarcoFalke)
Pull request description:
Currently, the `ci/lint_imagefile` may pick the wrong (non-native) architecture due to the missing prefix.
For example, assuming the user has previously pulled an s390x image:
```
$ podman run --rm 'docker.io/s390x/debian:bookworm' dpkg --print-architecture
exec /usr/bin/dpkg: exec format error
```
Now, `debian:bookworm` will refer to the same image:
```
$ podman run --rm 'debian:bookworm' dpkg --print-architecture
exec /usr/bin/dpkg: exec format error
```
However, `docker.io/debian:bookworm` works fine:
```
$ podman run --rm 'docker.io/debian:bookworm' dpkg --print-architecture
arm64
```
(Also includes a nit-fix from https://github.com/bitcoin/bitcoin/pull/30499#discussion_r1686470495)
ACKs for top commit:
paplorinc:
utACK fa7bee13bf
hebasto:
ACK fa7bee13bf.
Tree-SHA512: 4b6d562c14c67bef984ad25f6a3a1ef7f1059dc2859c603c45083b36bcacafa3248fc74176e2e4626fdc39507e9353f458ddbc4077f805c03e970df46af02224
0388ad0d65 depends: switch zmq to CMake (Cory Fields)
fefb3bbe5b depends: add zeromq no librt patch (fanquake)
a522ef1542 depends: add zeromq cmake minimum patch (fanquake)
cbbc229adf depends: add zeromq windows usage patch (fanquake)
2de68d6d38 depends: add zeromq builtin sha1 patch (fanquake)
0c8605253a depends: add zeromq mktemp macos patch (fanquake)
Pull request description:
This picks up a change, which is a switch to building zeromq with CMake. It includes a number of patches, some which have already been upstreamed (see each patch for details).
ACKs for top commit:
hebasto:
ACK 0388ad0d65.
Tree-SHA512: 5567e432b4e4e0446c41d502bd61810a80b329dea2399b5d9d9f6e79acc450d1c6ba861c8238ba895de98338cfc5dc44ad2bf86ee8c222ecb3fbf47d6eb60da4
fa8d73e86e lint: Use consistent out-of-tree build for python and test_runner (MarcoFalke)
fa0f859885 doc: Clarify intent of ./ci/lint_run_all.sh (MarcoFalke)
fa9ad59f87 lint: Use $CI_RETRY_EXE when building ./ci/lint_imagefile (MarcoFalke)
Pull request description:
Fixes https://github.com/bitcoin/bitcoin/issues/30496
Seems odd to sometimes do an out-of-tree build (via `./ci/lint_imagefile`, see `test/lint/README.md`) and sometimes not (via Cirrus CI, see `./ci/lint_run_all.sh`).
Fix it by doing an out-of-tree build consistently in the same location.
Also, fix `$CI_RETRY_EXE`, while touching this.
ACKs for top commit:
josibake:
utACK fa8d73e86e
willcl-ark:
utACK fa8d73e86e
paplorinc:
utACK fa8d73e86e
Tree-SHA512: 4181ca14299a798850f5e05f180f3305a3378081ca8dabf6ab2da6115997cc17f6ef0f10db9b2b31618e59231083e5c4a971432d27b4d77903e655be21155abb
Previous code was confusing and brittle. For example, the full import
"source ./ci/test/00_setup_env.sh" and $PATH overwrite was not needed.
Fix it by simply copying the exe to /ci_retry and use that in
$CI_RETRY_EXE.
This is also a fix, because previously ci/lint_imagefile did use an
empty $CI_RETRY_EXE.
a8e3af1a82 qa: Do not assume running `feature_asmap.py` from source directory (Hennadii Stepanov)
9bf7ca6cad qa: Consider `cache` and `config.ini` relative to invocation directory (Hennadii Stepanov)
a0473442d1 scripted-diff: Add `__file__` argument to `BitcoinTestFramework.init()` (Hennadii Stepanov)
Pull request description:
This PR includes changes split from https://github.com/bitcoin/bitcoin/pull/30454. They improve the functional test framework, allowing users to [run individual functional tests](https://github.com/hebasto/bitcoin/issues/146) from the build directory in the new CMake-based build system.
This functionality is not available for out-of-source builds using the current Autotools-based build system, which always requires write permissions for the source directory. Nevertheless, this PR can be tested as suggested in https://github.com/bitcoin/bitcoin/pull/30463#issuecomment-2232618421:
1. Make an out-of-source build:
```
$ ./autogen.sh
$ mkdir ../build && cd ../build
$ ../bitcoin/configure
$ make
```
2. Create a symlink in the build directory to a functional test:
```
$ ln --symbolic ../../../bitcoin/test/functional/wallet_disable.py ./test/functional/
```
3. Run this symlink:
```
$ ./test/functional/wallet_disable.py
```
The last command fails on the master branch:
```
Traceback (most recent call last):
File "/home/hebasto/git/build/./test/functional/wallet_disable.py", line 31, in <module>
DisableWalletTest().main()
^^^^^^^^^^^^^^^^^^^
File "/home/hebasto/git/bitcoin/test/functional/test_framework/test_framework.py", line 106, in __init__
self.parse_args()
File "/home/hebasto/git/bitcoin/test/functional/test_framework/test_framework.py", line 210, in parse_args
config.read_file(open(self.options.configfile))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/hebasto/git/bitcoin/test/config.ini'
```
and succeeds with this PR.
ACKs for top commit:
maflcko:
tested ACK a8e3af1a82🎨
glozow:
ACK a8e3af1a82, tested with the steps in op
stickies-v:
ACK a8e3af1a82
Tree-SHA512: 899e4efc09edec13ea3f5b47825d03173fb21d3569c360deda7fa6a56b99b4d24e09ad4f0883bad1ee926b1c706e47ba07c6a6160c63c07c82b3cf4ae5816e91
7703884ab1 Fix MSVC warning C4273 "inconsistent dll linkage" (Hennadii Stepanov)
Pull request description:
Broken out of https://github.com/bitcoin/bitcoin/pull/30454.
When using CMake, the user can select the MSVC runtime library to be:
1) Statically-linked (with the corresponding `x64-windows-static` vcpkg triplet) or
2) Dynamically-linked (with the corresponding `x64-windows` vcpkg triplet)
In the latter case, the compiler emits the [C4273](https://learn.microsoft.com/en-us/cpp/error-messages/compiler-warnings/compiler-warning-level-1-c4273) warning.
As the "Necessary on some platforms" comment does not apply to MSVC, skip the declaration for MSVC.
The MSVC build system in the master branch supports the statically-linked runtime only: ed739d14b5/build_msvc/common.init.vcxproj.in (L65)
ACKs for top commit:
sipa:
utACK 7703884ab1
sipsorcery:
utACK 7703884ab1.
theuni:
utACK 7703884ab1
Tree-SHA512: a42e1a0d48973217462e703c418f3e9ef9cb5236267c1bf32912aacaf68976cdd2b9229168523f7c2a99ee3f2fb1bf8add4f342796bdb1e4063ca026b761db51
d318c4ef56 depends: bump libmultiprocess for CMake fixes (Cory Fields)
Pull request description:
Broken out of #30454 . Bumped [even further](4883197abc (r1684802528)) after https://github.com/chaincodelabs/libmultiprocess/pull/98 was merged upstream.
hebasto Presumably this approach works now with the CMake branch?
ACKs for top commit:
ryanofsky:
Code review ACK d318c4ef56.
Tree-SHA512: 4b5491f73c0063d09e8339829cd831b1f4c441dd7b55a22037c9337c80cce19bb00a3e5cf925efa77d6d4e89ab45482f40f5799bc14948f8cabcbad3c3549430
When using CMake, the user can select the MSVC runtime library to be:
1) Statically-linked (with the corresponding `x64-windows-static` vcpkg
triplet) or
2) Dynamically-linked (with the corresponding `x64-windows` vcpkg
triplet)
In the latter case, the compiler emits the C4273 warning.
As the "Necessary on some platforms" comment does not apply to MSVC,
skip the declaration for MSVC.
The CMake WIN32_WINNT autodetection is broken, and must be set
manually. We may want to set is explicitly in any case, but the
brokenness should also be fixed upstream.
Also patch out depends paths, that would cause non-determinism.
Co-authored-by: fanquake <fanquake@gmail.com>
ff4f3deb7b depends: use CMake to build FreeType (fanquake)
Pull request description:
Switches Freetype to be built with CMake.
ACKs for top commit:
theuni:
ACK ff4f3deb7b
hebasto:
ACK ff4f3deb7b, I've verified the actual compile options, they look sane.
Tree-SHA512: e9e4348975998539fde88a84d110d53dbac50ae9cc3fa692d15e09313d6fdb6acb3bb23533786a645fc836091075b4487d6de42ef78ba3a44de46d06360aef4f
Also pulls out the guarding assert and calls it explicitly before the test function is called. This is already done before the existing call of the test function so it was not needed there.
- "Valid snapshot file, but referencing a snapshot block that turns out
to be invalid, or has an invalid parent" has been addressed in #30267
- "An ancestor of snapshot block" - If chain tip refers to blocks in this context then any successful load is addressing this because if we have synced past the snapshot base block we fail because we don't need assumeutxo anymore. And if this is about headers then this is the `test_headers_not_synced()` case.
- "A descendant of the snapshot block" - If this refers to blocks the
`test_snapshot_with_less_work()` addressed this and if it is just headers in this case again it would be represented in all of the successful loads in the test.
Co-authored-by: Alfonso Roman Zubeldia <alfonsoromanz24@gmail.com>
