6a68343ffb doc: Prepend 'build/' to binary paths under 'src/' in docs (Lőrinc)
91b3bc2b9c doc: Update documentation generation example in developer-notes.md (Lőrinc)
Pull request description:
In [the other readmes](6ce50fd9d0/src/test/README.md (L19)) we've provided a default build directory instead, unified the `developer-notes.md` to specify it explicitly.
In the next commit I've used this default to go over each reference to our binaries and changed their in-source references to the build directory.
Some of these changes were in example outputs - I haven't validated that the outputs are still the same.
I haven't modified the build folders in the devtools.
ACKs for top commit:
maflcko:
review ACK 6a68343ffb
pablomartin4btc:
ACK 6a68343ffb
fanquake:
ACK 6a68343ffb - we still need to followup with other scripts/devtools, and likely unify what we are doing in some way, but this is an improvement.
Tree-SHA512: 905d9c68cafe1e405e98d6aa089d7a36a34c9e03403df5c67ac2c9a98cfa54a0305b647cb92247dcb9f49e9b509a8ba88367392b95618c67059684c67b6c36fb
fa09cb41f5 refactor: Remove unused LogPrint (MarcoFalke)
3333415890 scripted-diff: LogPrint -> LogDebug (MarcoFalke)
Pull request description:
`LogPrint` has many issues:
* It seems to indicate that something is being "printed", however config options such as `-printtoconsole` actually control what and where something is logged.
* It does not mention the log severity (debug).
* It is a deprecated alias for `LogDebug`, according to the dev notes.
* It wastes review cycles, because reviewers sometimes point out that it is deprecated.
* It makes the code inconsistent, when both are used, possibly even in lines right next to each other (like in `InitHTTPServer`)
Fix all issues by removing the deprecated alias.
I checked all conflicting pull requests and at the time of writing there are no conflicts, except in pull requests that are marked as draft, are yet unreviewed, or are blocked on feedback for other reasons. So I think it is fine to do now.
ACKs for top commit:
stickies-v:
ACK fa09cb41f5
danielabrozzoni:
utACK fa09cb41f5
TheCharlatan:
ACK fa09cb41f5
Tree-SHA512: 14270f4cfa3906025a0b994cbb5b2e3c8c2427c0beb19c717a505a2ccbfb1fd1ecf2fd03f6c52d22cde69a8d057e50d2207119fab2c2bc8228db3f10d4288d0f
41051290ab cmake: Ignore build subdirectories within source directory (Hennadii Stepanov)
6ce50fd9d0 doc: Update for CMake-based build system (Hennadii Stepanov)
9730288a0c ci: Migrate CI scripts to CMake (Hennadii Stepanov)
c360837ca5 cmake, lint: Adjust `lint_includes_build_config` (Hennadii Stepanov)
3885441ee0 cmake: Add presets for native Windows builds (Hennadii Stepanov)
7681746b20 cmake: Add vcpkg manifest file (Hennadii Stepanov)
8b6f1c4353 cmake: Add `Coverage` and `CoverageFuzz` scripts (Hennadii Stepanov)
65bdbc1ff2 cmake: Add `docs` build target (Hennadii Stepanov)
fb75ebbc33 cmake: Add compiler diagnostic flags (Hennadii Stepanov)
e821f0a37a cmake: Migrate Guix build scripts to CMake (Hennadii Stepanov)
747adb6ffe cmake: Add `Maintenance` module (Hennadii Stepanov)
1f60b30df0 cmake: Add `APPEND_{CPP,C,CXX,LD}FLAGS` cache variables (Hennadii Stepanov)
2b43c45b13 cmake: Add `AddWindowsResources` module (Hennadii Stepanov)
973a3b0c5d cmake: Implement `install` build target (Hennadii Stepanov)
84ac35cfd4 cmake: Add cross-compiling support (Hennadii Stepanov)
0d01c228a7 build: Generate `toolchain.cmake` in depends (Hennadii Stepanov)
91a799247d depends: Add host-specific `cmake_system_version` variables (Hennadii Stepanov)
9b31209b4c depends: Rename `cmake_system` -> `cmake_system_name` (Hennadii Stepanov)
4a5208a81d Revert "build, qt: Do not install *.prl files" (Hennadii Stepanov)
6522af62af depends: Amend handling flags environment variables (Hennadii Stepanov)
90cec4d251 cmake: Add `MULTIPROCESS` option (Hennadii Stepanov)
bb1a450dcb cmake: Build `bitcoin-chainstate` executable (Hennadii Stepanov)
aed38ea58c cmake: Build `bitcoinkernel` library (Hennadii Stepanov)
975d67369b cmake: Build `test_bitcoin-qt` executable (Hennadii Stepanov)
10fcc668a3 cmake: Add `WITH_DBUS` option (Hennadii Stepanov)
5bb5a4bc75 cmake: Add `libqrencode` optional package support (Hennadii Stepanov)
57a6e2ef4a cmake: Build `bitcoin-qt` executable (Hennadii Stepanov)
30f642952c cmake: Add `WERROR` option (Hennadii Stepanov)
c98d4a4c34 cmake: Add `REDUCE_EXPORTS` option (Hennadii Stepanov)
a01cb6e63f cmake: Add `HARDENING` option (Hennadii Stepanov)
a8a2e364ac cmake: Add Python-based tests (Hennadii Stepanov)
3d85379570 cmake: Add fuzzing options (Hennadii Stepanov)
908530e312 cmake: Add `SANITIZERS` option (Hennadii Stepanov)
8bb0e85631 cmake: Build `bench_bitcoin` executable (Hennadii Stepanov)
801735163a cmake: Add external signer support (Hennadii Stepanov)
353e0c9e96 cmake: Add `systemtap-sdt` optional package support (Hennadii Stepanov)
d2fda82b49 cmake: Add `libzmq` optional package support (Hennadii Stepanov)
ae7b39a0e1 cmake: Add `libminiupnpc` optional package support (Hennadii Stepanov)
6480e1dcdb cmake: Add `libnatpmp` optional package support (Hennadii Stepanov)
e73e9304a1 cmake: Build `bitcoin-util` executable (Hennadii Stepanov)
027c6d7caa cmake: Build `bitcoin-tx` executable (Hennadii Stepanov)
d10c5c34c3 cmake: Add wallet functionality (Hennadii Stepanov)
ab2e99b0d9 cmake: Create test suite for `ctest` (Hennadii Stepanov)
959370bd76 cmake: Build `test_bitcoin` executable (Hennadii Stepanov)
b27bf9700d cmake: Build `bitcoin-cli` executable (Hennadii Stepanov)
a9813df826 cmake: Build `bitcoind` executable (Hennadii Stepanov)
97829ce2d5 cmake: Add `FindLibevent` module (Hennadii Stepanov)
3118e40c61 cmake: Build `bitcoin_consensus` library (Hennadii Stepanov)
809a2f1929 cmake: Build `bitcoin_util` static library (Hennadii Stepanov)
0a9a521a70 cmake: Build `bitcoin_crypto` library (Hennadii Stepanov)
958971f476 cmake: Build `univalue` static library (Hennadii Stepanov)
752747fda8 cmake: Generate `obj/build.h` header (Hennadii Stepanov)
1f0a78edf3 cmake: Build `minisketch` static library (Hennadii Stepanov)
12bfbc8154 cmake: Build `leveldb` static library (Hennadii Stepanov)
51985c5304 cmake: Build `crc32c` static library (Hennadii Stepanov)
db7a198f29 cmake: Build `secp256k1` subtree (Hennadii Stepanov)
dbb7ed14e8 cmake: Add `ccache` support (Hennadii Stepanov)
cedfdf6c72 cmake: Redefine/adjust per-configuration flags (Hennadii Stepanov)
b6b5e732c8 cmake: Add global compiler and linker flags (Hennadii Stepanov)
f98327931b cmake: Add `TryAppendLinkerFlag` module (Hennadii Stepanov)
4a0af29697 cmake: Add `TryAppendCXXFlags` module (Hennadii Stepanov)
35cffc497d cmake: Add POSIX threads support (Hennadii Stepanov)
fd72d00ffe cmake: Add position independent code support (Hennadii Stepanov)
07069e2bb0 cmake: Add introspection module (Hennadii Stepanov)
27d687fc1f cmake: Add `config/bitcoin-config.h` support (Hennadii Stepanov)
fe5cdace5f cmake: Print compiler and linker flags in summary (Hennadii Stepanov)
70683884c5 cmake: Introduce interface libraries to encapsulate common flags (Hennadii Stepanov)
a2317e27b7 cmake: Add root `CMakeLists.txt` file (Hennadii Stepanov)
Pull request description:
This PR introduces a new CMake-based build system, which is a drop-in replacement for the current Autotools-based build system.
ML announcement: https://groups.google.com/g/bitcoindev/c/hgKkfQWzrTo
As discussed during the recent CoreDev meetup in April, the switch from Autotools to CMake is intended to happen as soon as possible after branching 28.x off, which means that 29.0 will be built using CMake.
This PR branch is essentially the [staging branch](https://github.com/hebasto/bitcoin/tree/cmake-staging), with every change reviewed and tested by a group of contributors, including (in alphabetical order):
- [**achow101**](https://github.com/achow101)
- [**fanquake**](https://github.com/fanquake)
- [**maflcko**](https://github.com/maflcko)
- [**m3dwards**](https://github.com/m3dwards)
- [**pablomartin4btc**](https://github.com/pablomartin4btc)
- [**real-or-random**](https://github.com/real-or-random)
- [**ryanofsky**](https://github.com/ryanofsky)
- [**sipsorcery**](https://github.com/sipsorcery)
- [**TheCharlatan**](https://github.com/TheCharlatan)
- [**theStack**](https://github.com/theStack)
- [**theuni**](https://github.com/theuni)
- [**vasild**](https://github.com/vasild)
Reviewing in a separate staging repo was suggested in https://github.com/bitcoin/bitcoin/pull/27060#issuecomment-1431798320.
The accompanying changes to the OSS-Fuzz project are available in https://github.com/hebasto/oss-fuzz/pull/8.
Please refer to the [build options parity table](https://gist.github.com/hebasto/2ef97d3a726bfce08ded9df07f7dab5e). The "auto" value is no longer available; non-default values must be specified explicitly. Additionally, the new default values have been chosen to suit the everyday build experience for the majority of developers.
System requirements for using the CMake-based build system:
- CMake >= 3.22 (if not available in your system's repository, it can be downloaded from https://cmake.org/download/)
- a build tool of your choice:
- any Make (GNU Make is no longer a requirement); GNU Make is still required to build depends
- Ninja (https://ninja-build.org/)
- MSBuild
- Xcode
A note for Windows users: The default installation of the latest version of MSVC 17.10.4 includes both CMake 3.28.3 and the vcpkg package manager).
---
We, the build system developers, kindly ask reviewers to refrain from making suggestions that are not directly related to the migration process or can be implemented separately. Bugs in the scripts and errors in the updated documentation should be the focus of this PR. Please be advised that comments not aligned with this PR's goal may be ignored.
Thank you all for your understanding.
ACKs for top commit:
maflcko:
review ACK 41051290ab🐥
sipsorcery:
ACK 41051290ab.
vasild:
ACK 41051290ab
TheCharlatan:
ACK 41051290ab
pablomartin4btc:
tACK 41051290ab
i-am-yuvi:
tACK [`4105129`](41051290ab)
theuni:
ACK 41051290ab.
fanquake:
ACK 41051290ab
Tree-SHA512: 6c1445054436c6c00ad63bfa0f19d64091a2b25c9bd694f85bf2218ac358ffb774d6c000685b3ca1e9b50401babed989fa2a0694b774c211d226bfd1944c9b39
Regenerate mainnet seeds from new sources without the need for hardcoded
data. Result has 512 nodes from each network type except cjdns, for
which only eight nodes were found that match the seed node criteria.
Pull additional nodes from virtu's crawler. Data includes sufficient
Onion and I2P nodes to align the uptime requirements for these networks
to that of clearnet nodes (i.e., 50%). Data also includes more than
three times the number of CJDNS nodes currently hardcoded into
nodes_main_manual.txt, so hardcoded nodes becomes obsolete.
41ad84a00c seeds: Use fjahr's more up to date asmap (Ava Chow)
d8fd1e0faf seeds: Fixed seeds update (Ava Chow)
f1f24d7214 seeds: Add testnet4 fixed seeds file (Ava Chow)
8ace71c737 seeds: Remove manual onion and i2p seeds (Ava Chow)
ed5b86cbe4 seeds: Add testnet instructions (Ava Chow)
0676515397 seeds: Also pull from achow101 seeder (Ava Chow)
5bab3175a6 makeseeds: Configurable minimum blocks for testnet4's smaller chain (Ava Chow)
d2465dfac6 makeseeds: Shuffle ips after parsing (Ava Chow)
af550b3a0f makeseeds: Support CJDNS (Ava Chow)
d5a8c4c4bd makeseeds: Update user agent regex (Ava Chow)
Pull request description:
The [DNS seeder](https://github.com/achow101/dnsseedrs) that I wrote collects statistics on node reliability in the same way that sipa's seeder does, and also outputs this information in the same file format. Thus it can also be used in our fixed seeds update scripts. My seeder additionally crawls onion v3, i2p, and cjdns, so will now be able to set those fixed seeds automatically rather than curating manual lists.
In doing this update, I've found that `makeseeds.py` is missing newer versions from the regex as well as cjdns support; both of these have been updated.
I also noticed that the testnet fixed seeds are all manually curated and sipa's seeder does not appear to publish any testnet data. Since I am also running the seeder for testnet, I've added the commands to generate testnet fixed seeds from my seeder's data too.
Lastly, I've updated all of the fixed seeds. However, since my seeder has not found any cjdns nodes that met the reliability criteria (possibly due to connectivity issues present in those networks), I've left the previous manual seeds for that network.
ACKs for top commit:
fjahr:
re-ACK 41ad84a00c
virtu:
ACK [41ad84a](41ad84a00c)
Tree-SHA512: 6ba0141f053d9d6ae7d8c9574f061be38f3e65b28de1d6660c1885ab942623b5a0ec70754b4fcfc5d98fe970f5f179a940d5880b5061ed698f7932500e01d3ee
5b4f34006d devtools, utxo-snapshot: Fix block height out of range (pablomartin4btc)
Pull request description:
<details>
<summary>Fixing a <a href="https://github.com/bitcoin/bitcoin/pull/28553#pullrequestreview-2251032570">bug</a> in <code>utxo_snapshot.sh</code>.</summary>
```
/contrib/devtools/utxo_snapshot.sh 840000 snapshot2.dat ./src/bitcoin-cli -datadir=${AU_DATADIR}
Do you want to disable network activity (setnetworkactive false) before running invalidateblock? (Y/n):
Disabling network activity
false
error code: -8
error message:
Block height out of range
```
And the user will see the following in the node and it would stay there if not reset:
```
2024-08-21T14:44:13Z UpdateTip: new best=00000000000000afa0cd000a16e244f56032735d41acd32ac00337aceb2a5240 height=235382 version=0x00000002 log2_work=69.987697 tx=17492185 date='2013-05-09T23:54:32Z' progress=0.016219 cache=71.0MiB(571085txo)
2024-08-21T14:44:13Z UpdateTip: new best=0000000000000087c5e0b820afff496b95ba44ad64640c73b234d3261d3f99d2 height=235383 version=0x00000002 log2_work=69.987750 tx=17492341 date='2013-05-09T23:54:47Z' progress=0.016219 cache=71.0MiB(571291txo)
2024-08-21T14:44:13Z UpdateTip: new best=000000000000014a4b5fddf3c8abb6209247255ca9e8df786b271dd1b2ac82a6 height=235384 version=0x00000002 log2_work=69.987804 tx=17492344 date='2013-05-10T00:20:18Z' progress=0.016219 cache=71.0MiB(571297txo)
2024-08-21T14:44:13Z SetNetworkActive: false
```
</details>
This is a "temporary" fix until #29553 gets merged, which will remove the script entirely.
Handle the "Block height out of range" error gracefully by checking if the node has synchronized to or beyond the required block height, otherwise without this validation the node would keep the network disabled if the user selected that option.
<details>
<summary>Provide a user-friendly message if the block height is out of range and exit the script cleanly.</summary>
```
/contrib/devtools/utxo_snapshot.sh 840000 snapshot2.dat ./src/bitcoin-cli -datadir=${AU_DATADIR}
Error: The node has not yet synchronized to block height 840001.
Please wait until the node has synchronized past this block height and try again.
```
</details>
ACKs for top commit:
achow101:
ACK 5b4f34006d
fjahr:
tACK 5b4f34006d
Tree-SHA512: 2b71286b627872d7cfdb367e29361afa3806a7ef9d65075b93892b735ff2ab729069e2f7259d30262909e73cef17fb7dca231615cc1863968cd042f4a2a4f901
Handle the Block height out of range error gracefully by checking if
the node has synchronized to or beyond the required block height,
otherwise without this validation the node would keep the network
disabled if the user selected that option.
Provide a user-friendly message if the block height is out of range
and exit the script cleanly.
The seeders now produce onion and i2p seeds, so there is no need to keep these
in the manual list.
Although should also be produced, there are not enough
good ones detected by the seeder, so we keep the manual seeds for them.
The crawlers are not guaranteed to output nodes in a random order, so
shuffle the ips list after parsing to break any biasing that may be
caused by the output order.
Update the user agent regex to match all 3 digits of the version number,
not just the first 2 digits.
Also updates it to include 24.2, 25.2, 26.1, 27.0, 27.1, 27.99, 28.0 and
28.99.
5215c925d1 Compare ASMaps with respect to specific addresses (virtu)
Pull request description:
Right now, we have no way to quantify the "degradation" of an ASMap over time in the context of Bitcoin's P2P network in a meaningful way. However, such data would be useful for:
1. Making sure the minimum shelf life of ASMaps is compatible with the release cycle (we wouldn't want to start shipping ASMaps with releases before making sure ASMaps typically do not become obsolete before the time of the next release)
2. Node operators eager to keep their ASMaps up-to-date between releases.
While `asmap-tool.py` has a `diff` command to perform a prefix-based comparison of two ASMaps, it is hard to reason about whether an old ASMap still is "good enough" or should be replaced with a newer one based on a prefix-based diff such as the following:
```shell
$ ./asmap-tool.py diff 1704463200_asmap.dat 1710770400_asmap.dat
[...]
# 2c0f:fc98::/32 was AS37282
# 2c0f:fcb8::/32 was AS37323
2c0f:ff18::/32 AS37044 # was unassigned
2c0f:ff98::/32 AS37113 # was unassigned
2c0f:ffa0::/32 AS37273 # was unassigned
# 76082350 (2^26.18) IPv4 addresses changed; 834271985742505274886878979424260 (2^109.36) IPv6 addresses changed
```
One option for a more Bitcoin-centric ASMap comparison comprises comparing ASNs for the addresses of Bitcoin nodes and reporting on the number/share of addresses of nodes with disagreeing ASNs. By applying this approach to a node's set of known peers, a node operator can estimate how many of the node's peers are mapped to out-of-date AS when using the currently deployed and an up-to-date ASMap as input.
This PR adds this functionality to `asmap-tool.py` by introducing a `diff_addrs` subcommand. In addition to two ASMaps, the subcommand reads addresses from a (`getnodeaddresses`-compatible) file, and computes statistics for those addresses:
```bash
$ ./asmap-tool.py diff_addrs 1704463200_asmap.dat 1710770400_asmap.dat <(bitcoin-cli getnodeaddresses 0)
275 address(es) reassigned from unassigned to AS51167
84 address(es) reassigned from AS198949 to AS15557
66 address(es) reassigned from AS45758 to AS45629
33 address(es) reassigned from AS174 to AS212238
[...]
1 address(es) reassigned from unassigned to AS399619
Summary: 919 (1.67%) of 54,901 addresses were reassigned.
```
When the `-s / --show-addresses` flag is used, addresses subject to reassignment are included in the output.
ACKs for top commit:
fjahr:
tACK 5215c925d1
achow101:
ACK 5215c925d1
brunoerg:
reACK 5215c925d1
Tree-SHA512: ebcf47754bce92794fad9f4c3bfc1c5e9daf077db5975f444c5135092eb6a26ecaa1eca6748a03ae0c87d9e45532426966fe8f3c17249b17f9dcad490d6dd3bf
8fee5355ee guix: fix suggested fake date for openssl -1.1.1l (Sjors Provoost)
Pull request description:
Using `2020-10-01` as the fake timestamp will cause many test failures with `/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv`. I didn't investigate why, but I guess because it's _before_ the test certificates were created. They expired in June 2022. I tried a month before that, which worked.
Also fixes layout of instructions.
ACKs for top commit:
achow101:
ACK 8fee5355ee
maflcko:
review ACK 8fee5355ee
Tree-SHA512: df5dd3aa961e25bd57d0b8b73daeb3ec76856b06e35277f24b6b19be81774512228f75e2b779afa8ea92fcc39beb869f43e0c57fba19ad16a82812e7c0bea38b
77ff0ec1f1 contrib: support reading XORed blocks in linearize-data.py script (Sebastian Falbesoner)
Pull request description:
This PR is a small follow-up for #28052, adding support for the block linearization script to handle XORed blocksdir *.dat files. Note that if no xor.dat file exists, the XOR pattern is set to all-zeros, in order to still support blockdirs that have been created with versions earlier than 28.x.
Partly fixes issue #30599.
ACKs for top commit:
achow101:
ACK 77ff0ec1f1
tdb3:
ACK 77ff0ec1f1
hodlinator:
ACK 77ff0ec1f1
Tree-SHA512: 011eb02e2411de373cbbf4b26db4640fc693a20be8c2430529fba6e36a3a3abfdfdc3b005d330f9ec2846bfad9bfbf34231c574ba99289ef37dd51a68e6e7f3d
2925bd537c refactor: use c++20 std::views::reverse instead of reverse_iterator.h (stickies-v)
Pull request description:
C++20 introduces [`std::ranges::views::reverse`](https://en.cppreference.com/w/cpp/ranges/reverse_view), which allows us to drop our own `reverse_iterator.h` implementation and also makes it easier to chain views (even though I think we currently don't use this).
ACKs for top commit:
achow101:
ACK 2925bd537c
maflcko:
ACK 2925bd537c🎷
Tree-SHA512: 567666ec44af5d1beb7a271836bcc89c4c577abc77f522fcc18bc6d4de516ae9b0df766d0bfa6dd217569e6878331c2aee1d9815620860375e3510dad7fed476
chainparams.cpp - workaround for MSVC bug triggering C7595 - Calling consteval constructors in initializer lists fails, but works on GCC (13.2.0) & Clang (17.0.6).
bda537f7c4 depends: remove ENV unsetting for darwin (fanquake)
1807760f09 guix: improve ENV unsetting for macOS (fanquake)
0b2aeee21d depends: patch explicit -lm usage out of Qt tools (fanquake)
Pull request description:
Now that we use the native compiler, and have fixed Qt, and these vars
are (almost) unset in Guix, we can remove the unsetting from our compiler
command here.
I couldn't manage to make a darwin-clang-cross only exclusion of `-lm` work properly
for Qt, so opted for just removing the explicit link entirely. I do not think this should have
any other unwanted side-effects.
Fixes#21552.
ACKs for top commit:
TheCharlatan:
ACK bda537f7c4
Tree-SHA512: 97a2d85de7d4b1d65717ecb521399ecba5f53863b8aef21af62ede5ceee59ee1a9392663da3a3852cad1b6d8b420dd4b0b5f0eea38d30a81785d8b2718620b5f
d1592d2eee guix: use gcc-12 to compile winpthreads (fanquake)
b23690e821 guix: use GCC 12.4.0 over 12.3.0 (fanquake)
8b41ede55e guix: consolidate back to GCC 12 toolchain for all HOSTS (fanquake)
Pull request description:
This PR contains 3 changes:
* Bump GCC in Guix from [12.3.0 to 12.4.0](https://gcc.gnu.org/gcc-12/). A patch was sent upstream, https://lists.gnu.org/archive/html/guix-patches/2024-06/msg01025.html, but has not landed.
* Consolidate all build environments back to using a GCC 12 toolchain. After #21778, the macOS environment is no-longer pinned to 11 (12 would otherwise cause issues building cctools). So, instead of requiring all builders to compile an additional GCC toolchain, use 12.
* Use GCC 12 to compile winpthreads. Currently, GCC 11 is used; which became apparent in https://github.com/bitcoin/bitcoin/pull/30452#issuecomment-2244715566.
ACKs for top commit:
TheCharlatan:
ACK d1592d2eee
hebasto:
ACK d1592d2eee.
Tree-SHA512: e3aa1fa3e69500c93180e07cb4684661247ec6bc45245f746538d81406ff1d8777131590307496dda3287a112b6633e4991168586ca4c2036fa3a57b1efa9c87
Using GCC 11 for the macOS build hasn't been required since #21778, and
at this point, given a toolchain is still needed (#30206), it makes more
sense to (re-)use 12, rather than make all builders compile another
GCC toolchain.
If the binaries don't exist, the Guix build has failed for some other
reason.
There's no need to check for unknown architectures, or executable
formats, as the only ones that could be built are those that we've
configured toolchains for in Guix.
We've also been doing this inconsistently across the two scripts.
9010b1343b contrib: c++ify test stubs after switching to c++ compilers (Cory Fields)
261f770333 contrib: rename cc to cxx in binary checking scripts (Cory Fields)
a38c960005 contrib: use c++ rather than c for binary tests (Cory Fields)
Pull request description:
From hebasto's CMake repo. See discussion here: https://github.com/hebasto/bitcoin/pull/252#discussion_r1664657488
Use CXX/CXXFLAGS rather than CC/CFLAGS to test our actual compiler for binary checks rather than the one we only forward to secp256k1.
ACKs for top commit:
hebasto:
ACK 9010b1343b.
fanquake:
ACK 9010b1343b
Tree-SHA512: 7b8788d7d3760103062eff10056c995e1ad14c0c487d9414683ad54d816c255d0ca751f4d0e2d2ad7f9e8a7101d8c7f1e9333fa5b137558ed68fa593c4b4ce6d
34c9cee380 clang-tidy: add check for non-trivial thread_local vars (Cory Fields)
Pull request description:
Forbid thread_local vars with non-trivial destructors.
This is a follow-up from: https://github.com/bitcoin/bitcoin/pull/30095#discussion_r1608423170
ACKs for top commit:
maflcko:
ACK 34c9cee380
TheCharlatan:
Re-ACK 34c9cee380
Tree-SHA512: 3a798607fb189a5bbc714ed6e86dea462fe29d366b790e96d10a7b4ffcf1f194da9b8f4cd0b82154408709b8e3c58d3f613d6311903bd65a76d8b556ab230d21
Introduce diff_addrs subcommand as means for a Bitcoin-centric
comparison of two ASMaps.
In addition to two ASMaps, the subcommand reads addresses from
a (getnodeaddresses-compatible) file, and provides information on
addresses that have mismatching ASN according to the two ASMaps.
b5fc6d46a3 guix: use glibc 2.31 (fanquake)
Pull request description:
Set minimum required glibc to 2.31.
The glibc 2.31 branch is still maintained: https://sourceware.org/git/?p=glibc.git;a=shortlog;h=refs/heads/release/2.31/master.
Remove the stack-protector check from test-security-check, as the test
no-longer fails, and given the control we have of the end, the actual
security-check test seems sufficient (this might also be applied to some
of the other checks).
Drops runtime support for Ubuntu Bionic 18.04 and RHEL-8 from the release binaries.
ACKs for top commit:
TheCharlatan:
ACK b5fc6d46a3
Tree-SHA512: ba7e727240fa0ebebfb8b749024c71cbfdec37c33b39627866d78f9318ccdc687fd5103a63ee0e98cf809d9954dde56b1b305691c33d1de275ed0519f716c921
3ab2520190 contrib: Fixup verify-binaries OS platform parsing (Ben Westgate)
Pull request description:
Closes#30145.
This PR solves two major issues in the `parse_version_string` function of verify-binaries:
1. `-aarch64` binaries cannot be specifically downloaded. The -platform string gets interpreted as a release candidate that doesn't exist due to containing sub-string "rc".
2. Specifying a platform with a "-" in the name causes the parser to ignore both "-platform" AND "-rcN" and download the potentially wrong (non-rc) version for every platform. This also prevented specifying just one platform binary the user wished to download.
It also updates the accompanying `test.py` to cover problem two and adds two examples that were formerly broken to `README.md` to show what is now possible. Including the most useful case of downloading only 1 specific platform's binary.
This improves the Bitcoin verify-binaries tools user experience by not:
1. Failing to download for inexplicable reasons,
2. Downloading more files than what the user told it to, or in the worst case
3. Downloading only the wrong files.
* A test was added to cover the command `verify-binaries/verify.py pub 22.0-x86_64-linux-gnu.tar.gz` which checks that _bitcoin-22.0-x86_64-linux-gnu.tar.gz_ downloads successfully AND ONLY _bitcoin-22.0-x86_64-linux-gnu.tar.gz_ downloads.
* The steps to reproduce each bug are in the referenced issue #30145. Explanation of the potential issue as well as reasoning for the way the bug was fixed are in my commit descriptions.
* This delivers the promised feature of "only download the binaries for a certain platform", by allowing strings with '-' to be accepted, allowing for single file downloads for any specific platform which was not always possible before.
* Removes 6 lines of code from the offending `parse_version_string` function, while fixing the bugs/errors, and extending the functionality to be practical for users with slow connections.
* Makes the error message more helpful when no file matches the provided platform string, now prints "Did you mean: `closest-match`" to help correct typos.
Thanks for reading my PR. I look forward to getting this helpful tool in its best shape yet.
Log of this branch passing the new test.py:
```
python3 test.py
✓ 'Nonexistent version should fail' passed
✓ 'Malformed version should fail' passed
✓ '--min-good-sigs 20 should fail' passed
- testing verification (22.0-x86_64-linux-gnu.tar.gz)
✓ '22.0-x86_64-linux-gnu.tar.gz should succeed' passed
- testing verification (22.0)
✓ '22.0 should succeed' passed
```
Log of master failing the new test.py:
```
python3 test.py
✓ 'Nonexistent version should fail' passed
✓ 'Malformed version should fail' passed
✓ '--min-good-sigs 20 should fail' passed
- testing verification (22.0-x86_64-linux-gnu.tar.gz)
✓ '22.0-x86_64-linux-gnu.tar.gz should succeed' passed
Traceback (most recent call last):
File "/home/ben/Documents/GitHub/bitcoin/contrib/verify-binaries/test.py", line 74, in <module>
main()
File "/home/ben/Documents/GitHub/bitcoin/contrib/verify-binaries/test.py", line 27, in main
assert len(v) == 1
^^^^^^^^^^^
AssertionError
```
ACKs for top commit:
stickies-v:
re-ACK 3ab2520190
willcl-ark:
re-ACK 3ab2520190
Tree-SHA512: 6093228bb876cd0ac84d1cd2630074e17a3f09c4b23325b9419d859a5721a802f928844575233b135df52b882287dd18d6fadf4419d88ec0a2cdf82db315329e
Parse platform strings with "-" or '.' correctly such as "linux-gnu" or
"x86_64-linux-gnu.tar.gz" to download the matching files or file. String
partition() is used to tolerate more dashes. Update `VERSION_EXAMPLE`
with a new string parsed correctly now. Fix "-aarch64" interpreted as a
release candidate due to sub-string "rc", causing all downloads to fail.
Now "rc" must immediately follow first "-" to indicate an [-rc] string.
Local variables `version_rc`, `version_os` renamed to `rc`, `platform`.
If "-rcN" is specified, `platform` is reassigned to remove the '-rcN'.
Changes are useful to only download one bitcoin core binary on slow
connections. Making `verify.py pub` more intuitive, robust, and
versatile. Closes#30145
When user types a platform string not found in any filename lets help
and say the platform closest to what they typed in a `f"No files
matched the platform specified. Did you mean: {closest_match}"` log.
Improves UX when unaware how we name our files.
Uses the difflib Python built-in which was already imported elsewhere.
Update test.py to test single file verification
verify-binaries/verify.py can accept an entire filename filter for its
"-platform" parameter now so let us test that it succeeds and downloads
and verifies only one file. `verify.py pub 22.0-x86_64-linux-gnu.tar.gz`
should get and verify only the requested binary. It is placed before the
existing <version> wide verification as it is a faster test and possibly
easier to break.
Update doc with examples now possible after bugfix
Add example to show release candidates now work with "-platform" strings
containing "-" and string provided can be from the middle of filename:
`./contrib/verify-binaries/verify.py --json pub 23.0-rc5-linux-gnu`
Change example 5 to not match example 3.
New examples to show platform can now be provided specifically enough to
download only a single binary down to its file extension:
`./contrib/verify-binaries/verify.py pub 25.2-x86_64-linux`
`./contrib/verify-binaries/verify.py pub 24.1-rc1-darwin`
`./contrib/verify-binaries/verify.py pub 27.0-win64-setup.exe`
This is the most common use if not verifying all files so users see it
as the first example for "only download the binaries for a certain
architecture and/or platform". Downloading one file is intuitively what
most will think this meant and this change delivers on that expectation.
Co-authored-by: stickies-v
7c298fe0df doc: rewrite some of the macdeploy docs (fanquake)
d042230f7a depends: swap mmacosx-version-min for mmacos-version-min (fanquake)
Pull request description:
Whilst `-mmacosx-version-min` and `-mmacos-version-min` remain aliases for each other, the later is preferred,
and I assume the former will be removed at some point in the future; see: https://github.com/llvm/llvm-project/pull/95374.
Somewhat of a followup to #21778. Rewrite some of the mac deploy docs.
ACKs for top commit:
theuni:
utACK 7c298fe0df
TheCharlatan:
ACK 7c298fe0df
hebasto:
ACK 7c298fe0df.
Tree-SHA512: 6493f087fde93e0eec319af0e105d163b3f047d8a03f7d4b0d6cd7c64b58d0a978b7d67c6b8dba5c6ccf8b10e188aab5dc98eec400b0546dc9ee801a689b4332
c7376babd1 doc: Clarify distinction between util and common libraries in libraries.md (Ryan Ofsky)
4f74c59334 util: Move util/string.h functions to util namespace (Ryan Ofsky)
4d05d3f3b4 util: add TransactionError includes and namespace declarations (Ryan Ofsky)
680eafdc74 util: move fees.h and error.h to common/messages.h (Ryan Ofsky)
02e62c6c9a common: Add PSBTError enum (Ryan Ofsky)
0d44c44ae3 util: move error.h TransactionError enum to node/types.h (Ryan Ofsky)
9bcce2608d util: move spanparsing.h to script/parsing.h (Ryan Ofsky)
6dd2ad4792 util: move spanparsing.h Split functions to string.h (Ryan Ofsky)
23cc8ddff4 util: move HexStr and HexDigit from util to crypto (TheCharlatan)
6861f954f8 util: move util/message to common/signmessage (Ryan Ofsky)
cc5f29fbea build: move memory_cleanse from util to crypto (Ryan Ofsky)
5b9309420c build: move chainparamsbase from util to common (Ryan Ofsky)
ffa27af24d test: Add check-deps.sh script to check for unexpected library dependencies (Ryan Ofsky)
Pull request description:
Remove `fees.h`, `errors.h`, and `spanparsing.h` from the util library. Specifically:
- Move `Split` functions from `util/spanparsing.h` to `util/string.h`, using `util` namespace for clarity.
- Move remaining spanparsing functions to `script/parsing.h` since they are used for descriptor and miniscript parsing.
- Combine `util/fees.h` and `util/errors.h` into `common/messages.h` so there is a place for simple functions that generate user messages to live, and these functions are not part of the util library.
Motivation for this change is that the util library is a dependency of the kernel, and we should remove functionality from util that shouldn't be called by kernel code or kernel applications. These changes should also improve code organization and make functions easier to discover. Some of these same moves are (or were) part of #28690, but did not help with code organization, or made it worse, so it is better to move them and clean them up in the same PR so code only has to change one time.
ACKs for top commit:
achow101:
ACK c7376babd1
TheCharlatan:
Re-ACK c7376babd1
hebasto:
re-ACK c7376babd1.
Tree-SHA512: 5bcef16c1255463b1b69270548711e7ff78ca0dd34e300b95e3ca1ce52ceb34f83d9ddb2839e83800ba36b200de30396e504bbb04fa02c6d0c24a16d06ae523d
Set minimum required glibc to 2.31.
The glibc 2.31 branch is still maintained:
https://sourceware.org/git/?p=glibc.git;a=shortlog;h=refs/heads/release/2.31/master.
Remove the stack-protector check from test-security-check, as the test
no-longer fails, and given the control we have of the end, the actual
security-check test seems sufficient (this might also be applied to some
of the other checks).
Drops runtime support for Ubuntu Bionic 18.04 and RHEL-8 from the release binaries.
429ec1aaaa refactor: Rename CTransaction::nVersion to version (Ava Chow)
27e70f1f5b consensus: Store transaction nVersion as uint32_t (Ava Chow)
Pull request description:
Given that the use of a transaction's nVersion is always as an unsigned int, it doesn't make sense to store it as signed and then cast it to unsigned everywhere it is used and displayed.
Since a few alternative implementations have recently been revealed to have made an error with this signedness that would have resulted in consensus failure, I think it makes sense for us to just make this always unsigned to make it clear that the version is treated as unsigned. This would also help us avoid future potential issues with signedness of this value.
I believe that this is safe and does not actually change what transactions would or would not be considered both standard and consensus valid. Within consensus, the only use of the version in consensus is in BIP68 validation which was already casting it to uint32_t. Within policy, although it is used as a signed int for the transaction version number check, I do not think that this change would change standardness. Standard transactions are limited to the range [1, 2]. Negative numbers would have fallen under the < 1 condition, but by making it unsigned, they are still non-standard under the > 2 condition.
Unsigned and signed ints are serialized and unserialized the same way so there is no change in serialization.
ACKs for top commit:
maflcko:
ACK 429ec1aaaa 🐿
glozow:
ACK 429ec1aaaa
shaavan:
ACK 429ec1aaaa💯
Tree-SHA512: 0bcd92a245d7d16c3665d2d4e815a4ef28207ad4a1fb46c6f0203cdafeab1b82c4e95e4bdce7805d80a4f4a46074f6542abad708e970550d38a00d759e3dcef1
In order to ensure that the change of nVersion to a uint32_t in the
previous commit has no effect, rename nVersion to version in this commit
so that reviewers can easily spot if a spot was missed or if there is a
check somewhere whose semantics have changed.
fa52e13ee8 test: Remove struct.pack from almost all places (MarcoFalke)
fa826db477 scripted-diff: test: Use int.to_bytes over struct packing (MarcoFalke)
faf2a975ad test: Use int.to_bytes over struct packing (MarcoFalke)
faf3cd659a test: Normalize struct.pack format (MarcoFalke)
Pull request description:
`struct.pack` has many issues:
* The format string consists of characters that may be confusing and may need to be looked up in the documentation, as opposed to using easy to understand self-documenting code.
This lead to many test bugs, which weren't hit, which is fine, but still confusing. Ref: https://github.com/bitcoin/bitcoin/pull/29400, https://github.com/bitcoin/bitcoin/pull/29399, https://github.com/bitcoin/bitcoin/pull/29363, fa3886b7c6, ...
Fix all issues by using the built-in `int` helpers `to_bytes` via a scripted diff.
Review notes:
* For `struct.pack` and `int.to_bytes` the error behavior is the same, although the error messages are not identical.
* Two `struct.pack` remain. One for float serialization in a C++ code comment, and one for native serialization.
ACKs for top commit:
achow101:
ACK fa52e13ee8
rkrux:
tACK [fa52e13](fa52e13ee8)
theStack:
Code-review ACK fa52e13ee8
Tree-SHA512: ee80d935b68ae43d1654b047e84ceb80abbd20306df35cca848b3f7874634b518560ddcbc7e714e2e7a19241e153dee64556dc4701287ae811e26e4f8c57fe3e
For example:
```bash
ADDITIONAL_GUIX_COMMON_FLAGS set in the ENV
ADDITIONAL_GUIX_ENVIRONMENT_FLAGS="--emulate-fhs" ./contrib/guix/guix-build
<snip>
INFO: Building f75199182133 for platform triple x86_64-linux-gnu:
...using reference timestamp: 1716905119
...running at most 10 jobs
...from worktree directory: '/bitcoin'
...bind-mounted in container to: '/bitcoin'
...in build directory: '/bitcoin/guix-build-f75199182133/distsrc-f75199182133-x86_64-linux-gnu'
...bind-mounted in container to: '/distsrc-base/distsrc-f75199182133-x86_64-linux-gnu'
...outputting in: '/bitcoin/guix-build-f75199182133/output/x86_64-linux-gnu'
...bind-mounted in container to: '/outdir-base/x86_64-linux-gnu'
ADDITIONAL FLAGS (if set)
ADDITIONAL_GUIX_COMMON_FLAGS: --no-substitutes
ADDITIONAL_GUIX_ENVIRONMENT_FLAGS: --emulate-fhs
ADDITIONAL_GUIX_TIMEMACHINE_FLAGS:
```
Adjust the security check for:
ld64.lld: warning: Option `-allow_stack_execute' is not yet implemented.
ld64.lld: error: -fixup_chains is incompatible with -no_pie
and to account for the embedding of LLVMs version number.
b59a027d95 contrib: drop dead get_machine from test sym check (fanquake)
e6aba463ad contrib: use env_flags in get_arch (fanquake)
Pull request description:
This isn't an issue right now (because the get_arch check is simple), but becomes one as soon as we want to use `lld` for linking, and need LDFLAGS (otherwise we call `ld` and fail, see it's usage in #21778). So I've split this out for review. It also makes sense to use the same flags for all compilation in these checks.
Also drops some dead code in test-symbol-check.
ACKs for top commit:
TheCharlatan:
ACK b59a027d95
Tree-SHA512: d8afc4144815369aae63cf6dc6e983af46f208c7043d6ea5c9c811152649c256a8e67eb6864ea9d385d87b6b049fece07710a84b90da325da7fc3f05efcaacd6
7f5ac4520d build: swap otool for (llvm-)objdump (fanquake)
Pull request description:
This tool is used in GUI packaging on macOS, and also somewhat of a blocker for #21778. The main issue is that some distros don't really ship this tool in a standard ways, i.e Ubuntu only ships `llvm-otool` with a version suffix, i.e `llvm-otool-17`, which makes it hard to find and use. Rather than trying to deal with that mess, switch to using the equivalent LLVM tool (objdump), which is a drop-in replacement.
ACKs for top commit:
TheCharlatan:
ACK 7f5ac4520d
theuni:
ACK 7f5ac4520d. Tested `make deploy` on native macOS. Looks good.
hebasto:
ACK 7f5ac4520d.
Tree-SHA512: ac978043f14fb448010542a4a7ce8c6c74b4cbd90f83b4cb4d0bff55974010f10a70b5354f65b239a8bd961d7a3aca22ca165b42954ca87879b9e0524db5f879
6abe772a17 contrib: Add asmap-tool (Fabian Jahr)
Pull request description:
This adds `asmap.py` and `asmap-tool.py` from sipa's `nextgen` branch: https://github.com/sipa/asmap/tree/nextgen
The motivation is that we should maintain the tooling for de- and encoding asmap files within the bitcoin core repository because it is not possible to use an asmap file that is not encoded.
We already had an earlier version of `asmap.py` within the seeds contrib tools. The newer version only had a small amount of changes and is still compatible, so the old version is removed from contrib/seeds and the new version is made available to `makeseeds.py`.
ACKs for top commit:
virtu:
ACK [6abe772](6abe772a17)
0xB10C:
ACK 6abe772a17
achow101:
ACK 6abe772a17
brunoerg:
ACK 6abe772a17
Tree-SHA512: cc2a82ffa4eb46fa0ce4ca769dd82f8d0d2f37fc3652aa748eeb060e1142f9da4035008fe89433e2fd524a4dc153b7b9c085748944b49137b37009b0c0be8afb
Similar to libtool, (llvm-)otool only exists with a version suffix
on some systems (Ubuntu), which makes it annoying to use/find. Avoid
this, by switching to objdump. Which is a drop-in replacement.
This is related to #21778, and the switchover to using vanilla LLVM for
macOS.
