fac012c726 ci: Temporarily disable bpfcc-tools (MarcoFalke)
Pull request description:
This works around package install errors, such as https://github.com/bitcoin/bitcoin/runs/23354020361. Should be possible to reproduce locally via `apt update && apt install bpfcc-tools` on noble:
```
python3-bpfcc : Depends: libbpfcc (>= 0.29.1+ds-1ubuntu4) but it is not going to be installed
ACKs for top commit:
hebasto:
ACK fac012c726, I have reviewed the code, it looks OK. And CI is green.
TheCharlatan:
ACK fac012c726
Tree-SHA512: 369e89bc0fbf4d75455c7c047ae904849267f82e647eeadd7ee6045393bd9812a5e682a58eb8d2ccd8c89a030e6665f054596fdcd0708f591ed4c693398125d1
80f8b92f4f remove libbitcoinconsensus (fanquake)
Pull request description:
This was deprecated in `v27.0`, for removal in `v28.0`. See discussion in PR #29189.
ACKs for top commit:
theuni:
Concept ACK and light review ACK 80f8b92f4f. My only hesitation here is that (afaics?) there's now nothing keeping undesired features like threading or globals from working their way into the interpreter in future commits.
m3dwards:
Concept ACK 80f8b92f4f
TheCharlatan:
ACK 80f8b92f4f
hebasto:
ACK 80f8b92f4f, I have reviewed the code and it looks OK.
Tree-SHA512: 17a62118aeb088f2695c892bb32794dfea3061e3cb7d9e8e9f1c06c3ff6f63a7587fa532e37edbb91fbc5a19b12c9a0f8e05fa9e8864aa07f92665375d847e80
fa75220ac5 ci: Use clang-18 in asan/fuzz/tsan task (MarcoFalke)
fad23a0646 ci: Bump clang+llvm in i686_multiprocess task (MarcoFalke)
Pull request description:
Use clang-18, which comes with bugfixes and sanitizer upgrades.
ACKs for top commit:
fanquake:
ACK fa75220ac5
Tree-SHA512: da251eb55d3a7e537110b03fb0fc10fc6e2f4406d55bdb33bdc91459daabfd3b8d1d20859a66718bb1b09126e11495f6584c52504a85eb09bde9eafafdaf0b0b
6c2990416e ci: Pull in qtbase5-dev instead of seperate low-level libraries (laanwj)
a3c6a13cb2 doc: Suggest installing dev packages for debian/ubuntu qt5 build (laanwj)
Pull request description:
Pretty much all library packages were renamed in the 64-bit time_t migration to add `t64` (even on 64-bit platforms).
Instead of complicating the doc with conditional package names, suggest installing the `-dev` packages which still have the same name, and besides that, are the right way to go about it as they contain the "user facing" C++ headers needed to build against Qt5. They pull in the necessary library packages through dependencies.
For Fedora, devel packages are already suggested.
This affects Ubuntu 24.04 and Debian Testing.
ACKs for top commit:
maflcko:
lgtm ACK 6c2990416e
hebasto:
ACK 6c2990416e.
Tree-SHA512: dae21b7d08fdb221b7b72c323fdaaa6d1a8b014f90e24d2beae64b1ae229fdbeb93d726a61e826447813e52a3cb9775426aefc6c44a4c5ccc541afabb89cb135
601edd8ee8 ci: use codespell 2.2.6 (fanquake)
52fa0d285f doc: fix some typos (crazeteam)
b5ed13a240 doc: Fix typos (RoboSchmied)
Pull request description:
Combines the recent PRs to fix typos so they can be merged.
ACKs for top commit:
brunoerg:
crACK 601edd8ee8
tdb3:
crACK 601edd8ee8
kristapsk:
cr utACK 601edd8ee8
Tree-SHA512: d054b1dad1336d6b9291cc5d5252d4debf6424a993d4edd6a97d7c15055a7fc48a333d30967f72e7dc9c6c1d9a9038ca8bb5e219c529f4c2365ea48404a508d0
fae70ba00d ci: Better tidy errors (MarcoFalke)
Pull request description:
Currently tidy errors are not nice, because the user may have to scroll up to see them in a large block of text. See for example (before) https://github.com/bitcoin/bitcoin/runs/19670551485
Fix that by `tee`ing the output to a file and summarizing the errors in the end again. See for example (after): https://github.com/bitcoin/bitcoin/runs/22647850662
ACKs for top commit:
hebasto:
ACK fae70ba00d, logs with errors look cleaner.
TheCharlatan:
ACK fae70ba00d
Tree-SHA512: dcaea557fed40089409d16ce2cbaa8a9cfbf047f601d5daadfee0823b0eed7badc12d803addc0b7b6bb3f1eaf5c787fccb2488475d32c4efd80835f386f761dd
64722e4359 ci: Drop `--enable-c++20` option (Hennadii Stepanov)
Pull request description:
This option has ceased to exist since https://github.com/bitcoin/bitcoin/pull/28349.
ACKs for top commit:
maflcko:
ACK 64722e4359
Tree-SHA512: bd392c331f775605615e1b236682269b83a1e6363a4d3f09c4d8d54495cf3d22973a921ebf6b8a9f813ba6c024d3324761f3291aaf7f473995f5eaa4c195bc43
636c9862cf ci: Bump `TIDY_LLVM_V` (Hennadii Stepanov)
Pull request description:
This PR switches to the latest [IWYU 0.22](https://github.com/include-what-you-use/include-what-you-use/releases/tag/0.22), which is compatible with Clang 18.
ACKs for top commit:
fanquake:
ACK 636c9862cf
Tree-SHA512: 78ce89244c5e487dd1be8b4bd2ca6f06d19b04b78289ebc21985110574053545dcce5eb622edf2bede2cf7bb58360170e976d30a4484a127d34dd17b1c604e9c
acc06bc91f ci, macos: Use `--break-system-packages` with Homebrew's python (Hennadii Stepanov)
ae5f72027f ci: Add workaround for Homebrew's python link error (Hennadii Stepanov)
Pull request description:
Homebrew [promoted](https://github.com/Homebrew/homebrew-core/pull/150390) `python@3.12` to the default `python3`. Now, our "macOS native" CI job is facing the following issues:
1. Installing `qt@5` [requires](https://github.com/bitcoin/bitcoin/actions/runs/8216848118/job/22471875454#step:4:51) re-installing `python@3.12`:
```
==> Fetching dependencies for qt@5: readline, python@3.12 and gettext
```
2. Re-installing `python@3.12` [fails](https://github.com/bitcoin/bitcoin/actions/runs/8216848118/job/22471875454#step:4:127) due to symbolic link conflicts on macOS `x86_64`:
```
==> Pouring python@3.12--3.12.2_1.ventura.bottle.tar.gz
Error: The `brew link` step did not complete successfully
```
3. Homebrew's `python@3.12` is marked as externally managed (according to PEP 668), necessitating different approaches for installing Python packages.
This pull request resolves all the issues mentioned above.
ACKs for top commit:
m3dwards:
reACK acc06bc91f to get the CI passing again.
Tree-SHA512: 82cf72bff328f1e0725342431ac14ad4bae7a758186d97db6c7a558e4b661dcbf3fabe536978e26e709c5f6f7f5c11aac46642634c6685f1291592d8d825ad87
fa58ae74ea refactor: Add missing include for USE_BDB, USE_SQLITE to bench/wallet_ismine.cpp (MarcoFalke)
fa31908ea8 lint: Check for missing or redundant bitcoin-config.h includes (MarcoFalke)
fa63b0e351 lint: Make lint error easier to spot in output (MarcoFalke)
fa770fd368 doc: Add missing RUST_BACKTRACE=1 (MarcoFalke)
fa10051267 lint: Add get_subtrees() helper (MarcoFalke)
Pull request description:
Missing `bitcoin-config.h` includes are problematic, because the build could silently pass, but produce an unintended result. For example, a slower fallback algorithm could be picked, even though `bitcoin-config.h` indicates that a faster feature is available and should be used.
As the build succeeds silently, this problem is not possible to detect with iwyu.
Thus, fix this by using a linter based on grepping the source code.
ACKs for top commit:
theuni:
Weak ACK fa58ae74ea.
TheCharlatan:
ACK fa58ae74ea
hebasto:
ACK fa58ae74ea, tested on Ubuntu 23.10 -- it catches bugs properly. I didn't review rust code changes.
Tree-SHA512: cf4346f81ea5b8c215da6004cb2403d1aaf569589613c305d8ba00329b82b3841da94fe1a69815ce15f2edecbef9b031758ec9b6433564976190e3cf91ec8181
Drop `git diff` command so it is easier to run CI locally if git checkout is a
worktree. Currently it fails because the directory is not recognized as a git
repository.
The `git diff` command was added recently in #28359 commit
fa07ac48d8 and can be avoided just by teeing the
patch to stdout
540282905d docs: ci multi-arch requires qemu (Max Edwards)
Pull request description:
On a fresh Debian system qemu isn't installed and therefore the multi-architecture CI system doesn't run.
This documentation notes that qemu is required and how to install it.
ACKs for top commit:
maflcko:
lgtm ACK 540282905d
willcl-ark:
utACK 540282905d
Tree-SHA512: 903c12cf7b16f2146b99a952577c5550d60faf65f9e72b9f4d9479b52228118ab46349e5130de5281b39da05d3bc0b4ae8a8165601e62ce145647a98ef197131
fa91bf2559 ci: Skip git install if it is already installed (MarcoFalke)
c65fde4831 ci: vary /tmp/env (Sjors Provoost)
Pull request description:
* Currently, running separate CI tasks at the same time may intermittently fail, because they race to read/write `/tmp/env`. Fix this by adding `$CONTAINER_NAME` to the file name.
* Also, add `$USER`, while touching the line, to allow different users to run the same CI task at the same time.
* Also, skip the git install if there is no need.
Ref: https://github.com/bitcoin/bitcoin/pull/29274
ACKs for top commit:
Sjors:
ACK fa91bf2559
BrandonOdiwuor:
ACK fa91bf2559
hebasto:
ACK fa91bf2559.
Tree-SHA512: 9a8479255a2afb6618f9d0796488d9430ba95266b90ce39536a9817c1974ca4049beeaab5355a38b25171f76fc386dbec06b1919aaa079f08a5a0c0a146232c8
4756114e50 [depends] Allow PATH with spaces in directory names. (Mark Friedenbach)
Pull request description:
The goal of this PR is to help close https://github.com/bitcoin/bitcoin/pull/28733. I reverted the change on `depends/config.guess` based on the feedback provided in the previous PR. I've also incorporated the test mentioned by maflcko
ACKs for top commit:
maflcko:
lgtm ACK 4756114e50
hebasto:
ACK 4756114e50, successfully built depends on Ubuntu 22.04.
TheCharlatan:
ACK 4756114e50
Tree-SHA512: ee257f6efd235839156bc236384f08d77b91debc3c257168368a71e70742639f28a3289572b8693609c1109062dc9968e461103d1f4f5679906506e94b54e649
aaaace2fd1 fuzz: Assume presence of __builtin_*_overflow, without checks (MarcoFalke)
fa223ba5eb Revert "build: Fix undefined reference to __mulodi4" (MarcoFalke)
fa7c751bd9 build: Bump clang minimum supported version to 14 (MarcoFalke)
Pull request description:
Most supported operating systems ship with clang-14 (or later), so bump the minimum to that and allow new code to drop workarounds for previous clang bugs.
For reference:
* https://packages.debian.org/bookworm/clang (`clang-14`)
* https://packages.ubuntu.com/jammy/clang (`clang-14`)
* CentOS-like 8/9 Stream: All Clang versions from 15 to 17
* FreeBSD 12/13: All Clang versions from 15 to 16
* OpenSuse Tumbleweed ships with https://software.opensuse.org/package/clang (`clang17`); No idea about OpenSuse Leap
On operating systems where the clang version is not shipped by default, the user would have to use GCC, or install clang in a different way. For example:
* https://packages.debian.org/bullseye/g++ (g++-10)
* https://packages.ubuntu.com/focal/g++-10
* https://apt.llvm.org/, or nix, or guix, or compile clang from source, ...
ACKs for top commit:
fanquake:
ACK aaaace2fd1
Tree-SHA512: 81d066b14cc568d27312f1cc814b09540b038a10a0a8e9d71fc9745b024fb6c32a959af673e6819b817ea7cef98da4abfa63dff16cffb7821b40083016b0291f
This is already used in multiple CIs, and will soon become a requirement
for most CIs, i.e when we migrate depends packages to use CMake, for
example:
https://github.com/bitcoin/bitcoin/pull/21778#issuecomment-1885576324.
Some of the CIs in 21778 are failing because CMake isn't available, so
just break this out and make CMake globally available.
The `ALLOW_HOST_PACKAGES` variable was introduced in bitcoin#10508 "to
speed up build and avoid timeout".
It is no longer the case for our CI infrastructure, which uses self-
hosted persistent workers and depends caching.
In the current circumstances, it does not seem worth porting this
feature to the upcoming CMake-based build system.
308aec3e56 build: disable external-signer for Windows (fanquake)
35537318a1 ci: remove --enable-external-signer from win64 job (fanquake)
Pull request description:
It's come to light that Boost ASIO (a Boost Process sub dep) has in some
instances, been quietly initialising our network stack on Windows (see
PR https://github.com/bitcoin/bitcoin/pull/28486 and discussion in https://github.com/bitcoin/bitcoin/issues/28940).
This has been shielding a bug in our own code, but the larger issue
is that Boost Process/ASIO is running code before main, and doing things
like setting up networking. This undermines our own assumptions about
how our binary works, happens before we run any sanity checks,
and before we call our own code to setup networking. Note that ASIO also
calls WSAStartup with version `2.0`, whereas we call with `2.2`.
It's also not clear why a feature like external signer would have a
dependency that would be doing anything network/socket related,
given it only exists to spawn a local process.
See also the discussion in https://github.com/bitcoin/bitcoin/issues/24907. Note that the maintaince of Boost Process in general,
has not really improved. For example, rather than fixing bugs like https://github.com/boostorg/process/issues/111,
i.e, https://github.com/boostorg/process/pull/317, the maintainer chooses to just wrap exception causing overflows
in try-catch blocks: 0c42a58eac. These changes get merged in large,
unreviewed PRs, i.e https://github.com/boostorg/process/pull/319.
This PR disables external-signer on Windows for now. If, in future, someone
changes how Boost Process works, or replaces it entirely with some
properly reviewed and maintained code, we could reenable this feature on
Windows.
ACKs for top commit:
hebasto:
re-ACK 308aec3e56.
TheCharlatan:
ACK 308aec3e56
Tree-SHA512: 7405f7fc9833eeaacd6836c4e5b1c1a7845a40c1fdd55c1060152f8d8189e4777464fde650e11eb1539556a75dddf49667105987078b1457493ee772945da66e
8ea45e626e build: use macOS 14 SDK (Xcode 15.0) (fanquake)
51c97ffb69 build: patch boost process for macOS 14 SDK (fanquake)
423949a13b depends: add -platform_version to macOS build flags (fanquake)
Pull request description:
This fixes: https://github.com/bitcoin/bitcoin/pull/28349#issuecomment-1748515277 (cross-compiling with C++20 for macOS). See https://developer.apple.com/xcode/cpp/#c++20 for C++20 support in Apples libc++, some features landed with Xcode 14.3, although many more landed with Xcode 15.0.
ACKs for top commit:
hebasto:
ACK 8ea45e626e.
TheCharlatan:
ACK 8ea45e626e
Tree-SHA512: 274ce2c9b9f8e4d755c07b8d0d4897a7f92708ac64e6afb7a3f75bdb485e863fc7f40badf3a88b129ce36f6cca72f768dc2ed7fba2bdf0bb6da2bf0c8fedee10
fad2392c58 ci: Use Ubuntu 24.04 Noble for asan (MarcoFalke)
fa83b65ef8 ci: Use Ubuntu 24.04 Noble for tsan,tidy,fuzz (MarcoFalke)
Pull request description:
23.10 will be EOL mid next year, so a bump is needed before then for the `master` branch (and possibly the `26.x` branch).
Doing the bump now is fine, because the clang version is pinned to 17 inside the CI tasks. So a default clang version change in the system image should not affect the tasks. Once clang-18 is available and the default in April next year (https://discourse.ubuntu.com/t/noble-numbat-release-schedule/35649#planned-and-potentially-disruptive-archive-wide-activities-2), the pinned version could be bumped (for CI tasks that require a pin, like tidy), or the pin can be removed (for CI tasks that usually do not require a pin, like fuzz or the sanitizers).
ACKs for top commit:
fanquake:
ACK fad2392c58
Tree-SHA512: c40aede4e2281a5d539d5f65d2c08a57bf92e4a00b4f45a4260b57b7443a63d1a0603115da4a3bbd100ac5f6ade3f2eda0916e4b565573741162a76294ec0ac5
It is confusing to treat commands as a single string. This change is
also required to support paths and strings with spaces in them in the
future.
This requires replacing TEST_RUNNER_ENV with a global export, because it
no longer works. See:
```bash
$ export ENV="A=1" && $ENV ls
bash: A=1: command not found...
```
Or in the CI task:
+ DIR_UNIT_TEST_DATA=/ci_container_base/ci/scratch/qa-assets/unit_test_data/
+ LD_LIBRARY_PATH=/ci_container_base/depends/i686-pc-linux-gnu/lib
+ BITCOIND=bitcoin-node make -j10 check VERBOSE=1
/ci_container_base/ci/test/03_test_script.sh: line 166: BITCOIND=bitcoin-node: command not found
https://github.com/bitcoin/bitcoin/pull/28954/checks?check_run_id=19096858944https://cirrus-ci.com/task/6718317604372480
fa01f884d3 ci: Add missing COPY for ./test/lint/test_runner (MarcoFalke)
faff3e3b46 lint: Report all lint errors instead of early exit (MarcoFalke)
Pull request description:
`all-lint.py` currently collects all failures. However, the `06_script.sh` does not, since July this year (https://github.com/bitcoin/bitcoin/pull/28103#discussion_r1268115806).
Fix this by printing all failures before exiting.
Can be tested by modifying (for example) two subtrees in the same commit and then running the linters.
ACKs for top commit:
kevkevinpal:
ACK [fa01f88](fa01f884d3)
TheCharlatan:
lgtm ACK fa01f884d3
Tree-SHA512: c0f3110f2907d87e29c755e3b77a67dfae1f8a25833fe6ef8f2f2c58cfecf1aa46f1a20881576b62252b04930140a9e416c78b4edba0780d3c4fa7aaebabba81
43de4d3630 doc: fix typos (Sjors Provoost)
Pull request description:
This PR fixes typos found by lint-spelling.py using codespell 2.2.6.
Our CI linter job uses codespell 2.2.5 and found fewer typos that I did locally. In any case it's happy now.
ACKs for top commit:
pablomartin4btc:
re ACK 43de4d3630
Tree-SHA512: c032fe86cb49c924a468385653b31f309a9db68c478d70335bba3e65a1ff3826abe80284fe00a090ab5a509e1edbf17e476f6922fb15d055e50f1103dad2ccb0
bbbbdb0cd5 ci: Add filesystem lint check (MarcoFalke)
fada2f9110 refactor: Replace <filesystem> with <util/fs.h> (MarcoFalke)
Pull request description:
Using `std::filesystem` is problematic:
* There is a `fs` namespace wrapper for it. So having two ways to achieve the same is confusing.
* Not using the `fs` wrapper is dangerous and buggy, because it disables known bugs by deleting problematic functions.
Fix all issues by removing use of it and adding a linter to avoid using it again in the future.
ACKs for top commit:
TheCharlatan:
ACK bbbbdb0cd5
fanquake:
ACK bbbbdb0cd5🦀
Tree-SHA512: 0e2d49742b08eb2635e6fce41485277cb9c40fe20b81017c391d3472a43787db1278a236825714ca1e41c9d2f59913865cfb0c649e3c8ab1fb598c849f80c660
fca0a8938e ci: remove "--exclude banman" for fuzzing in mac (brunoerg)
f9b286353f fuzz: call lookup functions before calling `Ban` (brunoerg)
Pull request description:
Fixes#27924
To not have any discrepancy, it's required to call lookup functions before calling `Ban`. If we don't do it, the assertion `assert(banmap == banmap_read);` may fail because `BanMapFromJson` will call `LookupSubNet` and cause the discrepancy between the banned and the loaded one. It happens especially in MacOS (#27924).
Also, calling lookup functions before banning is what RPC `setban` does.
ACKs for top commit:
maflcko:
lgtm ACK fca0a8938e
dergoegge:
ACK fca0a8938e
Tree-SHA512: a3d635088a556df4507e65542157f10b41d4f87dce42927b58c3b812f262f4544b6b57f3384eef1097ffdd7c32b8dd1556aae201254960cbfbf48d45551200f7
9f208c0171 ci: Switch IWYU to `clang_17` branch (Hennadii Stepanov)
Pull request description:
The IWYU version [0.21](https://github.com/include-what-you-use/include-what-you-use/releases/tag/0.21) has been tagged, and the `clang_17` branch is available now.
ACKs for top commit:
maflcko:
lgtm ACK 9f208c0171
Tree-SHA512: 8b8f8743d1c2719b6383b5a6a48356ac02a301d1ce9cee77f93cc04c12de22e9ac6b59e23550a589540e68292cfac0d85bacedc9ca26f6b589011d36ee1d38cf
It's not completely clear to me why this needs to be explicitly
specified in some environments, and not in others, while at the same time
that `llvm-symbolizer` is already in PATH, but this has fixed the 2 issues
outlined in #28147.
Use `LLVM_SYMBOLIZER_PATH` as the env var, as that is somewhat also used
inside LLVM, but not consistently, i.e it's checked for in the asan_symbolize
script, but not in in the ubsan_symbolize script, or from in compiler-rt.
fa25e8b0a1 doc: Recommend lint image build on every call (MarcoFalke)
faf70c1f33 Bump python minimum version to 3.9 (MarcoFalke)
fa8996b930 ci: Bump i686_multiprocess.sh to latest Ubuntu LTS (MarcoFalke)
Pull request description:
All supported operating systems ship with python 3.9 (or later), so bumping the minimum should not cause any issues. A bump will allow new code to use new python 3.9 features.
For reference:
* https://packages.debian.org/bullseye/python3
* https://packages.ubuntu.com/focal/python3.9
* FreeBSD 12/13 also ships with 3.9
* CentOS-like 8/9 also ships with 3.9 (and 3.11)
* OpenSuse Leap also ships with 3.9 (and 3.11) https://software.opensuse.org/package/python311-base
This is for Bitcoin Core 27.0 in 2024 (next year), not the soon upcoming 26.0 next month.
ACKs for top commit:
Sjors:
ACK fa25e8b0a1
jamesob:
ACK fa25e8b0a1 ([`jamesob/ackr/28211.1.MarcoFalke.bump_python_minimum_supp`](https://github.com/jamesob/bitcoin/tree/ackr/28211.1.MarcoFalke.bump_python_minimum_supp))
Tree-SHA512: 86c9f6ac4b5ba94a62ee6a6062dd48a8295d8611a39cdb5829f4f0dbc77aaa1a51edccc7a99275bf699143ad3a6fe826de426d413e5a465e3b0e82b86d10c32e
Since all bitcoin-ci-test images are now labeled, we can always
prune all dangling images, regardless of whether we are in
RESTART_CI_DOCKER_BEFORE_RUN.
To be safe, still prune all images if RESTART_CI_DOCKER_BEFORE_RUN
in case the filtering doesn't work, or if images were created on
an earlier version that did not assign labels.
This allows us or the user to perform batch operations on all
images produced by the ci, e.g. to prune all dangling images,
without affecting non-ci images.
fa2c894cbb ci: move-only CI_CONTAINER_ID to 02_run_container.sh (MarcoFalke)
fa695b4df0 ci: Work around podman stop bug (MarcoFalke)
fa09a031c1 ci: Add set -ex to 02_run_container.sh (MarcoFalke)
fac9abbf47 ci: Rename 04_install to 02_run_container (MarcoFalke)
Pull request description:
Sometimes, it seems that `podman stop` does not work. Presumably, it falls back to `podman kill`, which is async.
Try to work around this intermittent issue by using the `rm --force` over `stop`.
Example failing log https://cirrus-ci.com/task/4549784611061760?logs=ci#L238:
```
Restart docker before run to stop and clear all containers started with --rm
++ podman container stop --all
e4eca0766f87864d89fc230aa884a238c214cfbcd44cf76a4dbdb2a30c982009
++ echo 'Prune all dangling images'
Prune all dangling images
++ docker image prune --force
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
+++ docker run --cap-add LINUX_IMMUTABLE --rm --interactive --detach --tty --mount type=bind,src=/tmp/cirrus-build-1970593815,dst=/tmp/cirrus-build-1970593815,readonly --mount type=volume,src=ci_macos_cross_ccache,dst=/tmp/ccache_dir --mount type=volume,src=ci_macos_cross_depends,dst=/ci_container_base/depends --mount type=volume,src=ci_macos_cross_previous_releases,dst=/ci_container_base/prev_releases --env-file /tmp/env --name ci_macos_cross ci_macos_cross
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
time="2023-09-27T20:55:39Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Error: creating container storage: the container name "ci_macos_cross" is already in use by e4eca0766f87864d89fc230aa884a238c214cfbcd44cf76a4dbdb2a30c982009. You have to remove that container to be able to reuse that name: that name is already in use
ACKs for top commit:
hebasto:
ACK fa2c894cbb, I have reviewed the code and tested it locally.
Tree-SHA512: 31fca340c6bedaadf4dd51fa745d9b3969042cebc0c7c904ef18af3f2f986039ec4354ccdff1422fbf77cf223e4423857368dce53cfa67ef15c76b78d007eace
78d3062b68 ci: Install Android API 31 platform as Qt expects (Hennadii Stepanov)
Pull request description:
When building the `qt` package, it expects that the default (in Qt's view) Android API platform is installed.
During the recent Qt version [update](https://github.com/bitcoin/bitcoin/pull/28561), it has been changed:
```diff
--- a/mkspecs/features/android/sdk.prf
+++ b/mkspecs/features/android/sdk.prf
@@ -1,6 +1,6 @@
API_VERSION_TO_USE = $$(ANDROID_API_VERSION)
isEmpty(API_VERSION_TO_USE): API_VERSION_TO_USE = $$API_VERSION
-isEmpty(API_VERSION_TO_USE): API_VERSION_TO_USE = android-28
+isEmpty(API_VERSION_TO_USE): API_VERSION_TO_USE = android-31
ANDROID_JAR_FILE = $$ANDROID_SDK_ROOT/platforms/$$API_VERSION_TO_USE/android.jar
!exists($$ANDROID_JAR_FILE) {
```
This PR fixes the CI for the Android task and addresses https://github.com/bitcoin/bitcoin/pull/28561#issuecomment-1749180177.
Qt [docs](https://doc.qt.io/qt-5/android.html) still claim that Android API Level 21 and up are supported, however, I did not test every possible configuration.
NOTE: https://github.com/bitcoin/bitcoin/pull/28611 is still valid.
ACKs for top commit:
maflcko:
lgtm ACK 78d3062b68
jarolrod:
tACK 78d3062b68
Tree-SHA512: 781fba6d80aae7e6500854de14af0d30169c258e395b9e482a5430a7b4a2211a6181f8c9ee58543c896b431abf09e3e7c5573b9672ed128658f11f98a2006e7e
fa3b5e5e57 ci: Use nproc over MAKEJOBS in 01_base_install (MarcoFalke)
Pull request description:
Currently `$MAKEJOBS` is the default value in `01_base_install.sh` when building the container image.
This problem can't be fixed (see below), so just use `nproc` for now.
Other solutions would be bad:
* Passing in the `MAKEJOBS` as a dockerfile env would create a new image if the number of tasks are changed, seems verbose and confusing.
* Leaving `master` as-is would leave CPUs unused if there are more than `4`.
ACKs for top commit:
fanquake:
ACK - I think this is fine as-is, it's an improvement over the current state fa3b5e5e57
Tree-SHA512: 15014eb7b548945737b0fed5cd46f0cd4b72b1d54d044f60fce628f914053a2de6518878428a54822d236d08d6f257d8c464d3fb589400f4be56022d2ec8676d
de8f9123af test: cover read-only blockstore (Matthew Zipkin)
5c2185b3b6 ci: enable chattr +i capability inside containers (Matthew Zipkin)
e573f24202 unit test: add coverage for BlockManager (Matthew Zipkin)
Pull request description:
This PR adds unit and functional tests to cover the behavior described in #2039. In particular, that bitcoind will crash on startup if a reindex is requested but the `blk` files are read-only. Eventually this behavior can be updated with https://github.com/bitcoin/bitcoin/pull/27039. This PR just commits the test coverage from #27039 as suggested in https://github.com/bitcoin/bitcoin/pull/27039#issuecomment-1584915782
ACKs for top commit:
jonatack:
ACK de8f9123af modulo suggestions in https://github.com/bitcoin/bitcoin/pull/27850#discussion_r1319010039, tested on macOS, but not on Linux for the Linux-related change in the last push
achow101:
ACK de8f9123af
MarcoFalke:
lgtm ACK de8f9123af📶
Tree-SHA512: b9bd684035dcea11c901b649fc39f397a2155a9a8459f3348e67947e387e45312fddeccb52981aef486f8a31deebb5356a7901c1bb94b78f82c24192a369af73
fa07ac48d8 ci: Asan with -ftrivial-auto-var-init=pattern (MarcoFalke)
Pull request description:
This makes memory bugs deterministic. `-ftrivial-auto-var-init=pattern` is incompatible with other memory sanitizers (like valgrind and msan), but that is irrelevant here, because the address sanitizer in this fuzz CI config is already incompatible with them.
`-ftrivial-auto-var-init=pattern` goes well with `-fsanitize=bool` and `-fsanitize=enum`, but those are already enabled via `-fsanitize=undefined`. See https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#available-checks
ACKs for top commit:
fanquake:
ACK fa07ac48d8 - going to get back to fixing up the cxxflags usage in CI, but not a blocker here:
Tree-SHA512: 2ea6c5128a9cd262bdd1b1475c7e1be23ce2c520fad05f6c2aace6c29e203573323c0564d272e25da35ad5ff46fde488a5eae1ed14d3349e793d14a5e2533fb1
fa70cbd969 ci: Remove unused TEST_RUNNER_ENV="LC_ALL=C" from s390x task (MarcoFalke)
fa33354dcb ci: Remove /ro_base bind mount (MarcoFalke)
fa0df9d4c4 doc: Remove sudo from command that is already run as root (MarcoFalke)
Pull request description:
Remove some CI stuff no longer needed.
ACKs for top commit:
fanquake:
ACK fa70cbd969 - did not test the s390x job.
Tree-SHA512: 3a6ed0cfc855a92c2f834e59494c0a19a5647510247aece5e40a1aa78074894fe7454e684a1ea1f8f0662c50ac1caf2e390398b0fcfbf81544e6488fa9b8915e
fab7f5c01d ci: Add missing docker.io prefix to CI_IMAGE_NAME_TAG (MarcoFalke)
Pull request description:
Currently, the CI system may pick the wrong (non-native) architecture due to the missing prefix.
For example, assuming the CI_IMAGE_NAME_TAG is `debian:bookworm` and the user has previously pulled an s390x image:
```
$ podman run --rm 'docker.io/s390x/debian:bookworm' dpkg --print-architecture
exec /usr/bin/dpkg: exec format error
```
Now, `debian:bookworm` will refer to the same image:
```
$ podman run --rm 'debian:bookworm' dpkg --print-architecture
exec /usr/bin/dpkg: exec format error
```
However, `docker.io/debian:bookworm` works fine:
```
$ podman run --rm 'docker.io/debian:bookworm' dpkg --print-architecture
arm64
ACKs for top commit:
hebasto:
ACK fab7f5c01d.
Tree-SHA512: c423c5cd454a95fa3e67081411ca08d316b8c680a5bba49196c514b91df65d9cc46a47700cc00d9579327842615f98146d0ac50abb016616a9b17d042598dab6
fa56d17a4b ci: Add missing amd64 to win64-cross task (MarcoFalke)
Pull request description:
Currently the task will fail if run on non-`x86_64`.
Fix this by adding the missing `amd64`, similar to
7bf078f2b7/ci/test/00_setup_env_i686_multiprocess.sh (L11)
ACKs for top commit:
hebasto:
ACK fa56d17a4b
Tree-SHA512: faab1c5b945283b7e8d080bbcc8e9379c480cf6973506149ace5990cb4d04673f83f4bc36d08d5b4e9cb17a86fdbe23ac97ef4eab0e842616b367b8138229c58
This should fix the macOS-cross build on Cirrus CI containers.
Locally this was already working, because the SDK was cached in
/ci_container_base/ in the image, which is also the folder used for a
later CI run.
However, on Cirrus CI, when using an image *and* a custom BASE_ROOT_DIR,
the SDK will not be found in /ci_base_install/, nor in BASE_ROOT_DIR.
Fix this by normalizing *all* folders to /ci_container_base/.
9658d0dc17 ci: Run "macOS native x86_64" job on GitHub Actions (Hennadii Stepanov)
Pull request description:
From https://github.com/bitcoin/bitcoin/issues/28098:
> Thus, someone would have to sponsor an amount of roughly 5kUSD/mo for those two tasks.
> If the goal is to stay on a free plan, I think the only option is GitHub Actions CI.
---
**IMPORTANT NOTE**. We currently ship macOS release binaries for both architectures: `x86_64` and `arm64`. If this PR gets merged, only `x86_64` architecture will be tested on CI, which implies some [drawbacks](https://github.com/bitcoin/bitcoin/pull/28187#issuecomment-1658077549).
However, it has never been the case that our CI tested both architectures simultaneously. And we hope that GitHub Actions will soon host macOS `arm64` runners.
Historically, we moved from `x86_64` to `arm64` in https://github.com/bitcoin/bitcoin/pull/26388 less than a year ago.
---
Security concerns:
- https://github.com/bitcoin/bitcoin/issues/28098#issuecomment-1651432106
- https://github.com/bitcoin/bitcoin/issues/28098#issuecomment-1651688197
`GITHUB_TOKEN` permissions (from the build log in my personal repo):
```
2023-07-27T07:30:17.8313534Z ##[group]GITHUB_TOKEN Permissions
2023-07-27T07:30:17.8314113Z Contents: read
2023-07-27T07:30:17.8314608Z Metadata: read
2023-07-27T07:30:17.8314957Z Packages: read
2023-07-27T07:30:17.8315233Z ##[endgroup]
```
Comparison of resources:
| Resource | Current, Cirrus CI | Suggested, GitHub Actions |
|---|:-:|:-:|
| CPU | 4 | 4 \*\* |
| RAM, GB | 8 | 14 |
**\*\* NOTE**: However, [docs](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources) are mentioning:
> 3-core CPU (x86_64)
ACKs for top commit:
MarcoFalke:
re-ACK 9658d0dc17🏂
achow101:
ACK 9658d0dc17
jarolrod:
ACK 9658d0dc17
Tree-SHA512: 6123e68e6784cdf4e53c3e77b435709261db21f09091af2c22e667d3816a305fffb9d617297a5bc1bda18aaba84a6e210cec6a75c52afa7746a3780a67b69865
1c976c691c tidy: Integrate bicoin-tidy clang-tidy plugin (fanquake)
7de23cceb8 refactor: fix unterminated LogPrintf()s (fanquake)
0a1029aa29 lint: remove /* Continued */ markers from codebase (fanquake)
910007995d lint: remove lint-logs.py (fanquake)
d86a83d6b8 lint: drop DIR_IWYU global (fanquake)
Pull request description:
Demo of integrating the [bitcoin-tidy](https://github.com/theuni/bitcoin-tidy-plugin), [clang-tidy plugin](https://clang.llvm.org/extra/clang-tidy/) written by theuni into our tidy CI job.
The plugin currently has a single check, `bitcoin-unterminated-logprintf`. This would replace our current Python driven, `git-grep`-based, `.cpp` file only, lint-logs linter.
ACKs for top commit:
TheCharlatan:
ACK 1c976c691c
theuni:
ACK 1c976c691c
MarcoFalke:
re-ACK 1c976c691c👠
Tree-SHA512: 725b45c70e431d48e6f276671e05c694e10b6047cae1a31906ac3ee9093bc8105fb226b36a5bac6709557526ca6007222112d66aecec05a574434edc4897e4b8
This change aims to:
1) Remove our own `CCACHE_SIZE` environment variable that violates
Ccache's `CCACHE_*` namespace.
2) Introduce the `CCACHE_MAXSIZE` environment variable that is
documented since v3.3, which makes its usage consistent with other ones,
such as `CCACHE_DIR` and `CCACHE_NOHASHDIR`.
fabc04a4d9 ci: Keep system env vars as-is (MarcoFalke)
fa8dcdcc8b ci: Set PATH inside the CI env (MarcoFalke)
fac229ab1f ci: Remove P_CI_DIR and --workdir (MarcoFalke)
Pull request description:
This fixes a bug where the `$PATH` from the host is used inside the container. This will lead to bugs when the `$PATH` is different. For example on a host of Fedora 38, and a container of `debian:bullseye`.
This can be tested with the `FILE_ENV=./ci/test/00_setup_env_arm.sh` CI env. On master:
```
Error: crun: executable file `bash` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found
```
On this pull:
(everything passes)
ACKs for top commit:
TheCharlatan:
lgtm ACK fabc04a4d9
Tree-SHA512: 51d2affed91624d0e5b43a3ee1e696313f934e05fde6b5a19e8ac4e6666c1e7b667a444bf3de3f77190bcd00e81efb7576196afb0f6b5e788d1a50e7ddb28d7e
08eb5f1b67 ci: document that -Wreturn-type has been fixed upstream (Windows) (fanquake)
Pull request description:
`noreturn` attributes have been added to the mingw-w64 headers, 1690994f51, meaning that [from 11.0.0 onwards](https://www.mingw-w64.org/changelog/), you'll no-longer see `-Wreturn-type` warnings when using `assert(false)`.
Add -Wno-return-type to the Windows CI, where is should have been all
along, and document why it's required. This can be dropped when we are
using the fixed version of the mingw-w64 headers there.
Drop the -Werror -Wno-return-type special case from our build system.
-Wreturn-type is on by default in Clang and GCC.
The new mingw-w64 header behaviour can be checked on Ubuntu mantic, [which ships with 11.0.0](https://packages.ubuntu.com/mantic/mingw-w64), using:
```cpp
#include <cassert>
int f(){ assert(false); }
int main() {
return 0;
}
```
On Mantic (with 11.0.0):
```bash
x86_64-w64-mingw32-g++ test.cpp -Wreturn-type
# nada
```
On Lunar ([with 10.0.0](https://packages.ubuntu.com/lunar/mingw-w64)):
```bash
x86_64-w64-mingw32-g++ test.cpp -Wreturn-type
test.cpp: In function 'int f()':
test.cpp:3:25: warning: no return statement in function returning non-void [-Wreturn-type]
3 | int f(){ assert(false); }
| ^
```
ACKs for top commit:
TheCharlatan:
ACK 08eb5f1b67
Tree-SHA512: 9cd4310a96abd87bf8ceb37949ad0259fe4adee3367c604f4c4ad521a0cf09bdcc5dd305db19a0f45ce74c85178b0d739e2fca5ad0fc841ac935523a23b28a7f
This is needed for the next commit.
This also requires dropping CI_RETRY from the docker build step, which
is fine, because CI_RETRY should be called inside the build script, not
outside.
Also, fix a doc typo.
(Similar to the doc comment in ci/lint_imagefile)
Also, rename docker-entrypoint.sh to container-entrypoint.sh
Also, add copyright header to touched files.
`noreturn` attributes have been added to the mingw-w64 headers, meaning
that from 11.0.0 onwards, you'll no-longer see `-Wreturn-type` warnings
when using assert(false):
1690994f51.
Add -Wno-return-type to the Windows CI, where is should have been all
along, and document why it's required. This can be dropped when we are
using the fixed version of the mingw-w64 headers there.
Drop the -Werror -Wno-return-type special case from our build system.
-Wreturn-type is on by default in Clang and GCC.
6c97757a48 script: appease spelling linter (Jon Atack)
1316119ce7 script: update ignored-words.txt (Jon Atack)
146c861da2 script: update linter dependencies (Jon Atack)
92408224a4 test: fix PEP484 no implicit optional argument types errors (Jon Atack)
f86a301433 script, test: add missing python type annotations (Jon Atack)
Pull request description:
With these updates, `./test/lint/lint-python.py` and `./test/lint/lint-spelling.py` should be green again for developers using relatively recent Python dependencies, in particular mypy 0.991 (released 11/2022) and later. Please see the commit messages for details.
ACKs for top commit:
fanquake:
ACK 6c97757a48
Tree-SHA512: 8a46a4d36d5978affdcecf4f2ace20ca1b52d483e098304911a2169afe60ccb9b042fa90c04b762d94f3ce53d2cafe6f24476ae839867a770c7f31e7e7242d99
These work for me now. If they still don't work in other setups,
maybe we can better document the issues.
```bash
time FILE_ENV="./ci/test/00_setup_env_s390x.sh" ./ci/test_run_all.sh
...
Running tests: coins_tests from test/coins_tests.cpp
PASS: qt/test/test_bitcoin-qt
Running tests: coinstatsindex_tests from test/coinstatsindex_tests.cpp
...
Stop and remove CI container by ID
+ docker container kill 617bef8accb87530e5fbb03ff07b3b9f0aa9e3030d4da424c9612d153ab98dbf
617bef8accb87530e5fbb03ff07b3b9f0aa9e3030d4da424c9612d153ab98dbf
real 51m37.809s
```
fae7c50d20 test: Run fuzz tests on macOS (MarcoFalke)
Pull request description:
Any reason not to?
ACKs for top commit:
jamesob:
Github ACK fae7c50d20
dergoegge:
utACK fae7c50d20
Tree-SHA512: e45122d73fafb17cea312258314b826cb0745e08daadd28465f687ec02d4c127d2f8cbe20179a9fff5712038850c02c968abb4838fa088b7555e28709317d3a3
